A fair question.
I would suggest much like SFH, its a numerical one.
Building a home for one family that may exclude others, and/or exclude them from floors other than the main floor, might be seen as marginally unfortunate, but reasonable.
Doing so for two families (stacked townhome) perhaps likewise.
Once you get to 3-storey walk-ups with no unit limit, we could be talking 20+ units which are inaccessible, on a single lot.
Again, I'm not saying I'm against that, but I certainly think there may be arguments to be heard on the question.