The big difference between modernism and other architectural styles that preceded it is that previous architectural styles thought buildings should achieve energy efficiency and engage pedestrians positively.
Modernists thought of 'the house as a machine' and intended their works to be an artistic expression. This was incredibly selfish and has hurt the millions of people worldwide that have had to make themselves at home in energy-intensive eye-sores.
Look no further than at U of T for a fair comparison of what architectural styles with similar budgets and purposes have provided us with. Students love to have classes in the older buildings on campus for they have beautiful large windows and finely crafted details all over than engage and entertain. Older buildings have deliciously functional courtyards that have become well-loved by all. Modernist buildings in contrast provide their users with a horrible experience... windowless classrooms with blinding lights and clad in cold boring materials abound. Public spaces were badly thought out and feel sketchy and unnatural.
Why should we save buildings then their developers and designers clearly didn't give a damn about those that'd come to inhabit them?
P.S. I love Robarts for it quite clearly tries. I also like some early modernist buildings, so it's not like there's no exceptions to my point above, but it quite clearly applies to many.
Many modernists tried to design buildings that would engage pedestrians positively. They just believed that pedestrians would prefer to address buildings through green spaces and plazas than through conventional streetfront entrances. But I understand your points.