Being middle class is a huge handicap, it would appear.

Be careful interpreting that: admission is not the same as matriculation. Consider a rich kid who gets admitted to five Ivies vs. a middle class kid who only gets admitted to one: each can attend only one; both will get an elite education.

There's still a gap in attendance, but IIRC it's significantly smaller.