Measuring the things universities say they want students to learn

Monday, February 13th, 2012

The big open secret in American higher education, Jonathan Zimmerman says, is that most institutions have no meaningful way to measure the quality of their instruction:

My New York University colleague Richard Arum and the University of Virginia’s Josipa Roksa recently tracked several thousand undergraduates as they moved through two dozen U.S. universities. They found that almost half of them didn’t significantly improve their reasoning or writing skills during the first two years of college. And after four years, subsequent research showed, more than one-third of students still showed no significant gains in these areas.

Arum and Roksa based their conclusions on results from the College Learning Assessment, or CLA, an essay test that tries to measure the things universities say they want students to learn: critical thinking, complex reasoning and written expression. One sample question provides several documents about an airplane that crashed, then asks students to advise an executive about whether his company should purchase that type of plane. Another test item presents crime data from a city and asks students to counsel its mayor about how to respond to criticisms of his policing policies.

The CLA was administered to more than 2,300 students at 24 institutions, ranging from big state universities and selective liberal arts schools to historically black and Latino institutions. Forty-five percent of the students showed no significant gains on the CLA between their freshman and sophomore years, and 36% didn’t improve significantly between their freshman and senior years.

And why should they? College students spend about 12 hours a week studying, on average, and one-third of them report studying less than five hours per week. More than half the students in Arum and Roksa’s sample said they had not taken a single class in the semester before they were surveyed that required a total of 20 pages of writing.


  1. I’ll take the critique, but I’d really love to see some finer-grained detail. G is a pretty hard constraint, and the bottom tier of incoming students probably isn’t capable of being taught what they’d need to improve outcomes on those kinds of tests without too great a degradation in the educational experience of their classmates.

    If the book shows no learning across all SAT deciles for incoming students, that would be damning indeed. But if the bottom third or so isn’t learning anything, that doesn’t really much surprise me.

Leave a Reply