In his speech at the recent New Yorker conference, Malcolm Gladwell explains the mismatch between the metrics we use to assess potential new hires and new hires’ actual performance on the job. (This is the subject of his upcoming book, Outliers: Why Some People Succeed and Some Don’t.)
If we look at professional-sport combines, teacher credentials, and law-school applications, they have zero predictive capacity. The best quarterbacks score the worst on the IQ test given to quarterbacks, teachers with additional training and credentials teach no better than other teachers, and lawyers who get into a prestigious law school via affirmative action do just as well in their career as higher-scoring lawyers.
But I think he misses — or skirts — the issue of why these predictive metrics aren’t very predictive. It’s not simply that hard, objective criteria are bad. If you have a set of physical metrics that predict athletic performance in the population at large — height, weight, vertical leap, etc. — they won’t predict performance in a tiny sample of athletes who have already been selected for a particular level of performance. In the case of athletes at a combine, of course, these are athletes in the top fraction of a percent of players, but the same thing would presumably happen if we looked only at players between the 50th and 51st percentile.
In the case of teachers, the issue is that the training and credentialing have never been intended to improve teaching performance. The credentials are there to keep out competition. It is a highly unionized profession, after all.
A bigger issue in Gladwell’s analysis though — at least as far as I can tell from his short speech, which is just a summary — is how he defines good and bad teachers. Apparently he defines good teachers as those whose students improved the most in percentile rank over the school year and bad teachers as those whose students dropped the most in percentile rank over the school year, which seems like it would grossly exaggerate the performance difference between good and bad teachers, because the teacher is just one tiny variable in a system with a lot of noise — and because percentile scores aren’t z-scores. (A jump of one standard deviation can take you from 31st percentile to 69th, a 38-rank jump, or from 97.7th to 99.9th, a 2-rank jump.) I suppose I’ll have to see the study he’s referencing.
Gladwell’s last example seems particularly odd, given that we know that affirmative action backfires when law schools admit less-qualified black applicants:
Easily the most startling conclusion of his research: [UCLA law professor Richard] Sander calculated that there are fewer black attorneys today than there would have been if law schools had practiced color-blind admissions — about 7.9% fewer by his reckoning. He identified the culprit as the practice of admitting minority students to schools for which they are inadequately prepared. In essence, they have been “matched” to the wrong school.
[...]
While some students will outperform their entering academic credentials, just as some students will underperform theirs, most students will perform in the range that their academic credentials predict. As a result, in elite law schools, 51.6% of black students had first-year grade point averages in the bottom 10% of their class as opposed to only 5.6% of white students. Nearly identical performance gaps existed at law schools at all levels. This much is uncontroversial.
[...]
The Sander study argued that the most plausible explanation is that, as a result of affirmative action, black and white students with similar credentials are not attending the same schools. The white students are more likely to be attending a school that takes things a little more slowly and spends more time on matters that are covered on the bar exam. They are learning, while their minority peers are struggling at more elite schools.Mr. Sander calculated that if law schools were to use color-blind admissions policies, fewer black law students would be admitted to law schools (3,182 students instead of 3,706), but since those who were admitted would be attending schools where they have a substantial likelihood of doing well, fewer would fail or drop out (403 vs. 670). In the end, more would pass the bar on their first try (1,859 vs. 1,567) and more would eventually pass the bar (2,150 vs. 1,981) than under the current system of race preferences. Obviously, these figures are just approximations, but they are troubling nonetheless.
On the other hand, “success” — unlike bar-exam pass rates — can be hard to measure, and potential lawyers may vary dramatically in how much “success” they achieve, because such success means great sacrifice, a point Gladwell has made elsewhere:
But what did Hunter achieve with that best-students model? In the nineteen-eighties, a handful of educational researchers surveyed the students who attended the elementary school between 1948 and 1960. [The results were published in 1993 as “Genius Revisited: High IQ Children Grown Up,” by Rena Subotnik, Lee Kassan, Ellen Summers, and Alan Wasser.] This was a group with an average I.Q. of 157 — three and a half standard deviations above the mean—who had been given what, by any measure, was one of the finest classroom experiences in the world. As graduates, though, they weren’t nearly as distinguished as they were expected to be. “Although most of our study participants are successful and fairly content with their lives and accomplishments,” the authors conclude, “there are no superstars . . . and only one or two familiar names.” The researchers spend a great deal of time trying to figure out why Hunter graduates are so disappointing, and end up sounding very much like Wilbur Bender. Being a smart child isn’t a terribly good predictor of success in later life, they conclude. “Non-intellective” factors — like motivation and social skills — probably matter more. Perhaps, the study suggests, “after noting the sacrifices involved in trying for national or world-class leadership in a field, H.C.E.S. graduates decided that the intelligent thing to do was to choose relatively happy and successful lives.” It is a wonderful thing, of course, for a school to turn out lots of relatively happy and successful graduates. But Harvard didn’t want lots of relatively happy and successful graduates. It wanted superstars, and Bender and his colleagues recognized that if this is your goal a best-students model isn’t enough.