José Luis Ricón presents a systematic review of the effectiveness of mastery learning, tutoring, and direct instruction and draws these conclusions about Bloom’s two-sigma problem:
Bloom noted that mastery learning had an effect size of around 1 (one sigma); while tutoring leads to d=2. This is mostly an outlier case.
Nonetheless, Bloom was on to something: Tutoring and mastery learning do have a degree of experimental support, and fortunately it seems that carefully designed software systems can completely replace the instructional side of traditional teaching, achieving better results, on par with one to one tutoring. However, designing them is a hard endeavour, and there is a motivational component of teachers that may not be as easily replicable purely by software.
Overall, it’s good news that the effects are present for younger and older students, and across subjects, but the effect sizes of tutoring, mastery learning or DI are not as good as they would seem from Bloom’s paper. That said, it is true that tutoring does have large effect sizes, and that properly designed software does as well. The DARPA case study shows what is possible with software tutoring, in the case the effect sizes went even beyond Bloom’s paper.
Also, other approaches to education also have shown large effect sizes, and so one shouldn’t privilege DI/ML here. The principles behind DI/ML (clarity in what is taught, constant testing, feedback, remediation) are sound, and they do seem more clearly effective for disadvantaged children, so for them they are worth trying. For gifted children, or in general intelligent individuals, the principles of the approaches do still make sense, but how much of an effect do they have? In this review I have not looked at this question, but suffice to say that I haven’t found numerous mentions of work targeting the gifted.
That aside, if what one has in mind is raising the average societal skill-level by improving education, that’s a different matter, and that’s where the evidence from the DI literature is less compelling, the effects that do seem to emerge are weaker, perhaps of a quarter of a standard deviation at best. ML does fare better in the general student population, and for college students too.
As for the effect of diverse variables on the effects, studies tend to find that the effects of DI/ML fade over time — but don’t fully disappear — and that less skilled students benefit more than highly capable ones, and the effects vary greatly on what is being tested. Mastery learning, it seems, works by overfitting to a test, and the chances that those skills do not generalise are nontrivial. As in Direct Instruction, if what is desired is mastery of a few key core concepts, especially with children with learning disabilities, it may be well suited for them. But it is yet unclear if DI are useful for average kids. For high SES kids, it seems unlikely that they would benefit.
(Hat tip to Gwern.)
Hey Isegoria, I’m not sure I ever did thank you for that Bloom’s 2-Sigma article. I tutored a kid failing math for a little bit this year to both help him and to test this whole tutoring thing. I literally told the mom about Bloom’s results and offered to tutor her kid for free.
I was astounded not only at the things he was missing, but at the things he had learned the wrong way years before (e.g. subtraction of negative numbers, > and < signs). Things learned the wrong way took many times more effort to overcome than it would have taken to learn right the first time.
I guess in all we did 12 sessions of about 80 minutes each (before I would time out) and he went from 40% (and about to fail getting in to the high school he wanted) to getting 84% and being one happy kid with two happy parents.
A few notes:
- the kid (14yo) was full of energy and motivated to come by each week. I do not know why. This may have been 50% of the value of the work.
- My estimate of his potential before the tutoring was that he could get 60% and a pass. 84% was a shock.
- it was hard work for me because I had to identify the areas that needed work and then often create new questions to drill him on his weaknesses. I had to be "on" at a time I usually like to be quiet. I really did time out after 80 minutes.
- The textbooks weren't awful but by design left out a lot of explanation so the teachers had something to do. This limited their usefulness without having a teacher or tutor present, although tbf only a few kids would ever learn from a textbook.
- Past exams are the bomb. You get to learn what the question formats are, suffer less stress during the actual exam and are just faster at answering when minutes are precious.
- We did the entire year's curriculum in 8 weeks including areas missing from the previous three years. The last 4 weeks were spent on past exams. They spend several hours a week for four years teaching this stuff and it is mind numbing for everyone.
- The kid got better at working solo, mostly at the past exam stage. He'd do an exam during the week then another one at our session. During the session he'd spend most of the time working but punctuated with a bit of work when he'd finished one of the questions.
- The results went so well I felt awful I hadn't tutored my eldest kid the same way for the same test.
- I demanded silence about my role from the parents since I do not want to be inundated with requests. I have my own kids to deal with.
In 2nd grade, my school district tested to find the 25 or so brightest kids. We were then pushed through 3rd, 4th, 5th grade material in 2 years. Drill & kill math, speed reading, etc. Some on your own work, sone organized. But the same teacher for 2 years.
Usually that school district got 0-2 National Merit SemiFinalists. The year we graduated it was 13. Back to 0-2 the next years.
Exactly what worked so well, I cannot pinpoint. But the pessimism about finding something that works better for the gifted kids is unwarranted.
Finding something that is politically acceptable is going to be difficult. One opinion heard on why that program was a once and only was that there were too many blue collar and farmer kids in the class, and not enough doctor and lawyer’s kids.
Tutoring works mostly because of fast feedback systems.
Learning is (effectively) ALL learning to do.
Learning to do requires several items
1. Intent — student must be focused in the moment on doing/learning. Focused on the hot redhead or the next game ain’t gonna work.
2. Practice — student must be doing, not listening/reading. Until they are good enough
3. Feedback — if a student isnt expert enough, or even if they are … student must have a feedback system in place to warn them if they go astray.
Of course tutoring works. It addresses the 3 big issues in learning … unless the tutor is too cute.
Fyi… commenting on you site is extra hard from cell. Submit button disappears usually.
Aretae,
There’s also the “someone is paying attention to me” effect.
Remember that the early “time and motion” guys found that making random changes to the workplace produced just as much effect as the carefully planned ones; speculation stemmed from that which led to the suspicion that what might have been more important to things improving was that someone, anyone, was paying attention to the employees.
Kid sitting in a classroom with thirty other students is not going to do as well with that impersonal sort of instruction as one who is one-on-one with someone else, regardless of the quality of instruction. Johnny responds to attention, period. That’s just the way it is.
What’s worth looking at, in my opinion, is examining just what the hell creates this effect; it’s almost placebo-like, in terms of monitoring performance. If you have someone on-scene, and watching, small groups do better than if left alone, even if the monitor has zero clue what the hell is going on. It’s the psychology of it all–”We’re being observed…”.
There may be ways to boost certain pupils’ performance in certain subjects in certain contexts, but smallish tweaks to the pedagogic process mean little in comparison to a few key things, namely: interest — the student must find the material intrinsically rewarding or recognize its extrinsic necessity; material — the material must exist, must be of the highest standard, and must be readily available; and focus — “the greatest minds of my generation are figuring out how to make people click 0.1% more ads” is a literal description, the “attention economy” is another way of saying the re-engineering of your and your family’s consciousness, and these platforms are unironically and unhyperbolically a biohazard and the greatest threat to the security of the national mind ever devised by man, and ordinary television is increasingly looking positively benign in comparison.
Other notes: a professor, the quintessential product of the system, will never produce a meaningful reform to the system; the state has no interest in producing citizens capable of thinking for themselves; it’s quite difficult for a less-intelligent adult to meaningfully teach a more-intelligent child; if we (for some value of “we”) really, truly cared about any of this, we would re-order our civilization such that “teacher” would be a great and manly profession, highly paid, highly desired, and to call oneself a “teacher” would be a great privilege, rather than being something performed, for the most part, by the women who couldn’t get much of anything else going.
The Great CVLR Plan of Education Reform: Construct really great libraries of marble, wood, gold, and faraday; fill them with the smell of ancient, leather-bound books; and lock the literate cheeldrin in them for eight hours a day. If you don’t care enough to have a library of the classics, you don’t care at all — you shouldn’t even have a school, much less a bloodsucking teacher’s union. Purveyors of daycare should be called what they are: babysitters… of babies. The illiterate and unread, toddlers and up, can be given pellet guns and released into the forests to spend their days learning something-anything as far away from the merciless fat well-unionized terrorists as possible.
P.S. Under the Dictatorship of the CVLR, the collection of “safety” statistics will be verboten on pain of death, the “responsible” courses of action implied by the observation of these statistics, including but not limited to the inexorable encroachment of the bureaucratic nanny state, having been finally, under the auspices of my steely, aristocratic gaze, recognized as being the most dangerous thing of all.
And the last words heard by the offender, heretic, and Breaker of the Faith shall be, Lies, damned lies, and statistics!
The craftsman/apprentice model has worked for centuries as has tutoring and other one on one training. The software messiahs are mostly full of crap since they don’t understand human nature.
Kirk:
I think I can sweep the factory lighting effect “someone is paying attention to me” under the “intent” rug.
How much attention I pay to what I’m doing has something (a lot) to do with social pressure.
That said … I agree that this facet of intent is a part of improvement and that I didn’t call it out.
That’s an awesome story, my anonymized friend. I wouldn’t feel too guilty that you didn’t tutor your eldest kid the same way — unless your eldest actually listens to you and does what you ask.
Is anyone else finding that the “submit” button disappears? It seems fine on Safari on iOS.
Isegoria,
It has to do, I think, with screen size issues, and maybe Android.
Reading on a Note 8 (large phone), and for years before (Note 5, Note 3), when I hold the phone in portrait orientation … once I type into the text box a bit, it grows and covers the submit button.
If it’s only a line or two, sometimes I can hit the invisible submit button behind the comment text box, but if it’s more than that line, I can’t.
Only recently, I’ve succeeded in addressing this 10y issue by rotating my phone to landscape, which doesn’t resize the textbox, thus not covering the submit comment button.