Do online math programs work?
In August 2022, three researchers at Khan Academy, a popular math practice website, published the results of a massive, 99-district study of students. It showed an effect size of 0.26 standard deviations (SD) — equivalent to several months of additional schooling — for students who used the program as recommended.
A 2016 Harvard study of DreamBox, a competing mathematics platform, though without the benefit of Sal Khan’s satin voiceover, found an effect size of 0.20 SD for students who used the program as recommended. A 2019 study of i-Ready, a similar program, reported an effect size in math of 0.22 SD — again for students who used the program as recommended. And in 2023 IXL, yet another online mathematics program, reported an effect size of 0.14 SD for students who used the program as designed.
Those gains, and many others like them reported each year, are impressive. Since use of these tools is widespread, one could be forgiven for asking why American students are not making impressive gains in math achievement. John Gabrieli, an MIT neuroscientist, declares himself “impressed how education technology has had no effect on…outcomes.” He was talking about reading but could equally have called out mathematics, the other big area in which education technology is widely used but growth in achievement has not followed.
A clue is in those wiggle words “students who used the program as recommended.” Just how many students do use these programs as recommended — at least 30 minutes per week in the case of Khan Academy? The answer is usually buried in a footnote, if it’s reported at all. In the case of the Khan study, it is 4.7 percent of students. The percentage of students using the other products as prescribed is similarly low.
Thirty minutes per week is comically little. People spend that much time zipping and unzipping their trousers. Mastery is produced when innate ability meets external motivation, and external motivation is based either on avoiding risk or seeking reward. “We just need to motivate these kids” talk invariably stems from the same basic weaselry as corporate “we’re a family” or “take ownership of your work” bullshit, to say nothing of whatever the governmental equivalent is, probably the latest New And Improved™ formulation of “ask what you can do for your country”. Offering young white men the opportunity to forcibly dethrone long-toothed hostile geriatrics from their positions of status and power and wealth might move student mathematics performance. It’s worth a shot. What’s the worst thing that could happen that’s worse than what’s already happening?
And, yes, I am, in fact, proposing to make teenagers mayors, sheriffs, auditors, aspiring defense contractors, movie directors, CIA spooks, USPIS wagies, field generals, and whatever else. It’ll be hilarious.
My gloriously dry wit was lost to a 503 error page. Is the server working now?
The bug was probably due to a split tab in Brave. Ahem, as I was saying, if Khan does not enthrall 95% of the students who get thrown at it (perhaps by tiger moms resembling Yuen Qiu playing the landlady from Kung Fu Shuffle), then competitors should take the big risk of launching competitive services that engage students even more than Khan Academy does. Doing so in an age where students can use AI-LLMs to search for answers is tricky.
(I just tried posting a comment without a split tab and got another 503 page.)