A proposal for an archive revisiter

Thursday, November 8th, 2018

In his long list of statistical notes, Gwern includes a proposal for an archive revisiter:

One reason to take notes/clippings and leave comments in stimulating discussions is to later benefit by having references & citations at hand, and gradually build up an idea from disparate threads and make new connections between them. For this purpose, I make extensive excerpts from web pages & documents I read into my Evernote clippings (functioning as a commonplace book), and I comment constantly on Reddit, LessWrong, HN, etc. While expensive in time & effort, I often go back, months or years later, and search for a particular thing and expand & integrate it into another writing or expand it out to an entire essay of its own. (I also value highly not being in the situation where I believe something but I do not know why I believe it other than the conviction I read it somewhere, once.)

This sort of personal information management using simple personal information managers like Evernote works well enough when I have a clear memory of what the citation/factoid was, perhaps because it was so memorable, or when the citations or comments are in a nice cluster (perhaps because there was a key phrase in them or I kept going back & expanding a comment), but it loses out on key benefits to this procedure: serendipity and perspective.

As time passes, one may realize the importance of an odd tidbit or have utterly forgotten something or events considerably changed its meaning; in this case, you would benefit from revisiting & rereading that old bit & experiencing an aha! moment, but you don’t realize it. So one thing you could do is reread all your old clippings & comments, appraising them for reuse.

But how often? And it’s a pain to do so. And how do you keep track of which you’ve already read? One thing I do for my emails is semi-annually I (try to) read through my previous 6 months of email to see what might need to be followed up on10 or mined for inclusion in an article. (For example, an ignored request for data, or a discussion of darknet markets with a journalist I could excerpt into one of my DNM articles so I can point future journalists at that instead.) This is already difficult, and it would be even harder to expand. I have read through my LessWrong comment history… once. Years ago. It would be more difficult now. (And it would be impossible to read through my Reddit comments as the interface only goes back ~1000 comments.)

Simply re-reading periodically in big blocks may work but is suboptimal: there is no interface easily set up to reread them in small chunks over time, no constraints which avoid far too many reads, nor is there any way to remove individual items which you are certain need never be reviewed again. Reviewing is useful but can be an indefinite timesink. (My sent emails are not too hard to review in 6-month chunks, but my IRC logs are bad – 7,182,361 words in one channel alone – and my >38k Evernote clippings are worse; any lifestreaming will exacerbate the problem by orders of magnitude.) This is probably one reason that people who keep journals or diaries don’t reread Nor can it be crowdsourced or done by simply ranking comments by public upvotes (in the case of Reddit/LW/HN comments), because the most popular comments are ones you likely remember well & have already used up, and the oddities & serendipities you are hoping for are likely unrecognizable to outsiders.

This suggests some sort of reviewing framework where one systematically reviews old items (sent emails, comments, IRC logs by oneself), putting in a constant amount of time regularly and using some sort of ever expanding interval between re-reads as an item becomes exhausted & ever more likely to not be helpful. Similar to the logarithmically-bounded number of backups required for indefinite survival of data (Sandberg & Armstrong 2012), Deconstructing Deathism – Answering Objections to Immortality, Mike Perry 2013 (note: this is an entirely different kind of problem than those considered in Freeman Dyson’s immortal intelligences in Infinite in All Directions, which are more fundamental), discusses something like what I have in mind in terms of an immortal agent trying to review its memories & maintain a sense of continuity, pointing out that if time is allocated correctly, it will not consume 100% of the agent’s time but can be set to consume some bounded fraction.

[...]

So you could imagine some sort of software along the lines of spaced repetition systems like Anki, Mnemosyne, or Supermemo which you spend, say, 10 minutes a day at, simply rereading a selection of old emails you sent, lines from IRC with n lines of surrounding context, Reddit & LW comments etc; with an appropriate backoff & time-curve, you would reread each item maybe 3 times in your lifetime (eg first after a delay of a month, then a year or two, then decades). Each item could come with a rating function where the user rates it as an important or odd-seeming or incomplete item and to be exposed again in a few years, or as totally irrelevant and not to be shown again – as for many bits of idle chit-chat, mundane emails, or intemperate comments is not an instant too soon! (More positively, anything already incorporated into an essay or otherwise reused likely doesn’t need to be resurfaced.)

This wouldn’t be the same as a spaced repetition system which is designed to recall an item as many times as necessary, at the brink of forgetting, to ensure you memorize it; in this case, the forgetting curve & memorization are irrelevant and indeed, the priority here is to try to eliminate as many irrelevant or useless items as possible from showing up again so that the review doesn’t waste time.

More specifically, you could imagine an interface somewhat like Mutt which reads in a list of email files (my local POP email archives downloaded from Gmail with getmail4, filename IDs), chunks of IRC dialogue (a grep of my IRC logs producing lines written by me +- 10 lines for context, hashes for ID), LW/Reddit comments downloaded by either scraping or API via the BigQuery copy up to 2015, and stores IDs, review dates, and scores in a database. One would use it much like a SRS system, reading individual items for 10 or 20 minutes, and rating them, say, upvote (this could be useful someday, show me this ahead of schedule in the future) / downvote (push this far off into the future) / delete (never show again). Items would appear on an expanding schedule.

[...]

As far as I know, some to-do/self-help systems have something like a periodic review of past stuff, and as I mentioned, spaced repetition systems do something somewhat similar to this idea of exponential revisits, but there’s nothing like this at the moment.

Students don’t know how they study and learn best

Wednesday, November 7th, 2018

Some progressive teachers take pride in allowing students to choose how they study and learn best, but there’s a serious flaw they overlook: students don’t know how they study and learn best:

Karpicke, Butler, and Roediger III (2009) (1) explored study habits used by college students. They surveyed 177 students and asked them two questions. For the sake of this post, I will only focus on question one:

What kind of strategies do you use when you are studying? List as many strategies as you use and rank-order them from strategies you use most often to strategies you use least often.

The results? Repeated rereading was by far the most frequently listed strategy (84% reported using) and 55% reported that it was their number one strategy used. Only 11% reported practicing recall (self-testing) of information and 1% identified practicing recall as their number one strategy. This is not good for student-choice of study. 55% of those surveyed intuitively believed that rereading their notes best utilized their study time…assuming students intended on using their time most effectively. This is just not so.

A phenomenon known as the testing effect indicates that retrieving information from memory has a great effect on learning and strengthens long-term retention of information (2). The testing effect can take many forms, with the most important aspect being students retrieve information. A common saying in my room is to make sure my students are only using their brain…if you’re using notes, the textbook, or someone else’s brain, you’re not doing it right. While many correctly see this attempt as a great way to regulate and assess one’s knowledge, the act of recalling and retrieving strengthens long-term retention of information.

This is not so with repetitive rereading. Memory research has shown rereading by itself is not an effective or efficient strategy for promoting learning and long-term retention (3). Perhaps students believe the more time I spend studying, the more effective the learning. Is it correct to believe that the longer I study something and keep it in my working memory, the better I will remember it? No.

Stop when you’re almost finished

Friday, November 2nd, 2018

Performance psychologist Noa Kageyama recommends harnessing resumptive drive, or the Zeigarnik effect, to get yourself to practice when you don’t feel like it:

Bluma Zeigarnik described a phenomenon way back in 1927, in which she observed while sitting in a restaurant that waiters seemed to have a selective memory. As in, they could remember complicated customers’ orders that hadn’t yet been filled, but once all the food had been served (or maybe when the bill was paid?), it’s as if the order was wiped from their memory.

Back in her lab, she found that indeed, participants were much more likely to remember tasks they started but didn’t finish, than tasks that were completed (hence, the Zeigarnik effect).

Another form of the Zeigarnik effect — and the one more relevant to what we’re talking about here — is the observation that people tend to be driven to resume tasks in which they were interrupted and unable to finish.

Researchers at Texas Christian University & University of Rochester ran a study on this form of the Zeigarnik effect.

Subjects were given eight minutes to shape an eight-cube, three-dimensional puzzle into five different forms. They were told to work as quickly as possible, and given three minutes to complete the first two puzzles as practice.

Then they were given five minutes to solve the last three puzzles.

The researchers deliberately made the second practice puzzle difficult — one that was unlikely to be solved within the time available. And just as they had hoped, only 6 of the 39 participants solved the difficult puzzle.

After their time was up, the participants had eight minutes of free time to do as they wished while the researcher running the experiment left the room to retrieve some questionnaires they accidentally forgot to bring, saying they would be back in “5 or 10 minutes.” This was all a ruse, of course, to see what the participants would do when left alone.

Despite there being other things in the room to do (e.g. a TV, magazines, newspaper, etc.), 28 of the 39 participants (72%) resumed working on the puzzles.

[...]

Of the six who completed the difficult puzzle, only one (17%) resumed working on the puzzles (and did so for one minute and 18 seconds).

Of the 33 who did not complete the challenging puzzle, 27 (82%) resumed working on the puzzle, and on average, spent more than two and a half times as long (3:20) working on the puzzles.

So, when interrupted in the middle of a task, not only were participants more motivated to resume working on that task, but they also continued working on it for much longer.

[...]

So instead of thinking about practicing for an hour, or having to work on 10 excerpts, or memorize a concerto, just tune your instrument. Or play a scale really slowly. Or set the timer for five minutes and pick one little thing to fix. And if at the end of five, you don’t feel like continuing, put your instrument away and try again later.

Don’t feel like studying? Just crack open the book. Work on one math problem. Write three sentences of your essay. Create two flash cards.

Second, once you’ve finally gotten yourself into the mood to practice or study, try stopping in the middle of a task. Meaning, if you’re working on a tricky passage that has you stumped, test out a few solutions, but leave yourself a few possible solutions remaining before taking a practice break. Stop when you’re almost finished solving the math problem. Or in the middle of a sentence.

It’s not what you know, but whether you use it

Thursday, November 1st, 2018

Two researchers from the City University of New York did a study of basketball players to discern a difference between the practice habits of the best free throw shooters (70% or higher) and the worst free throw shooters (55% or lower):

Difference #1: Goals were specific

The best free throw shooters had specific goals about what they wanted to accomplish or focus on before the made a practice free throw attempt. As in, “I’m going to make 10 out of 10 shots” or “I’m going to keep my elbows in.”

The worst free throw shooters had more general goals — like “Make the shot” or “Use good form.”

Difference #2: Attributions of failure were specific

Invariably, the players would miss shots now and again, but when the best free throw shooters missed, they tended to attribute their miss to specific technical problems — like “I didn’t bend my knees.” This lends itself to a more specific goal for the next practice attempt, and a more thoughtful reflection process upon the hit or miss of the subsequent free throw. Far better than saying “I suck” or “What’s wrong with me?” or “Crap, I’m never going to get this.”

In contrast, the worst performers were more likely to attribute failure to non-specific factors, like “My rhythm was off” or “I wasn’t focused” which doesn’t do much to inform the next practice attempt.

It’s not what you know, but whether you use it

You might be thinking that perhaps the worst performers didn’t focus on specific technical strategies because they simply didn’t know as much. That perhaps the best performers were able to focus on technique and strategy because they knew more about how to shoot a free throw with proper form.

The researchers thought of this as well, and specifically controlled for this possibility by testing for the players’ knowledge of basketball free throw shooting technique. As it turns out, there were no significant differences in knowledge between experts and non-experts.

So while both the top performers and the worst performers had the same level of knowledge to draw from, very few of the worst performers actually utilized this knowledge base. Meanwhile, the best performers were much more likely to utilize their knowledge to think, plan, and direct their practice time more productively.

Any idiot can train himself into the ground

Sunday, October 28th, 2018

Performance psychologist Dr. Noa Kageyama discusses the importance of mentally disengaging from work and practice:

A group of German and US researchers conducted a study of 109 individuals. The setup was pretty simple, consisting of two surveys, spaced 4 weeks apart to see how participants’ mental and emotional states might change over time.

The researchers were primarily interested in the relationship between psychological detachment (our ability to disengage from work during our “off” hours — a key factor in greater well-being and performance), exhaustion (feeling fatigued, emotionally drained/overwhelmed, and unable to meet the demands of our work), time pressure, and pleasurable leisure activities (the degree to which we engage in activities that recharge our batteries and balance out our work demands).

There were a couple interesting findings that came out of the resulting data.

Exhaustion begets exhaustion

You would think that emotionally exhausted folks would be more detached and disengaged from work in their off-work hours. Paradoxically, the opposite seems to be true.

The data suggest that individuals who were exhausted had an increasingly difficult time disconnecting from work concerns as the weeks went by. The idea being, when we’re exhausted, we tend not to do our best work, which makes us feel less capable of meeting the demands of the situation, which makes us worry more and expend even more energy, effort, and time trying to make up for our sub-par work, which only keeps the cycle of worry/practice/exhaustion going.

To use a music example, when we have a big audition coming up, there’s a tendency to worry more about our level of preparation, which leads us to practice more, worry more, and obsess more, which in turn makes it harder to disengage, take a break, and recoup our energy outside of the practice room, so we can dive back in refreshed, recharged, and ready to do our most productive and focused work.

Indeed, someone recently suggested to me that while our instinct when behind in our work is to put in a few extra hours at the office after work to catch up, what ends up happening is that we get home late, feel even more tired and drained, get less rest and relaxation, and return to work tired yet again to repeat the cycle. Instead, she suggested that it’s more productive to go home early, get quality R&R, and go to work early the next morning, fresher, more productive, and more motivated to get things done.

Time pressure makes things worse

The other finding was that time pressure seems to make detaching from work more difficult if you’re already feeling exhausted. As in, exhausted folks find it increasingly difficult to mentally detach from work and get the mental/physical break they need when they feel like they’re on a time crunch.

This makes sense too, as the less time we have to prepare, and the closer we get to the day of a big audition, the more likely we are to worry, stress, and obsess about it, even when we’re not practicing.

[...]

As Olympic marathoner Keith Brantly once said, “Any idiot can train himself into the ground; the trick is working in training to get gradually stronger.”

If you’re going to practice, you might as well do it right

Saturday, October 27th, 2018

The most valuable lesson Noa Kageyama learned from playing the violin was, if you’re going to practice, you might as well do it right:

I began playing the violin at age two, and for as long as I can remember, there was one question which haunted me every day.

Am I practicing enough?

I scoured books and interviews with great artists, looking for a consensus on practice time that would ease my conscience. I read an interview with Rubinstein, in which he stated that nobody should have to practice more than four hours a day. He explained that if you needed that much time, you probably weren’t doing it right.

And then there was violinist Nathan Milstein who once asked his teacher Leopold Auer how many hours a day he should be practicing. Auer responded by saying “Practice with your fingers and you need all day. Practice with your mind and you will do as much in 1 1/2 hours.”

Even Heifetz indicated that he never believed in practicing too much, and that excessive practice is “just as bad as practicing too little!” He claimed that he practiced no more than three hours per day on average, and that he didn’t practice at all on Sundays.

[...]

Here are the five principles I would want to share with a younger version of myself. I hope you find something of value on this list as well.

1. Focus is everything
Keep practice sessions limited to a duration that allows you to stay focused. This may be as short as 10-20 minutes, and as long as 45-60+ minutes.

2. Timing is everything, too
Keep track of times during the day when you tend to have the most energy. This may be first thing in the morning, or right before lunch. Try to do your practicing during these naturally productive periods, when you are able to focus and think most clearly. What to do in your naturally unproductive times? I say take a guilt-free nap.

3. Don’t trust your memory
Use a practice notebook. Plan out your practice, and keep track of your practice goals and what you discover during your practice sessions. The key to getting into “flow” when practicing is to constantly strive for clarity of intention. Have a crystal clear idea of what you want (e.g. the sound you want to produce, or particular phrasing you’d like to try, or specific articulation, intonation, etc. that you’d like to be able to execute consistently), and be relentless in your search for ever better solutions.

When you stumble onto a new insight or discover a solution to a problem, write it down! As you practice more mindfully, you’ll began making so many micro-discoveries that you will need written reminders to remember them all.

4. Smarter, not harder
When things aren’t working, sometimes we simply have to practice more. And then there are times when it means we have to go in a different direction.

I remember struggling with the left-hand pizzicato variation in Paganini’s 24th Caprice when I was studying at Juilliard. I kept trying harder and harder to make the notes speak, but all I got was sore fingers, a couple of which actually started to bleed (well, just a tiny bit).

Instead of stubbornly persisting with a strategy that clearly wasn’t working, I forced myself to stop. I brainstormed solutions to the problem for a day or two, and wrote down ideas as they occurred to me. When I had a list of some promising solutions, I started experimenting.

I eventually came up with a solution that worked, and the next time I played for my teacher, he actually asked me to show him how I made the notes speak so clearly!

5. Stay on target with a problem-solving model
It’s extraordinarily easy to drift into mindless practice mode. Keep yourself on task using the 6-step problem solving model below.

1. Define the problem (What result did I just get? What do I want this note/phrase to sound like instead?)
2. Analyze the problem (What is causing it to sound like this?)
3. Identify potential solutions (What can I tweak to make it sound more like I want?)
4. Test the potential solutions and select the most effective one (What tweaks seem to work best?)
5. Implement the best solution (Reinforce these tweaks to make the changes permanent)
6. Monitor implementation (Do these changes continue to produce the results I’m looking for?)

Or simpler yet, try out this model from Daniel Coyle’s excellent book The Talent Code.
1. Pick a target
2. Reach for it
3. Evaluate the gap between the target and the reach
4. Return to step one

A fantasy world that stood in as a facsimile for the real one

Tuesday, October 23rd, 2018

It should come as no surprise that D&D players test well:

A group of Grade 9 students in Texas who substantially outperformed their district on a statewide standardized test all had one surprising thing in common: they all were members of the school’s Dungeons & Dragons club.

The real question:

A coincidence? Otherwise, how does a fantasy role-playing game produce improved test scores? The obvious explanation is that the club draws the bright kids who are already academically inclined. But many of the kids in the club at the Title I school had histories of struggling with academics.

For Kade Wells, the teacher who runs the club at Davis Ninth Grade School outside Houston, the answer is simple: “Playing Dungeons & Dragons makes you smarter.”

The two explanations aren’t mutually exclusive.

In one striking example, educational researcher and teacher Alexandra Carter used a student-modified version of Dungeons & Dragons as the centerpiece of a yearlong program with a Grade 3 class that combined math, reading, writing, and social studies. Many students in the class struggled with academic and behavioral challenges, but rooting their core subjects in the game produced remarkable results.

In a paper she authored recounting the experience, Carter describes a wealth of student success stories, both behavioral and academic. “I was able to see progress in all of the students,” summarizes Carter, “and was especially impressed with the work that those who struggled the most produced.”

Carter observes that a great deal of the project’s success hinged on students being motivated to learn and practice skills that applied to the game. Students often have trouble appreciating the value of what they learn in school when it is abstracted from its real-world purpose. In this case, learning was meaningful for the students because it had traction in a fantasy world that stood in as a facsimile for the real one, the central dynamic of play and a key feature of its value for development and learning.

Practice smarter, not harder

Saturday, October 20th, 2018

Researchers looked into what “practice smarter, not harder” really means:

A group of researchers led by Robert Duke of The University of Texas at Austin conducted a study several years ago to see if they could tease out the specific practice behaviors that distinguish the best players and most effective learners.

Seventeen piano and piano pedagogy majors agreed to learn a 3-measure passage from Shostakovich’s Piano Concerto No. 1. The passage had some tricky elements, making it too difficult to sight read well, but not so challenging that it couldn’t be learned in a single practice session.

The students were given two minutes to warm up, and then provided with the 3-measure excerpt, a metronome, and a pencil.

Participants were allowed to practice as long as they wanted, and were free to leave whenever they felt they were finished. Practice time varied quite a bit, ranging from 8 1/2 minutes to just under 57 minutes.

To ensure that the next day’s test would be fair, they were specifically told that they may NOT practice this passage, even from memory, in the next 24 hours.

When participants returned the following day for their test, they were given 2 minutes to warm up, and then asked to perform the complete 3-measure passage in its entirety, 15 times without stopping (but with pauses between attempts, of course).

Each of the pianists’ performances were then evaluated on two levels. Getting the right notes with the right rhythm was the primary criteria, but the researchers also ranked each of the pianists’ performances from best to worst, based on tone, character, and expressiveness.

That led to a few interesting findings:

  1. Practicing longer didn’t lead to higher rankings.
  2. Getting in more repetitions had no impact on their ranking either.
  3. The number of times they played it correctly in practice also had no bearing on their ranking. (wait, what?!)

What did matter was:

  1. How many times they played it incorrectly. The more times they played it incorrectly, the worse their ranking tended to be.
  2. The percentage of correct practice trials did seem to matter. The greater the proportion of correct trials in their practice session, the higher their ranking tended to be.

[...]

Of the eight strategies above, there were three that were used by all three top pianists, but rarely utilized by the others. In fact, only two other pianists (ranked #4 and #6) used more than one:

6. The precise location and source of each error was identified accurately, rehearsed, and corrected.

7. Tempo of individual performance trials was varied systematically; logically understandable changes in tempo occurred between trials (e.g. slowed things down to get tricky sections correct; or speeded things up to test themselves, but not too much).

8. Target passages were repeated until the error was corrected and the passage was stabilized, as evidenced by the error’s absence in subsequent trials.

What’s the common thread that ties these together?

The researchers note that the most striking difference between the top three pianists and the rest, was how they handled mistakes. It’s not that the top pianists made fewer mistakes in the beginning and simply had an easier time learning the passage.

The top pianists made mistakes too, but they managed to correct their errors in such a way that helped them avoid making the same mistakes over and over, leading to a higher proportion of correct trials overall.

The top performers utilized a variety of error-correction methods, such as playing with one hand alone, or playing just part of the excerpt, but there was one strategy that seemed to be the most impactful.

Strategically slowing things down.

After making a mistake, the top performers would play the passage again, but slow down or hesitate – without stopping – right before the place where they made a mistake the previous time.

This seemed to allow them to play the challenging section more accurately, and presumably coordinate the correct motor movements at a tempo they could handle, rather than continuing to make mistakes and failing to identify the precise nature of the mistake, the underlying technical problem, and what they ought to do differently in the next trial.

A combination of scolding and re-instruction

Friday, October 19th, 2018

Did legendary college basketball coach John Wooden rely more on praise or criticism?

Psychologists Roland Tharp and Ronald Gallimore were interested in education and learning, and thought that observing and analyzing John Wooden’s teaching methods might deepen their understanding of learning. Or more specifically, help them understand how more teachers can get the very best out of their students.

So, over the course of 15 practices during the 1974–1975 season (Wooden’s last at UCLA), they sat, observed, and systematically tracked Wooden’s specific coaching behaviors — which added up to 2326 “acts of teaching” in total.

So how much of this was praise? And how much was criticism?

Very little, actually.

Just over half (50.3%) of Wooden’s behaviors were pure instruction — specific statements about what to do or how to do it. No judgment. No approval or disapproval. Just information.

The next most frequently occurring coaching behavior (12.7%) was called a “hustle.” This was basically a cue or reminder to act on some previous instruction. For instance “Drive!” or “Harder!” or, of course, “Hustle!”

Next most frequent was what the researchers affectionately named a “Wooden,” a unique feedback technique that was a combination of scolding and re-instruction (8%). This was designed to make it clear he was not satisfied, but followed by an immediate reminder of the correct way to do something. For example, “How many times do I have to tell you to follow through with your head when shooting?” or “I have been telling some of you for three years not to wind up when you throw the ball! Pass from the chest!”

Next up were praise (6.9%), scolds (6.6%), positive modeling — or how to do something (2.8%), and negative modeling — or how not to do something (1.6%).

So, altogether, ~75% of Wooden’s teaching acts contained specific information geared at providing the athlete with a clearer picture of what to do or what not to do. The researchers felt that this was a major contributor to his coaching success, and it also makes perfect sense given that Wooden, at his core, always saw himself as an educator.

After all, simply knowing that something is good or bad is not especially helpful if you don’t know what exactly should be repeated or changed the next time. Otherwise, it’s just more shots in the dark.

Another of the researchers’ interesting findings was their observation of how Wooden modeled behavior.

When Wooden saw something he didn’t like, and stopped practice to correct the incorrectly executed technique, he would immediately demonstrate the correct way to do the technique, then show everyone the incorrect way the athlete just did it, then model the correct way again.

This correct-incorrect-correct demonstration was usually very brief and succinct, rarely lasting longer than 5 seconds, but making it very clear what his expectations were, and how to meet these expectations.

That rubber-ducking rubber-ducker!

Tuesday, October 16th, 2018

Anyone who has ever tried to solve a problem knows that the surest way to solve it is to call someone over and then explain how it just doesn’t make sense. That someone doesn’t even have to be a real person. (The Pragmatic Programmer calls this rubber-ducking, since explaining all your problems to a cute little toy works just fine.)

A group of researchers studied how to maximize the talking-aloud effect:

109 participants were tasked with solving different variations of the Tower of Hanoi puzzle (try it yourself right here) in the fewest number of moves, before being given a final test on the most challenging variation (to see how effectively they could transfer what they’ve learned to a new problem).

Participants were randomly assigned to one of five groups, each of which was designed to test a different kind of thinking aloud.

Before each move, the “metacognitive” group was asked to answer questions like “How are you deciding which disk to move next?” or “How do you know that this is a good move?” The idea was to get them to adopt a higher-level process focus, by thinking about what they were doing (consciously monitoring performance) and how they were doing — i.e. whether the move was a good one or not (evaluating success/failure/effectiveness).

The “if-then” group’s instructions were a little more rigidly structured, but similarly intended to get them focused on the problem-solving process: “Before each move, I want you to tell me where you are going to move each disk, and why. Specifically, I want you to state this in an ‘if-then’ statement, for example, ‘if I move this disk to this peg, then this will happen’.”

The “problem-focused” group was asked to answer questions like “What is the goal of the problem?” or “What are the rules of the problem?” before each move. The idea was to give them some structure, but not at the higher process level of the other two groups.

The “think-aloud” control group was given no real structure to guide their thinking, but simply told to “think out loud while you are solving this problem. Try to keep talking as much as you can so that I can hear what you are thinking about as you solve the problem.”

The “silent” control group was given no additional instructions beyond the standard instructions for the puzzle, so did no verbalizing of their thoughts.

[...]

On average, the control groups (silent and think-aloud) made more mistakes than the two process-focused (metacognitive and if-then) groups. This was true for every variation of the puzzle during the practice trials — from the easiest 2-disk version to the more complex 5-disk version.

Then, when the participants were tested on their ability to solve the most challenging 6-disk puzzle (to see how effectively they could transfer what they learned from the practice puzzles), the control groups made an average of 2.5 error moves for every correct move vs. just 1 error move for the process-focused groups.

The problem-focused group fared somewhere in the middle. Better than the control groups, but not as good as the process-focused groups.

[...]

1. Unless we are guided, we tend not to focus on or engage in process-level thinking. It’s more natural for us to simply execute a skill, stop, and repeat the skill on “autocorrect” mode until the problem seems to go away. Like playing a passage over and over until it sounds better. Hitting forehand volleys over and over until we get into a groove and everything seems peachy.

Except that in “solving” problems on this implicit level, while we may be able to work ourselves up to a pretty high level of performance in the short term, it involves making more mistakes during the process, and we don’t actually figure out what the solution is, so therefore can’t apply it very effectively to future problems that we might encounter.

2. When, on the other hand, we focus on what we are doing and why we are doing it (whether we are verbalizing these out loud or not), we can not only solve problems more efficiently, but transfer those solution to similar new problems we might encounter later.

How experts get even better

Monday, October 15th, 2018

A team of researchers in the UK asked expert and intermediate players of Gaelic football to perform 10 kicks from the ground (like a penalty kick in soccer) and 10 from their hands (like punting a football) at target zones on the gym wall for points and then had the players practice for 15 minutes, once per week, for four weeks, to compare how experts practice versus non-experts:

Experts work on their weaker areas; intermediates work on their stronger skill.

The experts spent a greater percentage of their time working on their weaker kick — 66% of the time, compared to the intermediate athletes who devoted only 27% of their time to improving their weaker kick.

Not surprisingly, the experts demonstrated significant improvement on their weaker kick from the pre-test to the post-test (improving from 14.4 points to 19.9 points). Their improvement was also more permanent, as their scores remained stable 6 weeks later on the retention test (19.4 points at retention test).

Conversely, while the intermediate players did make significant improvements to their stronger kick from pre-test to post-test (8 points to 14.7 points), their improvement was less stable, as they regressed on the retention test (12.7). And more importantly perhaps, their weaker kick did not improve at all.

Experts put in fewer repetitions, but expend more effort and energy on each one.

Both expert and intermediate footballers spent the same total amount of time practicing, but experts logged fewer practice attempts than the intermediate group (43.9 vs 56.4 practice attempts).

However, results from the effort and enjoyment assessments suggest that the elite performers expended more effort on each practice attempt.

Specifically, the experts rated their practice sessions as being less enjoyable than the intermediate players (57.7% for the experts vs. 75.8% for the intermediates, where a rating of 65-70% equals riding on an exercise bike at a comfortable pace for 20 minutes).

The experts also rated their practice as requiring more mental effort than the intermediate players (57.9% vs. 30.7%, where higher scores=more effort).

The experts rated their practice as requiring more physical effort as well (58.8% vs. 46.8%, where higher scores=more effort).

This is likely due to the experts and intermediate players’ focus on weaker vs. stronger skills. The more repetitions the experts did of their weaker kick, the less enjoyable they rated their practice time to be. And the more repetitions the intermediate players did of their stronger kick, the easier and less effortful they found their practice to be.

Experts do more planning before each practice attempt.

Based on the voice recordings of their spoken-aloud thoughts during practice, the researchers found that experts did more thinking and planning before each practice attempt.

On average, the experts made almost twice as many statements per attempt than their intermediate counterparts (3.3 statements vs 1.7 statements). In particular, they made more “monitoring and planning” statements before each kick. In other words, they seemed to be able to better utilize feedback from the previous kick and form a clearer plan for what they were going to do in the subsequent kick.

Experts do more random practice.

Experts spent less time engaged in a “blocked” style of practice — spending 17% of their practice sessions in this format, as compared with 22% for the intermediate players. Note: For this study, blocked practice was defined as spending at least 60% of the practice attempts in one 5-minute block on just one kick, with only one switch between kicks per 5-minute block.

The expert footballers also spent more time engaged in “random” practice — with 26% of their practice being considered random, compared with the intermediates who at 3%, did almost none of this kind of practice. Note: For this study, random practice was defined as 4 or fewer consecutive trials before switching to the other kick. Or in other words, to be considered random practice, athletes could do no more than 4 kicks of the same kind in a row.

Learn at night and relearn in the morning

Friday, October 12th, 2018

We already know that (1) spacing practice out results in better learning and long-term retention than cramming it all together, and (2) sleep enhances learning and long-term retention.

A team of French researchers combined these two effects into a simple practice scheduling hack:

Two groups of 20 participants were tasked with learning the French translations of 16 Swahili words. All 40 participants went through the same exact training, but there was one teensy difference.

One group (“wake” group) had their first study session at 9am, and their relearning session at 9pm on the same day. The other group (“sleep” group) had their first study session at 9pm, and their relearning session at 9am the following morning.

[...]

The researchers kept track of how much practice the participants needed to get all 16 translations correct. The sleep group got to perfect recall in about half the time that it took the wake group (3.05 cycles through the list vs. 5.80 cycles). Plus, every single participant in the sleep group got a perfect score within 5 attempts, whereas 75% of the wake group needed more practice.

[...]

A full half a year later, the sleep group continued to out-remember the wake group (8.67 correct vs. 3.35).

We need to completely rewrite the textbooks on how to teach teachers

Monday, September 10th, 2018

We need to completely rewrite the textbooks on how to teach teachers:

That’s according to a new report just published by the National Council on Teacher Quality. The report describes a vast and severe failure of teacher-training courses and the textbooks that accompany them to convey evidence-based practices; while delivering unsupported anecdotal evidence and well-debunked myths in spades. The report is accompanied by a letter of support signed by an assortment of professors of psychology and learning sciences from universities around the world.

The report finds that out of 48 texts used in teacher-training programs none accurately described fundamental evidence-based teaching strategies comprehensively. Only 15 percent had more than a single page devoted to evidence-based practices; the remainder contained either zero or only a few sentences on methods that have been backed up by the decades of scientific findings that exist in the field of educational psychology.

Missing from these textbooks were detailed explanations of six core strategies that have been found to be backed by evidence, which every teacher should know and use. The strategies aren’t new; they were identified by the Institute of Education Sciences, the research arm of the U.S. Department of Education, as being the most effective techniques in all classrooms regardless of age or subject in guidance released in 2007.

Six Core Strategies Identified by Institute of Education Sciences

Incentives boost effort on IQ tests

Saturday, August 25th, 2018

Will intelligence test-taking performance increase if people are paid $75 to do well on the test? James Thompson takes a look:

This is an interesting question, because critics of intelligence testing have argued that some groups get low scores because they are not interested in the test, and can’t see the point of solving the problems. Perhaps so, although if you don’t get motivated by trying to solve problems that might be diagnostic in itself.

Gilles Gignac decided to have a look at this argument, seeing whether the offer of winning $75 Australian dollars boosted intelligence test scores in university students. For once, I am not too bothered by the subjects being university students, because they tend to have modest funds and healthy appetites.

[...]

The financial incentive was observed to impact test-taking effort statistically significantly. By contrast, no statistically significant effects were observed for the intelligence test performance scores.

[...]

One reason why test-taking motivation is correlated with intelligence test scores may be that bright people like solving problems. If they have to take a test, they look forward to it, knowing they usually do well, and are interested in finding out precisely how well they do. Less able students don’t like tests, and particularly get discouraged when they relate to difficult subjects.

Self-consciously tall young men who went on to study at Cambridge

Saturday, August 18th, 2018

While listening to the audiobook version of The Hitchhiker’s Guide to the Galaxy, I started thinking about the writing style, and I was immediately reminded of Monty Python’s Flying Circus.

I did a little digging, and it turns out that Douglas Adams was a self-consciously tall young man who went on to study at Cambridge — just like John Cleese, whose autobiography, So, Anyway…, I very much enjoyed, especially as an audiobook, with Cleese himself narrating.

Adams went on to be discovered by Graham Chapman — tall, Cambridge grad — and co-wrote a Monty Python sketch with him:

Adams is one of only two people other than the original Python members to get a writing credit (the other being Neil Innes).

Adams had two brief appearances in the fourth series of Monty Python’s Flying Circus. At the beginning of episode 42, “The Light Entertainment War”, Adams is in a surgeon’s mask (as Dr. Emile Koning, according to on-screen captions), pulling on gloves, while Michael Palin narrates a sketch that introduces one person after another but never gets started. At the beginning of episode 44, “Mr. Neutron”, Adams is dressed in a pepper-pot outfit and loads a missile onto a cart driven by Terry Jones, who is calling for scrap metal (“Any old iron…”). The two episodes were broadcast in November 1974.

Anyway, I found Adams’ style very, very English, and thus Stephen Fry‘s narration fit it very, very well. What’s that? Why, yes, Stephen Fry is conspicuously tall, isn’t he? I wonder where he went to… Oh! Cambridge! Fancy that.