Henry Shevlin and I sat down to discuss a topic that is currently driving both of us slightly insane: the impact of AI on education.
On the one hand, the educational potential of AI is staggering. Modern large language models like ChatGPT offer incredible opportunities for 24/7 personal tutoring on any topic you might want to learn about, as well as many other benefits that would have seemed like science fiction only a few years ago. One of the really fun parts of this conversation was discussing how we personally use AI to enhance our learning, reading, and thinking.
On the other hand, AI has clearly blown up the logic of teaching and assessment across our educational institutions, which were not designed for a world in which students have access to machines that are much better at writing and many forms of problem-solving than they are.
And yet… there has been very little adaptation.
The most obvious example is that many universities still use take-home essays to assess students.
This is insane.
We discuss this and many other topics in this conversation, including:
* How should schools and colleges adapt to a world with LLMs?
* How AI might exacerbate certain inequalities.
* Whether AI-driven automation of knowledge work undermines the value of the skills that schools and colleges teach today.
* How LLMs might make people dumber.
Conspicuous Cognition is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Links
* John Burn-Murdoch, Financial Times, Have Humans Passed Peak Brain Power?
* James Walsh, New York Magazine, Everyone Is Cheating Their Way Through College
* Rose Horowitch, The Atlantic, Accommodation Nation
Transcript
Note: this transcript is AI-generated and may contain mistakes.
Dan Williams
Welcome everyone. I’m Dan Williams, and I’m back with my friend and co-conspirator, Henry Shevlin. Today we’re going to be talking about a topic which is close to both of our hearts as academics who have spent far too long in educational institutions: the impact of AI on education and learning in general, but also more specifically on the institutions—the schools and universities—that function to provide education.
There’s a fairly simple starting point for this episode, which is that the way we currently do education was obviously not built for a world in which students have access to these absolutely amazing writing and problem-solving machines twenty-four seven. And yet, for the most part, it seems like many educational institutions are just carrying on with business as usual.
On the one hand, the opportunities associated with AI are absolutely enormous. Every student has access twenty-four seven to a personal tutor that can provide tailored information, tailored feedback, tailored quizzes, flashcards, visualizations, diagrams, and so on. On the other hand, we’ve quietly blown up the logic of assessment and lots of the ways in which we traditionally educate students—most obviously with the fact that many institutions, universities specifically, still use take-home essays as a mode of assessment, which, at least in my view (and I’m interested to hear what Henry thinks), is absolutely insane.
So what we’re going to be talking about in this episode are a few general questions. Firstly, what’s the overall educational potential when it comes to AI, including outside of formal institutions? What are the actual effects that AI is having on students and on these institutions? How should schools and universities respond? And then most generally, should we think of AI as a kind of crisis—a sort of extinction-level threat for our current educational institutions—or as an opportunity, or as both?
So Henry, maybe we can start with an opening question: in your view, what is the educational potential of AI?
Henry Shevlin
I think the educational potential is insane. I almost think that if you were an alien species looking at Earth, looking at these things called LLMs, and asking why we developed these things in the first place—without having the history of it—you’d think, “There’s got to be some kind of educational tool.” If you’ve read Neal Stephenson’s The Diamond Age, you see a prophecy of something a little bit like an LLM as an educational tool there.
I think AI in general, but LLMs specifically, are just amazingly well suited to serve as tutors and to buttress learning. Probably one key concept to establish right out the gate, because I find it very useful: some listeners may be familiar with something called Bloom’s two sigma problem. This is the name of an educational finding from the 1980s associated with Benjamin Bloom, one of the most prominent educational psychologists of the 20th century, known for things like Bloom’s taxonomy of learning.
Basically, he did a mini meta-analysis looking at the impact of one-to-one tutoring compared to group tuition. He found that the impact of one-to-one tutoring on mastery and retention of material was two standard deviations, which is colossal—bigger than basically any other educational intervention we know of. Just for context, one of the most challenging and widely discussed educational achievement gaps in the US, the gap between black and white students, is roughly one standard deviation. So this is twice that size.
Now, worth flagging, there’s been a lot of controversy and deeper analysis of that initial paper by Bloom. For example, the students in the tutoring groups were mostly looking at students who had a two-week crammer course versus students who had been learning all year, so there were probably recency effects. He was only looking at two fairly small-scale studies. Other studies looking at the impact of private tutoring versus group tuition have found big effects, even if not quite two standard deviations. And this makes absolute intuitive sense—there’s a reason the rich and famous and powerful like to get private tutors for their kids.
Dan Williams
Yeah.
Henry Shevlin
There’s a reason why Philip II of Macedon got a tutor for Alexander. And more broadly, I think we can both attest as products of the Oxbridge system: one of the key features of Oxford and Cambridge is that they have one-to-one tutorials (or “supervisions,” as the tabs call them). This is a really powerful learning method.
So even if it’s not two standard deviations from private tuition, it’s a big effect. Now, people might be saying, “Hang on, that’s private tuition by humans. How do we know if LLMs can replicate the same kind of benefits?” It’s a very fair question. In principle, the idea is that if it’s just a matter of having someone deal with students’ individual learning needs, work through their specific problems, figure out exactly what they’re misunderstanding and where they need help, there’s no reason a sufficiently fine-tuned LLM couldn’t do that.
I think this is the reason Bloom called it the “two sigma problem”—it was assumed that obviously you can’t give every child in America or the UK a private tutor. But if LLMs can capture those goods, everyone could have access to a private LLM tutor.
That said, I think the counter-argument is that even if we take something like a two standard deviation effect size on learning and mastery at face value, there are things a human tutor brings to the table that an AI tutor couldn’t. Social motivation, for one. I don’t know about your view, but my view is that a huge proportion of education is about creating the right motivational scaffolds for learning. Sitting there talking to a chat window is a very different social experience from sitting with a brilliant young person who’s there to inspire you. Likewise, I think it’s far easier to alt-tab out of a ChatGPT tutor window and play some League of Legends instead, whereas if you’re sitting in a room with a slightly scary Oxford professor asking you questions, you can’t duck out of that so easily.
So I think there are various reasons why we probably shouldn’t expect LLM tutors to be as good as human private tutors. …