Intro. [Recording date: March 3, 2025.]
Russ Roberts: Today is March 3rd, 2025, and before introducing today’s guest, I want to share the results of our annual poll of your favorite episodes of last year, 2024. I want to thank everyone who voted.
Here are the Top Ten:
- And tied for 1st place, two episodes:
I want to, again, thank everyone for voting. We’ll have links to all of those episodes that made the Top 10, and I would remind listeners that we have a category called Favorites where you can listen to past years’ favorite episodes of listeners.
And, now for today’s guest, author and educational consultant, Daisy Christodoulou. She was last here in February of 2025, talking about Coase, the rules of the game, and the costs of perfection. Her substack is entitled: No More Marking. And, for listeners not from the United Kingdom, ‘marking’ is what we call ‘grading’ in the United States–and I don’t know what they call it anywhere else. But, our topic for today is feedback in education and the potential of AI [artificial intelligence] to provide feedback. Daisy, welcome back to EconTalk.
Daisy Christodoulou: Great to be here again, Russ.
Russ Roberts: We’re going to base this conversation on some recent essays of yours at your substack, No More Marking, and we’ll link to those for readers to check out. Am I correct–that is what people in Britain call grading, right?
Daisy Christodoulou: Yes, yes, you’re right. So, marking in a technical sense in the United Kingdom means the application of a number to–or a number or grade to a piece of work. So, yes; although I will say that a number of people in the United Kingdom don’t think that’s what it means, either. Maybe in that sense we’re misnamed, but it is a great name.
Russ Roberts: Let’s start by talking about feedback. You write that it can be a thermometer or a thermostat. Explain that phrase.
Daisy Christodoulou: Absolutely. So, when it comes to education, I think anyone working in education will talk about the importance of feedback. It’s a very popular topic, I think, with teachers and with students–the idea that you need to get some idea of how you’re doing to improve.
But feedback is obviously a very, very important concept beyond education; and the origins of that–not a term that’s used a lot now, but maybe cybernetics–the origins of these sort of control systems, the origins of information technology–a lot of the origins of those fields of study are in the concept of feedback: of how you change a system based on inputs.
And so, a really nice way of thinking about feedback in that more general sense is to think about a thermometer and a thermostat. So, a thermometer is a measurement tool: it will measure the temperature. A thermostat will change the temperature based on the thermometer’s reading.
And, I apply that analogy to educational feedback by saying that a teacher can give feedback where they can just kind of give a measurement, read a measurement of the student’s work. And that would be maybe to just give the grade–to say, ‘This is the standard it is at.’ But, when we talk about feedback, what we are hoping is that you will be able to give the student some kind of information that will move them closer to the goal state.
So, the aim of feedback in that technical sense in any sector is to think of something that will move you closer to your goal state.
So, in the case of the thermometer, we have the reading and thermostat. If the reading of the thermometer is too low–so if the thermometer comes back 18 degrees Celsius and you have a goal state you want to achieve of 20, the thermostat will kick on the heating and it will then stop when we get to 20. And likewise, if your thermometer comes back at 22 and your goal state is 20, you will then switch on–this thermostat will switch on the air conditioning and bring that temperature down.
The aim with educational feedback is: We have a goal state. I was an English teacher, what will our goal state be? Our goal state is we want our students to read fluently, infer insightfully, write coherently. How do we move them from where they are at the moment to that goal state?
We could also think about the goal state in terms of grades. If you want to say, ‘Well, they are a grade C and we want to move them to a grade A,’ so, what we have to do–in some way, shape or form–is provide feedback that will close the gap between the actual state where the student is at the moment and the goal state. And, as I say, this is something that is crucial in so many parts of information technology of so many parts of the world, but it’s of big interest to teachers, as well.
Russ Roberts: Of course, in education, if I say to you, ‘You’ve earned a C,’ some people say, ‘Great. Oh, that’s fantastic. I didn’t think I was going to get such good grade,’ and they’re done. But, others will say, ‘Oh, I need to improve.’
And, as you point out, the letter grade or the number grade–a 73–doesn’t tell you how to improve. It just says, ‘There’s room for improvement.’
And, in addition, it doesn’t really, by itself, tell you where the holes are. There could be things that you’re doing at a good level–at a quality level–and there could be things that you’re doing that are inadequate: but overall it’s a C. In which case it’s not very helpful to tell you where to start other than to try harder. And, trying harder, by the way, is not usually a very helpful bit of advice.
I want to make–as we go on, I want to make a distinction between two kinds of feedback that we might consider.
One is writing. And, a lot of what you wrote about in your essays is about giving feedback on written essays. And, the second I would say is we would call content knowledge, which could include something simple like, ‘Do you get the facts right?’ But, it also could include, ‘Are you able to apply the knowledge you’ve gained to other things?’ Which would be the highest level.
My favorite bad piece of feedback that I received as a teacher is when a student gave me a 1 on a scale of 1 to 5 and said, ‘This course is unfair. Professor Roberts expects us to apply the material to things we’ve never seen before.’ And, I used to read that, of course, at the beginning of every class–because that was the goal of the course.
But these two types of feedback–suggestions, we might call them, on an essay, or evaluation on an essay; and then, content and application–I think, are different. Do you agree with me?
Daisy Christodoulou: Yes, I can see what you’re saying. Do you mean–what you’re talking about–both types of feedback could be in the form of something that the teacher has written at the bottom of a student’s response? Yeah. Yeah.
So, there are of course all kinds of different categories that you could say that those comments would be in. So, I’m concerned a lot now in my day job with technical accuracy of students’ writing. So, I tend to think a lot about things to do with sentence structure and apostrophes and tense. So, that’s one category.
Another category is just getting the facts right in terms of what they’re writing about. So, an example I always give is there was a play that I used to teach quite a lot, called “An Inspector Calls,” when I was an English teacher; and a really common basic factual error be the students would mix up two characters. There was a character called Gerald and a character called Eric and they’d mix them up.
And it wasn’t the worst mistake in the world, and it didn’t mean they couldn’t be making some other really excellent points. But it was really confusing. And it would confuse them. And it was not great. So, that’s a really good example.
And then–I think then the other bit you’re talking about is then your ability to perhaps have those higher-order analysis and application to other ideas and perhaps those things which really elevate a piece of writing to the top levels. Are those the kind of things that you’re thinking about?
Russ Roberts: Well, I’m thinking about the history of the comments I’ve seen typically on my kids’ papers and on my own papers, and on–sometimes when I was giving feedback, the feedback I would give to a certain kind of answer and it would–and the example I would give would be: I would write in the margin–or I would see in the margin–‘Awkward,’ or, ‘Confusing.’ You circle a paragraph and say, ‘Confusing.’ And, the student–it took me a long time before I realized, and you know this quite well, you write about it–the student reads that and goes, ‘Okay, now what?’ And, that’s different than, ‘Gerald and Eric are two different characters,’ and you go, ‘Oh yeah, I got them confused. Okay, I get it now. I might have to reread the play and learn.’
Daisy Christodoulou: Yes. Although–yeah. Go on, carry on.
Russ Roberts: No, go ahead.
Daisy Christodoulou: So, what I would say is that–actually I get the distinction you’re making of those different categories, but I would actually say in all of those categories, I suppose my slightly heretical thought for somebody like you [?who?] spent a lot of time giving and receiving written feedback, is that: written feedback is not optimal for any of these categories, and that written feedback is not a particularly effective way of doing feedback.
Russ Roberts: Okay. So, let’s take essay, both grading and education. You’re involved in a number of projects, which I find utterly fascinating. We’ll get into it in a little more detail. We touched on your previous episode. But you’re involved in some projects of thinking about: How do we scale grading and feedback? But, what I’m having in mind here is that a student writes an essay and it’s mediocre. There are many, many, many ways it can be mediocre: It could be written in the wrong tone, it could have actual errors, it could lack sparkle–it could be–so it’s–we would call it flat.
And as a result, it gets a mediocre grade, a C+, a B-, whatever it is, depending on the system. And, the student gets that grade and thinks, ‘How do I do this better?’
So, the next level up is to say, ‘Well, I’ve told you where it’s better. When I wrote Awkward, that meant you should try to rewrite that.’ And so, I’m curious what–if written–I’m going to disagree with you in a minute about written feedback. I think there is some written feedback that could help. But, if I don’t give that student written feedback, what do I do for them?
Daisy Christodoulou: All right, great question. So, let’s just also think about the–quick thing before we get into the meat of this, which is the big question I do want to get into–but also just about age groups here.
So, I taught students aged 11 to 18. That’s where a lot of my expertise is. I now do a lot of work with 11 to 18, but also younger, so 5 to 11. But, I’ve also been a peer-reviewed author on an academic paper where I got written feedback from the peer reviewers. And, what I’m saying, I think, holds true, I would say, across the age ranges. So, there’s some of what I’m saying, I think, that holds true from ages five up to however old you are when you’re writing your peer-reviewed papers. Some of what I’m saying is truer for certain age groups than others. But, there is a central point here that I think is true across all of the age groups. So, however you are involved with giving and receiving written feedback, I think there are points I make here which hold true across those ranges.
And the central point I will make, which we touched on last time I spoke to you, is that prose is not optimized for action. Prose is not optimized to move you on to that next level.
And, the central philosophical, theoretical piece behind this is, again, someone I talked about in the last episode I did with you and who you have talked about since and you talk about a lot, which is Michael Polanyi.
So, Michael Polanyi’s entire theory of tacit knowledge is that there are things that we can do but we cannot tell. There are a whole bunch of stuff out there that we do have this intuitive real understanding and sense of, but it’s very hard to put that into words.
And I can give you some examples. Polanyi’s classic example is riding a bike. You can think about–also if you’re swimming, learning to float. If you read an entire book on how to ride a bike and then you put the book down and you go and try and ride the bike, is that time spent reading the book better or worse than some time spent actually with your mum or dad on the bike, trying to get your balance? Obviously the latter is more important.
And so, the central point I’m making that I think holds true across the range is that prose is not optimized for improvement.
And let’s carry on. I can keep going. I’ll keep going. I’ve got–
Russ Roberts: Go on a little bit more–
Daisy Christodoulou: Let me give you an example. And so, as I say–I would say Polanyi is the kind of guiding philosophical point behind this. But, let me give you some really practical examples.
So, there is a modern education writer called Dylan Wiliam, a big name in the field of education and assessment and feedback. And, he has given the most brilliant example of how this plays out in a middle school classroom. And, he says he was visiting a middle school classroom and the students had all been given quite detailed written feedback on a science investigation that they’d just completed. And one of the students had been given the feedback, ‘You need to make your scientific inquiries more systematic.’ And, Dylan Wiliam says to the student, ‘What do you understand by that? What are you going to do differently next time?’ And, the student says, ‘I don’t know. If I’d known how to be more systematic, I would have been so the first time.’
Russ Roberts: Yeah.
Daisy Christodoulou: Now, it’s obviously a flippant response, and you can imagine the kind of student who gives that kind of response. I’ve taught students like that. But, when I read that–
Russ Roberts: It’s true–
Daisy Christodoulou: paragraph, when I read that paragraph in Dylan Wiliam, I cringed. Because I had spent large chunks of my life as a teacher writing feedback like that–large chunks, particularly of my Sunday evenings. And not only had I done that: I had done that because it was recommended as best practice.
So, the best practice, when I was training as a teacher–and I’ve since realized that this is often seen as best practice in many other schools and countries that I now work in, including the United States–the best practice is seen to take the language of the mark scheme and give that language back to the student. That, that’s the most transparent way of getting them to understand how to do better.
And the one I was probably guilty of because I would copy it from the mark scheme was saying to students–well, writing down, ‘You need to infer more insightfully.’ Now this is what Dylan Wiliam would say is True But Useless. It is TBU. It’s like telling an unsuccessful comedian to tell funnier jokes.
Russ Roberts: Yeah.
Daisy Christodoulou: Yes. Okay? it’s what–yes, it’s true. But what do you do? What is the action that you take as a result of that?
So, if we go back to the thermometer/thermostat, this is not closing the gap.
So: This leads on to my next point, which is that the thing you need to do to close the gap and the action you need to take–I’ve just explained via Polanyi why prose is not optimized to do that. So, a lot of people will say to me, ‘Oh, well, you know, that teacher’s feedback, it wasn’t precise or specific enough. They needed to really give the specific action step.’ And, my point is, prose is not good for this. You need something else.
So, what is that something else? Dylan Wiliam, he talks about it as being a recipe for action. And, he also, I think quite rightly, talks about sport: that actually often in sport, we have a much better idea of the action steps you need to take to get from your current state to your goal state.
And the crucial insight here, which I think is not fully appreciated enough in education, is that the actions you need to take to get from your actual state to your goal state can often look very different to either the actual state or the goal state.
Let me give a concrete example. You’ve written your piece of writing: you’ve written your essay. And let’s say that you need to infer more insightfully–that is true. This student needs to infer more insightfully. What is inference? What do you need to do to be better at inferring?
Well, actually, that’s a huge thing. And what we know you need to do is be better at inferring: you need to have a wider vocabulary and more background knowledge. That’s actually what allows you to make inferences.
So, what I would say is, in the case of that student who needs to infer more insightfully, they–you could do a sequence of lessons with them on teaching them new vocabulary and perhaps some new prefixes and suffixes which will help them expand their vocabulary even further. That would be an activity which will help them improve to move from the actual state to the goal state.
But it doesn’t look like writing an essay. It doesn’t look like the piece of work that you submitted or the piece of work that you’re going to be doing.
In fact, in those lessons where you’re learning about new words, you may never hold a pen or write. But my argument is, that that will help you get to your goal state.
And if we think about this, Dylan Wiliam [?] talked about this in the context of sports, and I am, too.
And, a big metaphor that I used in my second book, Making Good Progress, I use a metaphor which I think people really understand–they get this more than they do when I talk about it in the context of academics–I use the metaphor of running a marathon.
So let’s imagine you want to learn to run a marathon and you’ve never run a marathon before; and you have three to six months to train. Do you go out and run a marathon in every single training practice session? No, you do not. Do you think that the only way you can measure your improvement from your actual state to your goal state is to run a marathon and then run another one and then see if you’ve got any faster? No, you do not.
What you do is you set up a training plan. You will increase your mileage gradually, and you will build into that training plan activities that do not involve running. That do not look like running.
So, one of the reasons people often struggle with marathon running is they need to build up their muscles to allow them to get to those heavy mileages. So, you have to do, often–and I speak from experience: I have run one marathon–you have to do strength work in the gym and things like yoga and have massages and do some strength and conditioning. You have to do all these things which don’t look like running, to make yourself a better runner.
And the same is true in academics: that there are a bunch of things you can do that will make you a better writer that do not look like writing. And, I would argue that is true not just for total novices, but also for students who are at a more advanced level. And, with the marathon analogy, it would hold true as well: that spending time in the gym is important for novice runners. You look at elite marathon runners, they will do that, too.
So, I would say that one of the big issues we have in education with the written feedback–not the only issue, but one of the big issues–is that the written feedback kind of encourages you to maybe just focus on redoing the work in some way when what you actually need to do is to step back and think what–the term I use in my book is ‘model of progression.’ You need to step back and think, ‘There is a model of progression here. There are some steps we need to take. What are these steps we need to take? Which steps are missing?’ And, some of these steps will not look like doing another piece of writing. They would involve something very different.
Russ Roberts: Well, I think that’s extremely interesting and I agree a lot of it, but not all of it. So, let me give you where I’m a little bit–I disagree.
One of the challenges–the recent sports analogies are so attractive is that we know a lot about sports and we think about them and we can often measure improvement. Which is much harder in the kind of academic environment we’re thinking about. So, an analogy that I would agree with you a hundred percent on–I’ve used it before, it’s–I love it because it was so insightful for me–is in baseball: If you’re having trouble hitting the ball well, a coach will often say, ‘Keep your eye on the ball.’ It’s really good advice. It’s also really hard to do. And, if you don’t explain why it’s hard to do–and I didn’t realize this until I coached baseball–the reason it’s hard to do is because when you’re hitting a baseball, your body is turning counterclockwise, but as the ball comes in, your head has to turn clockwise because it’s seeing the ball coming from the pitcher’s hand.
And, this could be cricket for those of you listening at home outside the United States or places where baseball is played. But, when the ball is coming towards you, as it gets closer and closer, if you’re going to watch it close to where it’s going to hit the bat, you have to turn your head in the opposite direction you’re turning your body. This is not natural. And, basically you have to train to do that. You have to practice that motion in some dimension–not literally probably, but being aware that is very useful. So, that’s good feedback, but it’s not enough; but it’s good feedback compared to the, ‘Keep your eye on the ball,’ which is useless feedback. So, that’s true.
Daisy Christodoulou: That’s a fantastic example. That’s a fantastic example.
Russ Roberts: Thank you. I’ve got one.
Daisy Christodoulou: Yeah. There’s similar things in football, in soccer.
Russ Roberts: Sure.
Daisy Christodoulou: So, I am terrible at understanding baseball; and it constantly pops up in a lot of academic literature and I just havepeople explain it to me. I’ve seen a couple of games. I’ve watched too much cricket. I’ll never get baseball. I didn’t play cricket, I watched a lot of cricket. I did play soccer and football. And, one of the bits of feedback you get then is people say, ‘Look, you’ve got to keep your head up. Keep your head up and look around the field.’ And, what happens with young players when they get the ball–and that is good advice–but what happens with young players is their first touch is poor, so whenever they get the ball, they’re looking down at their feet. Okay? They look down at their feet because they don’t have a great first touch and they’re still learning the mechanics of touching the ball. So, actually, if you say to someone learning to play football [soccer], if you say to them too early, ‘Keep your head up,’ well, they get the ball and they trip over the ball because they haven’t got their first touch sorted.
Russ Roberts: Or it’s very far away and it doesn’t matter that they’ve got their head up because their first touch was so bad. The analogy–
Daisy Christodoulou: Right. So, what you–
Russ Roberts: No, go ahead.
Daisy Christodoulou: What you need to do in that situation is not say, ‘Keep your head up.’ You need to say, like, ‘We’re going to do some passing drills.’
Russ Roberts: Yeah, exactly.
Daisy Christodoulou: And, again, to go back to my example about this being not just true for novices: What do all the elite clubs do? And all the–the really good ones who are pioneering and all the tiki-taka football and the new tactics? They do lots of passing drills–lots of small-sided passing drills, where you’re just trying to get that incredibly beautiful slick first touch that you don’t even have to think about. And, if you’ve got that and you can depend on that, sure, then you can get your head up and you can start looking around and scanning the pitch.
Russ Roberts: My other example, which I’ve joked about in print, is the advice, ‘Don’t bunch up,’ to young players. And, young players all surge around the ball, and the parents are screaming on the sidelines, ‘Don’t bunch up.’ And, every person there is saying, ‘Okay? Let’s stop for a minute. You run over–.’ It’s useless. It makes the parents feel better.
But, here’s where I don’t think it carries over into academic life. So, let me give you that. When I think about a badly written essay where inference is inadequate, where background knowledge is insufficient, where vocabulary is limited, often I think sufficient advice in many situations is: ‘Read more and write more.’ Practice. Practice doing this funny thing called thinking and this funny thing called writing. And, I think–I know I became a much better writer when I wrote more regularly. I don’t think I ever, ever wrote thoughtfully in the sense that I stepped back and said, ‘Now what can I do to make my writing better?’ I just wrote more. And I read more. Which also helps: they’re obviously related.
It’s just not obvious to me that–maybe it varies by age. But, it’s really hard–what you’re really getting at when you say, ‘Let’s do something more than tell people: Infer more thoughtfully,’ is, you got to teach them how to think. And, teaching people how to think, we’re not that good at. And, the process by which we do it here at Shalem College, which I adore and love and have become a total drinker of the Kool-Aid, is: Read difficult texts in the presence of thoughtful people and discuss deep questions. And, in that process, your brain will change in ways we don’t really–I call it a black box. I don’t really–we don’t understand that process very well, but I have no doubt that if it’s taught well, meaning not explained, but then exploration takes place, that your brain will get better.
Daisy Christodoulou: So, again, some of that I’d agree with, some I disagree with. I think some of it probably does–some of the differences I’m going to make here probably do apply to age. So, maybe what you’re saying is true for the age group and the cohort you’re teaching, whereas I’m thinking perhaps about younger and less able students.
Look, I love reading and writing and I would agree with a lot of what you said there in my own personal life in terms of you read more and you write more and you improve. And, I would also say in some ways that chimes a bit with the point I’m making, in that you can read things that don’t immediately seem directly linked to the thing it is that you are interested in, but the more you read and the more you know, the more you will be able to make links.
One of the things I’m very keen on and which I’ve written about in my first book is the value of knowledge and the value of committing knowledge to memory. Because, people often talk about creativity as being about making connections in unusual ways, but to make those connections, you have to know some stuff to begin with to connect between them.
So, my argument is that a large part of education really does involve remembering things. And, people don’t often like the word ‘memorization,’ but I think it is a crucial part of education. And I don’t think–I really dislike it when people talk about skills like creativity and communication in the abstract that actually they are tied to specific bodies of domain knowledge. And, I do think that the more knowledge you have and the more insights you have into different fields, the more connections and unusual connections you can make across those fields.
So, to that extent, I’d agree with you about reading more and writing more being important. I really don’t like it perhaps at primary and early secondary and even upper secondary when the solution to everything is read more or write more.
Russ Roberts: Fair enough.
Daisy Christodoulou: And, one of the things that teachers will say a lot is–they say, ‘A lot of students, they just need to build their writing stamina.’ There is a problem akin to maybe trying to move on to running the marathon too soon, if you try to get students to write too much too soon when they haven’t built the building blocks. Because, they will build bad habits, but they will embed those bad habits through repetition and through practice.
This is a real, real issue: that if you have a student who has poured their heart and soul into a piece of writing and they really want you to read it and give feedback and engage with it and they have so many technical errors that it is hard for you to read and engage it, that is a real problem.
And so, I do think there is an issue–a big issue–with running before you can walk, and that sometimes there is a pressure on teachers to–and I’ve seen this in the United States and the United Kingdom–to think that the way you show advancement is to get students to write more, to get students to do more.
So, when you said–the way this often transmits itself is learning by doing. And I want to nuance something I said before because I talked about Polanyi and I talked about, ‘Look, you can’t just read a book about riding a bike: you got to ride the bike.’ And so, people often say that’s learning by doing.
I want to really heavily nuance learning by doing. Okay? And this goes back to my marathon point. Let’s think about–this is the best analogy to have with it.
What I’ve said with a marathon is you do not want to go out and run a marathon in your first training session. You need to do some other activities which are going to help you build towards that. But, obviously, if one of those activities is just reading a book about the marathon, that might be some help, but it’s not enough. So, I do agree you have to learn by doing, but what I don’t agree is that the doing has to look like the end goal. Sometimes the doing has to look different.
Russ Roberts: Agreed.
Daisy Christodoulou: And for me, the goal of the teacher and the curriculum designer, and the assessor who is building the assessments into this to give us the feedback, is to design the right model of progression.
Now there can be wrong models of progression, because obviously what I’m saying here is you need to do stuff that doesn’t look like the end goal. But there’s a lot of things that don’t look like the end goal and some will be rubbish and won’t help you to the end goal.
So, let’s take the marathon example. If all you do before your marathon is read books about the marathon, that’s not the right model of progression. That’s not going to get you to run the 26 miles. And, I agree, that can be an issue because I think there’s–obviously, sometimes going out for a run is–it’s cold and raining: is there something I can do that doesn’t look like going out for a run? I’ll stay in and I’ll read the latest guide to the gear or whatever. So, that is an issue. [More to come, 30:38]