Stromberg et al 2026: "The Generative AI Learning Penalty: Evidence from Chinese Secondary Education" https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6868618 "Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design." 1/3
4
Replies
7
Boosts
The split between homework scores and exam scores is the whole story. Homework measures the artifact, exams measure the person, and generative AI improves the artifact directly.
What it displaces is retrieval practice, which is the part that builds durable recall. Two years is about how long it takes for that debt to come due.
Using it to check work you already attempted looks very different from using it to produce the work.
This is even simpler if you understand the difference between formative and summative assessment.
Formative assessment is intended to give feedback during learning. It tells you, your teachers, or both what you do and do not understand, so that you can focus learning on the things that you don’t understand. One of the neatest things I’ve seen in this space was an adaptive assessment using multiple choice questions where each distract or was intended to explore a different, specific, misunderstanding. At the end, it would give you a little summary of the things you misunderstood.
Summative assessment is the kind that you use for awarding grades. It tells you how well the course has worked in teaching each individual student. It doesn’t necessarily happen at the end of a complete course, you can have summative assessment after covering a set of key concepts or one specific skill and feed that into the final grade.
Homework is summative assessment. If you outsource this, all that you’re doing is giving you and your teachers less understanding about your progress. Once you get to the summative assessment, this disconnect will become visible.
@david_chisnall @aiappsapi @gvwilson
Sorry, I'm confused. Is homework summative or formative?
@SoftwareTheron @aiappsapi @gvwilson
It's normally formative. It's guiding your education as you learn things, not measuring whether you've learned things at the end.
It's sometimes summative, but usually because of bad course design.
