For fans of my thread from a couple weeks ago re: that anthropic paper about the effects of AI on learning, I collected it into a blog post with a slightly more professional tone. Should be a lot more shareable than a mastodon thread.
https://jenniferplusplus.com/reviewing-how-ai-impacts-skill-formation/
@jenniferplusplus I think this is an example of a big problem in the tech industry: they think they're better than anyone else. They showed no real understanding of what learning means or how it should be measured; they didn't bother to understand or link to relevant literature, and they filled the paper with some propaganda bullshit about the productivity advantages of AI. Instead of focusing on what seems important (the way you use the AI to build your solution and the predisposition to learn from it), they decided to focus on just one number to create engagement.
On the other hand, at least the statistics were good, and there was actually something interesting in it. Thank you for your analysis.
One additional learning is how we evaluate programmers. If these were junior programmers, most of the metrics employed in tech companies would say the "let AI do everything" group as the most productive, but they are not going to be good mid/senior programmers if promoted.
> "The control group kept running out of time because they struggled with syntax for exception handling, and for string formatting. They only stopped running out of time after the researchers added specific syntax guides for those 2 things to the project's instructions."
ahahahahahahahah hahahahah ahah ah
sigh
@chrisjrn I don't know why they thought it was a good idea to take away people's tools.
I mean, I have a theory, and it's that they didn't even think about the fact that these are people.
But still
@jenniferplusplus @chrisjrn yeah pretty wild that they can remove access to basic tooling and still make the (unproven) claim that AI was increasing productivity
@jenniferplusplus that is certainly borne out by your other observations.
I'm interested in why you think the 17% number is unusually large? I could imagine 17% if you allow for error propagation in subsequent tasks: people who know there's an error would tend to take steps that reduce it; people who don't know will propagate that error repeatedly without it being obvious
@chrisjrn It's the difference between a B grade and a failing one. It's a larger impact than chronic sleep deprivation (about 13%, based on a quick search). That doesn't make it impossible, but it's an extraordinary claim. And I doubted they had correspondingly extraordinary evidence
@jenniferplusplus (a) they clearly don't have extraodinary evidence, let alone (as you clearly point out) a valid experiment.
(b) the impact _is_ larger than a different known avoidable impediment, but I'm not sure the impacts can be reasonably compared without raising a lot of other questions; so mentioning that was weirdly distracting to me. 🤷
@chrisjrn You asked why I thought 17% was such a large gap. That's why. I'm not trying to compare the two effects, beyond setting context for what a small or large impact would be in this area.
And if you don't know what I'm talking about:
- ~3 weeks ago now, Anthropic published a paper and blog post on "How AI Impacts Skill Formation"
- the headline figure was that AI reduced test scores by 17% (lmao)
- however, the study has significant flaws that I think invalidate the comparison to the control group
- but there are also interesting patterns within the test group
I personally think my take is a fun read, and I'm completely unbiased, so I'm sure you'll like it too
https://jenniferplusplus.com/reviewing-how-ai-impacts-skill-formation/
