Strong claim from a recent article on using AI to evaluate teaching: "AI evaluation removes human bias in teaching assessment" https://www.sciencedirect.com/science/article/pii/S2666920X25000888
Sort of ironically, I asked AI to evaluate this claim. Gemini 2.5: this is an overstatement, but AI may serve as a "complementary component within a comprehensive, multi-method evaluation framework." https://docs.google.com/document/d/1V0PiO97GgCtLrqnhLlfo1PS7U5sD4kLbl04oPM7Z4aY/edit?usp=sharing
ChatGPT 4.1: The claim is "misleading in its absoluteness" https://drive.google.com/file/d/1p_69vnJnxYyGxGrEMD1BYwahbDGzqMJz/view?usp=sharing
#AIEd #EdDev #Teaching
Discussion
Loading...
Post
@dougholton It's interesting that they only compared AI evaluation to student evaluation, concluding "Our analysis shows that AI-based assessments strongly correlate with student perceptions, validating their role as an effective complementary evaluation tool."
How does that validate it as a complementary tool? The challenge is what standard to compare to. Ideally, you want to compare it to actual teaching effectiveness--but do we have that? Do we know how to measure teaching effectiveness?
@rspfau I'm just aware of research on traditional student evaluations of teaching (SETs) that show they have no correlation or even a negative correlation w/actual student learning:
* https://www.sciencedirect.com/science/article/abs/pii/S0191491X16300323?via%3Dihub
* https://www.insidehighered.com/news/2022/01/19/study-grade-satisfaction-major-factor-student-evals
* https://www.aaup.org/academe/issues/104-1/student-evaluations-teaching-are-not-valid#:~:text=The%20researchers%20also%20reviewed%20two%20rigorous%20studies,to%20perform%20better%20in%20the%20second%20class
They correlate more w/grade satisfaction: https://www.tandfonline.com/doi/full/10.1080/01973533.2020.1756817
And also there's a ton of research on how biased SETs are: https://docs.google.com/document/d/14JiF-fT--F3Qaefjv2jMRFRWUS8TaaT9JjbYke1fgxE/edit?usp=sharing
* https://www.sciencedirect.com/science/article/abs/pii/S0191491X16300323?via%3Dihub
* https://www.insidehighered.com/news/2022/01/19/study-grade-satisfaction-major-factor-student-evals
* https://www.aaup.org/academe/issues/104-1/student-evaluations-teaching-are-not-valid#:~:text=The%20researchers%20also%20reviewed%20two%20rigorous%20studies,to%20perform%20better%20in%20the%20second%20class
They correlate more w/grade satisfaction: https://www.tandfonline.com/doi/full/10.1080/01973533.2020.1756817
And also there's a ton of research on how biased SETs are: https://docs.google.com/document/d/14JiF-fT--F3Qaefjv2jMRFRWUS8TaaT9JjbYke1fgxE/edit?usp=sharing
bonfire.cafe
A space for Bonfire maintainers and contributors to communicate
Automatic federation enabled