NEW PUBLICATION: ESTIMATING THE PLAUSIBILITY OF LLM-GENERATED DEBUGGING FEEDBACK
A new paper by Brendan Toscano explores whether the quality of AI-generated debugging feedback for beginner programmers can be predicted automatically. While large language models can often fix simple coding errors, a correct fix doesn't always come with feedback that's actually useful to the student. Using an extended version of the IntroPyNUS dataset, the study analyzed over 10,000 records of student code, AI-generated repairs, hints, and explanations across five introductory Python problems to build models that estimate the plausibility of that feedback.
The best-performing model, a Linear SVM using five key features, achieved 92% accuracy in flagging plausible versus implausible feedback, while a Random Forest model reached over 86% accuracy in a more detailed multi-class scheme — notably without needing to know whether the repair itself was even correct.
More details can be found on the CILS Instagram page.