AllenAI has introduced TutorMoments, a framework designed to evaluate whether advanced large language models (LLMs) can effectively balance the challenge of deciding when to assist a student and when to allow them to work independently. The framework is built on real one-on-one math tutoring sessions, with transcripts from U.S. students in grades 2-7. These transcripts were annotated by experienced math teachers, highlighting key moments where a tutor had to choose between scaffolding a problem or pushing the student to think more deeply. The framework then tests how well an LLM can mimic a tutor's decision-making in simulated sessions, offering a new way to assess AI tutoring behavior.
TutorMoments works by pausing a transcript at a key decision point and handing the session to an LLM, which then takes over as the tutor for five simulated turns. The model's actions are scored based on whether it scaffolded when needed, pushed for rigor when appropriate, and avoided over-scaffolding. The scoring pipeline is grounded in teacher-defined annotations, with a majority vote determining the ground truth for each moment. Preliminary results show that models perform better when given an evaluation-aware prompt that explicitly outlines the trade-off between scaffolding and pushing for rigor, indicating that default helpfulness is insufficient for effective tutoring.
The framework is part of AllenAI's commitment to open research, with the release of a dataset of de-identified tutoring transcripts, the code for running the replay pipeline, and model-generated replays of key moments for reproducibility. The goal is to provide educators and researchers with a clearer way to evaluate how AI tutors handle critical pedagogical decisions, ultimately helping build tutors that adapt to each student's needs rather than doing the work for them.
Source: huggingface
