This playbook provides a technical yet practice-oriented protocol for evaluating AI-generated feedback through rapid, repeatable cycles. It is intended for product teams, researchers, and learning organizations that need to understand not just whether an AI feedback system works, but how and where it can be improved.

Why rapid-cycle evaluation?

As AI tools become more embedded in learning environments, the quality of feedback becomes increasingly important. Unlike right-or-wrong outputs, feedback must be judged against more nuanced dimensions: whether it is instructionally aligned, actionable, cognitively and emotionally appropriate, and capable of helping students improve without removing the cognitive work of learning.

Traditional evaluation or research approaches are often too slow to keep pace with product development. This playbook outlines a faster alternative: a structured rapid-cycle evaluation process that uses authentic student work, human benchmark feedback, and rubric-based scoring to generate actionable signals within shorter timeframes, rather than over several months or an entire school year.

What to expect

The playbook provides a step-by-step process for:

• Collecting authentic student responses.
• Generating AI and human benchmark feedback.
• Structuring feedback data for evaluation.
• Developing and applying a rubric to assess feedback quality.
• Analyzing rubric scores to identify AI strengths and opportunities for improvement.
• Using findings to inform subsequent product and evaluation cycles.

The protocol draws on Leanlab Education's rapid-cycle evaluation work with Quill and is grounded in lessons learned from multiple rapid evaluation cycles. This project was funded by a grant from Learning Commons. Quill and Leanlab Education designed and carried out the evaluation, including collecting and de-identifying all student data. Learning Commons did not collect, access, or process any student personal information.

High-quality feedback is one of the most powerful supports for student learning, but also one of the most challenging instructional practices to evaluate consistently and well. Effective feedback does more than signal whether a response is right or wrong: it helps students understand how to improve, supports productive struggle, and keeps them engaged in the learning process.

These same challenges apply to AI-generated feedback. While AI systems can deliver feedback instantly and at scale, speed alone does not guarantee that the feedback is useful. In practice, low-quality AI feedback often falls into predictable traps: it gives away answers too quickly, provides repetitive or generic responses, focuses on low-priority issues, or fails to connect meaningfully to what the student actually wrote.

For teams building AI-powered learning tools, the challenge is not simply whether the system can generate feedback, but whether the feedback meaningfully supports learning. AI systems should be designed to improve over time, and meaningful improvement depends on timely, actionable evidence about how well the system is performing in practice. Rapid-cycle evaluation addresses this need by creating short, structured feedback loops that allow teams to:

• Collect authentic student responses in classroom contexts.
• Generate benchmark human feedback for comparison.
• Apply shared quality criteria consistently.
• Identify specific strengths and failure modes.
• Make targeted improvements between cycles.

The core rapid-cycle evaluation model brings together student responses, human and AI-generated feedback, and rubric-based scoring to produce evidence that can inform AI improvement. Each cycle follows this basic structure while allowing teams to adapt the process to their context.

Key components of a cycle

Each rapid cycle draws on three core datasets that together support comparison between AI-generated and human-written feedback:

• Student response dataset: Authentic student responses are collected from classroom tasks.
• Feedback dataset: For each student response, AI-generated feedback is collected from the product being evaluated, and teachers provide benchmark feedback to establish a human comparison set.
• Annotation dataset: Both human-written and AI-generated feedback are scored against a shared rubric to assess quality across key dimensions.

Cycle timing and adaptation

Each cycle typically takes place over a short, focused period (e.g., 4 to 8 weeks), but may be longer depending on a team's needs, and is intentionally designed to be flexible and adaptive. As teams learn from the data, they can iteratively refine feedback instructions, improve evaluation processes, and strengthen rubric definitions in real time.

Key outputs of a cycle

Each rapid cycle is designed to generate actionable findings that can directly inform improvements to AI feedback and strengthen the evaluation process. Key outputs include:

• AI feedback quality insights: Understand how AI feedback compares to human feedback across rubric criteria. Does it match, exceed, or fall short of educator-provided feedback?
• Rubric and process refinements: Clarify ambiguous criteria, strengthen scoring guidance, and improve annotator consistency based on what emerges during scoring. What needs to be refined to make annotation clearer, more consistent, and more reliable?
• Cycle improvement priorities: Translate findings or learnings into concrete next steps for strengthening future evaluation cycles, such as improving feedback writer guidance or increasing the diversity of student tasks and responses. What should be adjusted before the next cycle to strengthen the data and evaluation process?

Building towards scale

How can teams evaluate AI feedback at scale?

Seven steps, from defining scope through feeding findings back into the product. Open a step to see the full guidance and the applied example from our work with Quill.

1. 1 Define the evaluation scope

   Define the student task, instructional goals, and evaluation parameters that will guide the cycle.

   Prior to beginning any rapid-cycle evaluation of feedback, teams should align on the key elements that will shape both the data collected and how feedback quality is assessed. Establishing this shared understanding upfront helps ensure that findings are meaningful, actionable, and aligned with both instructional goals and evaluation objectives.

   While these elements are defined at the outset, rapid-cycle evaluation is inherently iterative. As teams learn from each cycle, all components of evaluation—from feedback instructions to rubric criteria to data collection processes—may be refined to improve clarity, consistency, and overall effectiveness.

   At minimum, teams should define and align on the following:

   • Subject and task scope: Clearly define the content area and the scope of the student task (e.g., social studies evidence-based writing at the sentence level), including any constraints or limitations that shape how feedback should be given.
   • Student task: Specify the activity that will generate student responses, including the prompt(s) and format.
   • Task parameters: Define the context and conditions of the task, such as grade level(s) and instructional setting(s), and any supports or constraints provided to students.
   • Feedback writers and annotators: Identify the appropriate level of experience or expertise required for those generating benchmark feedback and those evaluating feedback quality.
   • Feedback instructions: Establish initial guidance for feedback writers, including expectations for length and alignment to the student task.
   • Annotation criteria: Define the rubric used to assess feedback quality, ensuring it captures meaningful dimensions of quality and instructional effectiveness.
   • Annotation process: Outline how feedback will be collected, scored, and reviewed, including evaluator training, calibration, and quality assurance procedures.

   These elements provide the foundation for each evaluation cycle. As cycles progress, teams should expect to revisit and refine each component based on emerging data and insights, strengthening both the dataset and the evaluation process over time.

   Applied example: defining the evaluation setup

   This example draws on our first rapid-cycle evaluation and illustrates how these elements can be applied in practice.

   In this case, the subject, scope, and student task were shaped by the product context of Quill Reading for Evidence, an online literacy platform designed to help students strengthen their reading comprehension, writing, and language skills through text-based, interactive activities.

   Subject and task scope

   The evaluation centered on sentence-level, evidence-based writing tasks. Students read a nonfiction passage (~600–800 words) and responded to open-ended prompts by completing sentence-level claims using evidence from the text. For each student response, the system generated AI feedback, which was later evaluated alongside human-written feedback.

   Task parameters

   Data was collected from Grade 8 and 9 English Language Arts classrooms, including general and inclusion settings. Students completed the activity during regular instruction; teachers were asked to continue their business-as-usual instructional practices and provide support as they typically would (e.g., reading aloud text for struggling readers). Specific classroom supports were not standardized or systematically documented as part of the evaluation. In this example, all classrooms completed the same activity.

   Feedback writers and annotators

   To generate a human benchmark dataset, teachers with relevant experience in writing instruction in Grades 8–9 were recruited to provide feedback on each student response; these teachers did not implement Quill with students. In parallel, annotators were selected and trained to assess feedback quality using the rubric.

   Feedback instructions

   Feedback writers were instructed to provide 2–3 sentences of feedback per response without using AI. They were given the passage, prompts, and task context, but were not shown AI-generated feedback to avoid influencing their responses. Instructions were refined both within and across cycles to improve consistency and ensure feedback focused on helping students meet the writing task objective.

   Annotation criteria (rubric)

   An initial rubric for annotation was developed during evaluation planning, drawing on research, design principles, and domain expertise. The rubric established a working definition of feedback quality for this evaluation context. It also provided a foundation for annotation, with the expectation that criteria, examples, and scoring guidance would be refined as authentic student responses and feedback became available. (Further refinement and application of the rubric are discussed in Step 5.)

   Annotation process

   To support consistent evaluation, procedures were established for annotation training, calibration, and ongoing quality checks.

   Tip — start narrow

   In early cycles, consider focusing on a single activity or task as an operational test. This helps ensure core components—such as data collection, feedback instructions, and rubric application—are working as intended before expanding to broader contexts.

   Tip — start the rubric early

   Don't wait until all feedback is generated. Because rubric development is time-intensive, begin early and refine the rubric as real feedback examples become available.

   Key idea — design for iteration

   Treat all elements of the evaluation—tasks, instructions, rubric, and processes—as subject to refinement. Rapid-cycle evaluation improves not just AI feedback, but the evaluation approach itself.

2. 2 Collect student responses

   Gather authentic student work from real classroom contexts.

   Meaningful evaluation depends on meaningful inputs. In the context of AI feedback, this means collecting authentic student responses that reflect real learning conditions—whether in classrooms, tutoring settings, or independent practice—rather than synthetic or highly curated examples.

   To support this, teams should prioritize collecting student responses generated during typical use of the product or learning experience. These responses provide the most realistic context for assessing how well AI feedback supports student learning.

   During data collection, teams should consider the following:

   • Prioritize relevant work: Select responses from the stage(s) of interaction that best align with how the AI supports learning and what the team aims to improve. For example, teams may focus on first-attempt responses to evaluate initial feedback quality, or include revised work to examine how feedback supports improvement over time.
   • Capture sufficient diversity: Include a range of student responses across levels of accuracy or quality, and, where relevant, across prompt types or task conditions. This helps ensure the dataset captures a range of student performance and surfaces edge cases that challenge both feedback generation and annotation.
   • Maintain data quality: Remove or flag responses that are unusable for evaluation (e.g., nonsensical entries, system errors, or incomplete submissions).

   Applied example: student data collection

   Building on the context and task defined in Step 1, this example draws on the same first rapid-cycle evaluation to illustrate how student response data can be collected and prepared.

   Data collection

   Student responses were collected from authentic use of Quill Reading for Evidence activities in Grade 8 and 9 English language arts classrooms.1 All students completed the same activity ("Will AI-Powered Pets Robots Become Our Future Pets") during this cycle.

   Before collection, participating schools confirmed through MOUs that they had obtained the necessary consents for collecting and sharing student responses, consistent with applicable state and federal student privacy requirements.

   

   In this example, only students' first attempts (and the corresponding initial AI-generated feedback) were included.

   Response filtering

   Collected responses were reviewed to ensure they were suitable for feedback writing and evaluation. Responses were removed if they:

   • were nonsensical or did not reflect meaningful student input (e.g., random characters)
   • were associated with missing or invalid AI-generated feedback (e.g., system errors)

   Ensuring a usable dataset

   Quill's system flags student responses as optimal or non-optimal based on how well they meet the task criteria. All responses were retained in this example, but the distribution was reviewed to ensure there was not an overrepresentation of optimal responses.

   The final student dataset captured a range of student performance and included:

   146sentences for "because" claims

   145sentences for "but" claims

   Student data and privacy de-identification

   Student response data was extracted by a Quill researcher or data engineer and included Quill-assigned unique student IDs, school and classroom identifiers, Quill activity name, student response attempt number, student response text, and Quill-generated AI feedback. The extracted data did not include student names or other directly identifying student information; only Quill could link the unique student IDs to individual students. Quill and Leanlab researchers subsequently reviewed student responses for potentially identifying information within the text. Any identifiers retained in the shared dataset were newly generated by Leanlab for data organization and tracking purposes and could not be used to identify individual students.

   1. The evaluation was reviewed by Sterling IRB and determined to be exempt under Category 1 Exemption (DHHS). School MOUs confirmed that participating schools had obtained the necessary consents and established provisions for the ownership, licensing, and public release of anonymized data. Leanlab subsequently selected the Creative Commons Attribution 4.0 International (CC BY 4.0) license for the publicly released dataset.

   Key idea — input quality shapes evaluation quality

   The quality and structure of student response data directly shape what can be learned from evaluation. Collecting authentic and usable responses from clearly defined tasks is essential for producing meaningful insights about feedback quality.

3. 3 Generate human benchmark feedback

   Develop a comparison set of educator-written feedback.

   To evaluate the quality of AI-generated feedback, teams need a point of reference. Human-written feedback provides this benchmark, offering insight into how experienced educators respond to the same student work.

   This benchmark dataset supports:

   • Comparison between AI- and human-generated feedback at the criterion level
   • Capturing variation in how educators respond to student work

   During this phase, teams should consider the following:

   • Ensure independence of responses: Feedback writers should evaluate each student response without exposure to AI-generated feedback or system-generated labels, preserving the integrity of the benchmark.
   • Standardize inputs: Provide each writer with the same information in a consistent format. This helps ensure that differences in feedback reflect how writers respond, not differences in what was provided.
   • Capture usable variation: Expect and preserve variation in how educators respond. Differences in feedback are not noise: they provide a signal for understanding feedback quality and patterns across responses.

   The appropriate number of human benchmark feedback instances per student response will vary depending on your product and evaluation goals. Open-ended tasks may benefit from multiple benchmark responses to capture the range of instructional feedback, whereas more constrained tasks may require only one or two feedback responses. Teams should select a benchmark size that reflects the expected variability in how educators may respond to the student work.

   More feedback, more signal—when variation matters

   Multiple feedback writers can capture meaningful variation and strengthen the comparisons with AI-generated feedback. For more constrained tasks with less expected variation, however, a smaller set of responses may be sufficient.

   Applied example: human benchmark feedback

   Building on the context and student responses defined in Steps 1 and 2, this example draws on the same first rapid-cycle evaluation to illustrate how human benchmark feedback was generated.

   Preparing student responses for feedback

   Student responses were prepared to support consistent and unbiased feedback writing. Responses were:

   • Organized by claim prompt type (i.e., "because" or "but")
   • Stripped of system-generated labels (e.g., exemplar/non-exemplar flags)
   • Presented with only the student response and an anonymized ID for tracking and linking to other data

   Feedback guidance

   Teachers with relevant experience (as defined in Step 1) were recruited to provide feedback on each student response. Teachers were given a description of the Quill Reading for Evidence activity and were provided with the reading passage, the sentence-stem prompts that students completed, and sample student responses paired with non-AI feedback. They were instructed to provide 2–3 sentences of feedback per response without using AI tools.

   Resulting feedback dataset

   This process resulted in a dataset that combined human-written feedback with AI-generated feedback for each student response, enabling comparison across feedback sources. In this example:

   291student responses included

   2,328human-generated feedback responses (eight writers per student response)2

   291AI-generated feedback responses (one per student response)

   2. For our rapid-cycle evaluations, we front-loaded both student data collection and human feedback writing to maximize teacher availability and create a larger pool of data that could be drawn upon across evaluation cycles.

   Key idea — ensure benchmark integrity

   Human feedback only serves as a meaningful comparison point when it is generated under consistent and independent conditions. How feedback is collected shapes how AI performance can be interpreted.

4. 4 Structure and link feedback data

   Organize feedback into a unified dataset for annotation.

   Before the AI- and human-generated feedback collected in Step 3 can be evaluated using a rubric, it must be organized in a way that supports consistent scoring and comparison. Each feedback instance should be treated as a separate record and linked to the corresponding student response.

   During this phase, teams should consider the following:

   • Define the unit of analysis: Structure the dataset so that each row represents a single piece of feedback. This ensures that feedback can be independently scored and compared across sources.
   • Assign unique identifiers: Assign a unique ID (e.g., Feedback_ID; F001, F002, etc.) to each feedback instance so it can be tracked and scored.
   • Link feedback to student responses: Use a shared identifier (e.g., Response_ID) to associate each piece of feedback with the corresponding student response. Because multiple feedback entries may be linked to the same response, Response_IDs will repeat within the dataset, while each Feedback_ID remains unique. This allows multiple feedback entries (human and AI) to be compared for the same input (i.e., within-response analysis).
   • Label feedback source: Include a clear indicator (e.g., Feedback_Source) of whether feedback was generated by a human or an AI system to enable comparison across feedback sources.
   • Track feedback writers: Include a Writer_ID (or Teacher_ID) for human-generated feedback to identify which feedback writer produced each response. This supports deeper analysis of variation and consistency across writers, if needed.
   • Include additional variables as needed: Add additional fields relevant to your context (e.g., prompt type or task condition). These variables can support more nuanced analysis and interpretation of feedback quality.

   Applied example: structuring feedback data

   Building on the student responses and feedback generated in Steps 2 and 3, this example draws on the same first rapid-cycle evaluation to illustrate how feedback data were structured for annotation.

   Organizing feedback at the instance level

   Each piece of feedback (whether human- or AI-generated) was treated as a separate unit for annotation. A unique Feedback_ID was assigned to each feedback instance.

   Linking feedback to student responses

   Each feedback instance was linked to its corresponding student response using a shared Response_ID. This allowed multiple feedback responses (human and AI) to be associated with the same student response text. In this example, each student response had eight human-generated feedback instances and one AI-generated feedback instance.

   Response_ID R001 — student text

   "Some people think AI-powered pets are a good alternative to real pets because if someone cannot have a dog due to allergies, they can now have one risk free."

   Nine feedback instances linked to a single student response.

   Feedback_ID	Feedback_Text
   F001	Yes, people with allergies could enjoy an AI-powered pet. Can you think of any other details from the article that you could include in your claim?
   F002	Your statement here is clear and uses specific information from the article for support. To improve overall clarity, you might want to be more specific by explaining what is meant by the term "risk-free."
   F003	Thank you for your response. You stated, "If someone cannot have a dog due to allergies, they can now have one risk-free." This is a great start. However, it would help your response by adding additional reasons to support your claim.
   F004	This is a well-worded response to the prompt. Is an AI pet totally risk-free, or just free of allergy risk? It would be a good idea, with this reason, to give some personal details about a person with allergies.
   F005	Good job. An AI pet would not be an allergen risk; however, that may not make it completely risk-free.
   F006	What about other pets, such as cats? Would that work too? What do you mean by risk-free?
   F007	Excellent point, supported by the text. However, are they completely risk-free? Or is that risk related to the allergy issue? Be clear about which risk you mean.
   F008	Good job stating a specific example. You can still make your claim stronger by citing the paragraph where your evidence is mentioned.
   F009	Nice work! You used information from the text to explain why some people think AI-powered pets are a good alternative to real pets.

   Labeling feedback source

   Each feedback instance was labeled by source (e.g., Human or AI) to enable comparison across feedback sources.

   Response_IDStudent_TextFeedback_IDFeedback_TextFeedback_Source

   Tracking feedback writers

   Human-generated feedback was also associated with a Teacher_ID to identify which feedback writer produced each feedback response.

   Response_IDStudent_TextFeedback_IDFeedback_TextFeedback_SourceTeacher_ID

   Including additional variables

   Additional fields were included to support analysis and future scaling. For example, each student response was associated with an Activity_ID to identify the specific Quill Reading for Evidence activity it came from. While the activity could often be inferred from the response itself, including a structured identifier provided a cleaner, more reliable way to track and group responses across activities.

   This also made it easier to organize responses into broader topic categories and supported future evaluation cycles as additional activities (and topic areas) were introduced.

   Key idea — structure enables comparison

   Evaluation depends on how data is organized. Structuring feedback at the instance level and linking it to both source and student response ensures that comparisons between AI and human feedback are accurate and meaningful.

5. 5 Apply rubric-based evaluation

   Apply a structured rubric to assess feedback quality consistently.

   Once feedback data has been collected and structured, the next step is to evaluate its quality. This requires a clear and consistent framework for judging whether feedback supports student learning.

   Because feedback quality is inherently multidimensional, evaluation should move beyond overall impressions and focus on specific, observable characteristics of effective feedback.

   During this phase, teams should consider the following:

   • Refine and operationalize your definition of "high-quality feedback": Revisit the characteristics that define effective feedback in your context and ensure they are clearly translated into observable criteria that can be applied consistently during evaluation.
   • Refine or adapt the rubric as needed: Use authentic examples of student responses and feedback to test, clarify, and strengthen rubric criteria. Criteria should be clearly defined, observable, and specific enough to guide scoring and inform product iteration.
   • Provide task-specific scoring guidance: Pair the rubric with information about the student task, learning goals, and examples of successful responses so that feedback can be evaluated within the appropriate instructional context.
   • Train and calibrate annotators: Use practice examples, discussion, and calibration exercises to build shared interpretations of the rubric and improve scoring consistency.
   • Establish scoring rules: Define how criteria should be scored, how disagreements will be handled, and how results will be summarized and reported.
   • Monitor scoring quality: Periodically review agreement, revisit challenging cases, and recalibrate annotators to maintain consistency across the evaluation process.

   Tip #1 — refine in small batches

   Early in rubric refinement, apply the rubric to small sets of examples (e.g., 10–20) to quickly surface ambiguities, edge cases, and gaps in scoring guidance before scaling up annotation.

   Tip #2 — start early and build on existing work

   Rubric development and annotator calibration often take more time than expected. Whenever possible, begin rubric development early and leverage existing research-based frameworks as a starting point. Expect to refine criteria, examples, and scoring guidance through multiple rounds of review and calibration.

   Applied example: evaluating feedback quality

   Building on the structured dataset described in Step 4, this example draws on our first rapid-cycle evaluation to illustrate how feedback quality was defined, measured, and refined over time.

   Rubric development

   An early working rubric had been developed during evaluation planning, drawing on research, instructional expertise, and product goals. Rather than treating rubric development as a one-time activity, the rubric was refined as authentic examples of student responses and feedback became available. The team tested the rubric across multiple small batches before formal scoring began, revising definitions, scoring guidance, and examples to improve clarity, reduce ambiguity, and support more consistent scoring across annotators.

   Rubric framework

   The evaluation rubric was grounded in a productive coaching framework that defines feedback as a form of coaching intended to support student learning and revision. This framework informed the development and refinement of a set of observable criteria that could be applied consistently across feedback instances. (See the appendix for the full productive coaching framework, rubric criteria, and scoring guidance.)

   The resulting rubric evaluated feedback across eight independently scored criteria using a binary scale (Meets / Does Not Meet Criterion):

   Criterion	Focus
   Appropriate Feedback Decision	Does the feedback correctly determine whether revision is needed?
   Task-Aligned Revision	Does the feedback support the goals of the task?
   Not Answer Giving	Does the feedback avoid simply providing the answer?
   Actionable Revision	Does the feedback provide a clear next step?
   Manageable Feedback	Is the feedback appropriately scoped and digestible?
   Anchored in Student Response	Is the feedback grounded in what the student actually wrote?
   Acknowledges Strength	Does the feedback recognize what the student did well when appropriate?
   Appropriate Emotional Pitch	Is the tone supportive and respectful?

   Annotation training and calibration

   Annotators (N=8) completed initial training before formal scoring began, but training and calibration continued throughout the evaluation process. Initial training included:

   • Reviewing the rubric framework and scoring guidance
   • Discussing annotated examples
   • Completing practice scoring exercises
   • Participating in calibration discussions

   Early training activities were intentionally structured so that annotators encountered different rubric dimensions and feedback examples in a non-sequential order. This encouraged annotators to evaluate each criterion independently and reduced the influence of assumptions about how a feedback response might be scored on other dimensions. This approach was particularly useful during initial calibration as annotators learned the rubric.

   As scoring progressed, annotators regularly reviewed difficult cases and discussed disagreements. These conversations served as both reliability checks and ongoing learning opportunities, helping annotators develop shared interpretations of the rubric and apply criteria more consistently over time.

   Scoring approach

   Each piece of feedback was evaluated across all rubric criteria. To support efficient and reliable scoring, annotators were assigned responsibility for only a subset of the rubric. Specifically, each annotator pair focused on two criteria throughout the evaluation process. This approach allowed annotators to develop deeper familiarity with criterion definitions, examples, and edge cases while reducing cognitive load. It also reinforced criterion-level evaluation by encouraging annotators to assess each dimension independently rather than form holistic judgments about overall feedback quality.

   The scoring approach emphasized three principles:

   • Score each criterion independently
   • Score only what is present in the feedback
   • Avoid assigning overall quality ratings

   These design choices enabled analysis at the criterion level and made it easier to identify specific areas where AI-generated feedback aligned with or diverged from human-generated feedback.

   Reliability and ongoing refinement

   Annotation quality was monitored throughout the evaluation process. Calibration sessions and reliability checks, including percent agreement on a subset of feedback scored independently by two annotators, were used to identify:

   • Unclear rubric language
   • Inconsistent interpretations
   • Recurring edge cases

   Insights from these discussions informed ongoing revisions to rubric definitions, examples, and annotation guidance. They also highlighted which criteria were most difficult to interpret and apply consistently, helping to prioritize areas for refinement.

   Design decision — why we used binary scoring

   Given the short, targeted nature of sentence-level feedback, we selected a coarse-grained binary scale. This approach balanced specificity with efficiency: more detailed scales may capture additional nuance, but they also increase scoring time, effort, and cost. For longer or more complex feedback outputs, a finer-grained scale may be more appropriate.

   Tip — choosing an annotation platform

   Rubric-based annotation can be conducted using existing annotation or scoring platforms, or teams can create custom tools tailored to their evaluation workflow. For this project, our team developed an annotation platform to streamline the scoring process called Converge, available as an open-source resource on GitHub. Whatever approach is used, the platform should support an efficient, consistent annotation workflow and make it easy to export scoring data for analysis.

   Key idea — make quality observable

   A rubric does more than score feedback—it creates a shared definition of quality. Clear criteria, consistent scoring practices, and ongoing calibration help transform subjective impressions into reliable evaluation data.

6. 6 Analyze and interpret scores

   Analyze scores to identify strengths, weaknesses, and opportunities for AI improvement.

   Once annotation is complete, the focus shifts from scoring feedback to interpreting what the results reveal. The purpose of this step is to translate rubric scores into actionable evidence about AI performance by comparing AI- and human-generated feedback, identifying strengths and weaknesses, and prioritizing opportunities for improvement. The goal is not simply to summarize scores, but to generate insights that inform future product development and evaluation cycles.

   During this phase, teams should consider the following:

   • Compare AI- and human-generated feedback: Examine performance across evaluation criteria to identify where AI aligns with educator expectations and where meaningful differences remain. Depending on the evaluation goals and rubric, comparisons may focus on overall performance, individual criteria, or relevant subsets of responses.
   • Identify patterns across criteria: Look for dimensions where AI-generated feedback consistently performs well and areas where performance is weaker or less reliable.
   • Look beyond averages: Examine score distributions, consistency, and recurring failure patterns. Similar average scores may mask important differences in how reliably AI and human feedback meet quality expectations.
   • Explore meaningful subgroups: When relevant, examine whether performance varies across prompts, activities, student response types, grade levels, or other contextual factors.
   • Prioritize actionable findings: Focus on patterns that are instructionally important, frequent, and feasible to address through prompt revisions, model adjustments, product changes, or future evaluation cycles.

   Tip — look for patterns, not perfection

   Neither AI systems nor human feedback writers will perform perfectly. Prioritize recurring patterns that meaningfully affect feedback quality rather than isolated examples.

   Applied example: using rubric scores to identify AI strengths and improvement opportunities

   This example draws on the dataset developed during our first rapid cycle and illustrates how rubric scores can be organized and analyzed to examine AI performance. It outlines example analyses and interpretation strategies that teams may use to identify recurring strengths, meaningful gaps, and priorities for improvement.

   Example annotated dataset

   Following scoring, the annotated dataset contained one record for each feedback instance, linked to the corresponding student response, feedback source, contextual variables, and rubric scores. This analysis-ready dataset served as the foundation for comparing AI-generated and human benchmark feedback.

   The table below presents an abbreviated example of the dataset, featuring scores for 18 feedback instances linked to two student responses. To conserve space, the student response and feedback text are omitted, but each feedback record remains linked to its corresponding student response through the Response_ID and Feedback_ID fields.

   Table scrolls sideways on small screens.

   Note. 1 = Meets Criterion; 0 = Does Not Meet Criterion. Criterion columns follow the order in which the rubric criteria are presented in the playbook: Criterion 1 = Appropriate Feedback Decision; 2 = Task-Aligned Revision; 3 = Not Answer Giving; 4 = Actionable Revision; 5 = Manageable Feedback; 6 = Anchored in Student Response; 7 = Acknowledges Strength; 8 = Appropriate Emotional Pitch.

   Response ID	Feedback ID	Feedback Source	Teacher ID	C1	C2	C3	C4	C5	C6	C7	C8
   R010	F082	Human	1	1	1	1	1	1	1	0	0
   R010	F083	Human	2	1	1	1	1	0	1	0	0
   R010	F084	Human	3	1	1	1	1	1	1	0	0
   R010	F085	Human	4	1	1	1	1	1	1	0	0
   R010	F086	Human	5	1	1	1	1	1	1	0	0
   R010	F087	Human	6	1	1	1	1	1	1	0	1
   R010	F088	Human	7	1	1	1	1	1	1	0	0
   R010	F089	Human	8	1	1	1	1	1	1	0	0
   R010	F090	AI	–	1	1	1	1	1	1	1	1
   R090	F802	Human	1	1	0	1	1	0	1	1	1
   R090	F803	Human	2	1	1	1	1	1	1	0	1
   R090	F804	Human	3	1	1	1	1	1	0	0	1
   R090	F805	Human	4	1	1	1	1	1	1	0	1
   R090	F806	Human	5	1	1	1	1	1	1	0	1
   R090	F807	Human	6	1	1	1	1	1	1	0	1
   R090	F808	Human	7	1	1	1	1	0	1	0	1
   R090	F809	Human	8	1	1	1	1	1	1	1	1
   R090	F810	AI	–	1	1	0	1	1	1	0	0

   Example analyses

   The table above illustrates how an annotated dataset can be used to compare AI-generated and human benchmark feedback across multiple evaluation criteria. The examples shown draw on a small subset of scored feedback instances from two student responses and are intended solely to demonstrate the analysis process. Because the subset is limited in size, any observed patterns should not be interpreted as overall findings from the evaluation. Instead, teams should examine whether similar patterns emerge across the full evaluation dataset before drawing conclusions.

   Even within this small subset, several observations illustrate the types of questions teams might explore:

   • Look beyond overall agreement: For R010, AI and human feedback aligned across Criteria 1–6, with differences emerging only for Acknowledges Strength and Appropriate Emotional Pitch.
   • Compare performance by criterion: For R090, AI and human feedback broadly agreed that revision was needed, but the AI received "Does Not Meet" for Not Answer Giving, Acknowledges Strength, and Appropriate Emotional Pitch. This illustrates how feedback can reach a similar instructional decision while differing in how it supports and guides the student.

   Across the full dataset, teams could calculate mean scores for AI-generated and human benchmark feedback on each criterion. Because the criteria in this example are scored as 0 or 1, each mean can be interpreted as the proportion of feedback instances that met that criterion. For example, a mean of 0.80 would indicate that 80% of feedback instances received a score of 1. Comparing these criterion-level means can help determine whether patterns observed within individual response sets persist across the broader dataset and identify dimensions where AI-generated feedback most consistently aligns with or differs from the human benchmark.

   Complementary analyses across the full dataset may include:

   • Criterion-level means by source. Compare the proportion of AI and human feedback that meet each criterion.
   • AI–human gaps by criterion. Calculate the difference between AI and human mean scores to identify the largest areas of alignment or divergence.
   • Response-level patterns. When multiple feedback instances are available for each student response, examine the proportion that meet each criterion and whether AI feedback follows similar patterns across different levels of human performance.

   Together, these analyses help move evaluation beyond individual examples toward more systematic evidence about AI performance. By examining rubric scores across feedback instances and evaluation criteria, teams can identify recurring patterns, distinguish isolated observations from consistent trends, and prioritize improvements for future iterations of the AI system.

   Key idea

   Analyzing rubric scores across feedback instances and criteria helps teams identify where AI-generated feedback aligns with human benchmarks, where meaningful differences remain, and what to improve next.

7. 7 Iterate and improve

   Use evaluation findings to refine the AI system and guide future evaluation cycles.

   Rapid-cycle evaluation is designed to support continuous improvement rather than a one-time assessment of AI performance. After analyzing rubric scores and identifying recurring patterns, teams should translate those findings into targeted improvements to the AI system. The revised system can then be reevaluated using the same evaluation process, allowing teams to determine whether changes improved feedback quality and to identify new opportunities for refinement. Repeating this cycle helps ensure that system development remains grounded in evidence rather than anecdotal observations.

   During this phase, teams should consider the following:

   • Which rubric criteria consistently showed lower performance?
   • Which findings are most actionable and likely to improve instructional quality? What changes to prompts, system instructions, retrieval strategies, or other components could address the identified weaknesses?
   • How will improvements be evaluated during the next rapid-cycle evaluation?

   Tip — prioritize high-impact changes

   Focus on a small number of high-impact improvements between evaluation cycles. Limiting the scope of revisions makes it easier to determine whether subsequent improvements are attributable to the changes that were made.

   From evidence to action

   Evaluation findings should be translated into concrete improvement priorities. For example, analyses may reveal that AI-generated feedback consistently provides overly directive responses, suggesting the need to revise prompting or system instructions to better encourage scaffolded guidance. Alternatively, findings may indicate that feedback rarely acknowledges student strengths, highlighting an opportunity to strengthen positive reinforcement while maintaining instructional focus.

   After implementing improvements, teams should repeat the evaluation process using the same rubric and procedures whenever possible. Consistent evaluation methods allow results from successive rapid cycles to be compared more meaningfully and help determine whether revisions have improved AI-generated feedback over time.

   Key idea

   Rapid-cycle evaluation is most valuable when findings are used to improve the AI system and inform subsequent rounds of evaluation, creating an ongoing cycle of evidence-based refinement.

This playbook outlines a repeatable process for evaluating AI-generated instructional feedback, from defining the evaluation setup through translating findings into system improvements. Across each step, the goal is to produce evidence that is sufficiently structured, transparent, and instructionally grounded to support meaningful comparison and refinement.

Over time, the annotated datasets produced through this process may also support the development of automated evaluators trained to assess AI outputs against trusted educational rubrics. Such tools could make it possible to evaluate larger volumes of feedback more quickly and support more frequent rapid-cycle testing. However, automated evaluators should be treated as an extension of, rather than a substitute for, human judgment. Their usefulness will depend on the quality of the underlying rubric, the strength of the human-scored benchmark data, and ongoing checks that automated scores remain valid across activities, student responses, and system versions.

Used together, rigorous human evaluation and carefully validated automated evaluation may allow teams to move from occasional reviews toward more continuous, evidence-informed improvement of AI-generated instructional feedback.

The productive coaching framework behind the rubric, the assumptions it rests on, and the rules we use when scoring.

1. Productive coaching framework

   Productive coaching framework: rapid-cycle AI evaluation framework

   The Productive Coaching Framework was developed as part of our rapid-cycle evaluation of AI-generated instructional feedback for student responses from Quill Reading for Evidence activities. The framework articulates the underlying principles used to define productive feedback and provides general guidance for applying those principles during evaluation. It served as the conceptual framework for the project's feedback quality rubric, which translated these principles into observable criteria for scoring AI-generated and human benchmark feedback.

   Rubric introduction

   This rubric provides a framework to evaluate written feedback given to students, focusing on qualities of feedback that support productive coaching. The ultimate goal of productive coaching is to help students enter a state of productive struggle, the cognitively challenging but emotionally manageable process through which learners develop understanding by grappling with problems just beyond their current capability (Warshauer, 2015; Young et al., 2024). The primary focus of this rubric is not on the student state itself, but on the role of the feedback giver in creating the conditions for that state. For that reason, we use the term productive coaching to emphasize that it is the responsibility of the teacher, reviewer, or AI system to provide feedback in a way that invites challenge while maintaining clarity, support, and momentum.

   The rubric is designed for teachers seeking to refine their formative feedback practice, researchers studying feedback quality and effectiveness, and ed tech developers building AI-powered feedback systems needing a consistent way to judge whether feedback supports productive coaching. Effective feedback offers strategic hints, questions, or partial information that guide students toward solutions without eliminating the intellectual work of problem-solving. It balances support (preventing helplessness) with challenge (avoiding passive reception of answers). The rubric is binary and addresses whether specific criteria have been met or not. The emphasis is on describing the presence or absence of each feedback element rather than judging with value-laden terms (e.g., "bad" or "good").

   Criteria are rated independently to support targeted improvement, with no composite quality score calculated. However, users should note that criteria may interact in practice. For example, high actionability often depends on clear acknowledgment of student work; alignment with learning goals enables more focused, high-impact feedback; and emotional responsiveness may moderate how students receive challenging feedback. Product development teams should track these potential interdependencies over time to understand feedback effectiveness holistically.

   Designed for rapid-cycle product development

   The rubric is built for rapid cycle product development, where teams need feedback quality signals that are fast to generate, easy to compare across versions, and specific enough to drive iteration. The goal is not to produce a single quality score, but to identify which feedback characteristics are consistently present and which are missing so teams can make targeted improvements. Note that the defining characteristics for some rubric criteria (e.g., task aligned revision) will need to be customized to your product.

2. Key ideas behind the rubric

   At the heart of this feedback rubric are a few core beliefs about how students learn. We designed the rubric as a tool for growth and iteration to improve the quality of feedback over time, whether that feedback comes from a teacher or from AI. Every assumption below reflects the idea that feedback should ultimately help students move forward in their learning.

   Applicable to teachers and products alike

   We recognize that feedback can come from a human teacher or a digital product (like an AI tutor or an educational app). Each has its own strengths: a teacher might excel at empathizing and reading the situation, while a product might provide instant responses and consistent structure. Our underlying assumption is that good feedback is good feedback, no matter the source. This rubric is designed to work across both scenarios. The criteria focus on the qualities of the feedback itself—things like relevance, clarity, accuracy, and constructiveness—which apply to anyone or anything giving feedback. By doing so, the rubric encourages everyone (humans and AI systems alike) to play to their strengths and shore up their weaknesses. The ultimate goal is a high standard of feedback that supports student learning, whether that feedback is typed by a teacher in a classroom or generated by an algorithm in a learning app.

   Feedback is for learning, not grading

   We assume that feedback's primary role is to support learning, not to serve as a mini-grading system. Good feedback opens up a conversation and gives the student useful information about what they did and how they might improve. This rubric values comments that coach and guide students, rather than comments that judge or score their performance. This distinction matters because: 1) evaluative feedback (grades, scores) can trigger responses that reduce engagement (Lipnevich & Smith, 2009); 2) learning-focused feedback positions mistakes as opportunities rather than failures (Winstone & Carless, 2019; Dweck, 2006; Bjork et al., 2013); and 3) students process formative vs. summative feedback differently, with formative feedback more likely to prompt revision and improvement (Peterson & Irving, 2008).

   Productive struggle requires help, not answers

   A key principle behind our criteria is that learning happens when students grapple with challenges, but with a safety net. Learning occurs in the Zone of Proximal Development (Vygotsky, 1978), the space between what students can do independently and what they can achieve with guidance. Effective feedback supports productive struggle by giving students enough help to stay engaged and make progress, while still preserving the cognitive work they need to do themselves. We believe feedback should move students forward without removing the challenge of thinking, revising, and making meaning. Our rubric is grounded in this balance between support and challenge.

   Feedback can open a dialogue, not just deliver a verdict

   We assume effective feedback often sparks a two-way conversation with the student, rather than ending as a one-sided critique. The best feedback invites the student to respond or reflect, positioning the teacher/AI as a coach instead of a judge. By treating feedback as the start of a dialogue, we encourage students to think deeper and take an active role in their learning. Our rubric would value comments that include prompts or open-ended questions, since they keep the learning conversation going. For example, forms of dialogic feedback may include:

   • Questions that prompt student thinking: "What strategy could help you check this answer?"
   • Invitations to respond: "Try revising this paragraph and let's discuss your changes."
   • Requests for self-assessment: "How do you think your argument compares to the rubric criteria?"
3. How these ideas are put into action

   Feedback should begin with the right decision about whether revision is needed

   We assume that effective feedback depends first on accurately determining whether the student's response actually requires revision. Before feedback can be actionable, task aligned, or manageable, it has to correctly identify whether the student has met the goal of the task. If a response does not yet meet the goal, such as when it does not successfully complete a claim with relevant evidence, then feedback should direct the student to revise. If the response does meet the goal, feedback should indicate that the student can move on, even if optional suggestions for improvement are also offered. This assumption matters because feedback that misidentifies the student's status can create confusion and unnecessary work. For example, telling a student "Great job, you can move on" when the response is actually incomplete can allow misunderstandings to persist, while telling a student "This needs revision" when the response already meets the goal can undermine confidence and waste effort. Our rubric therefore treats accurate identification of whether revision is needed as a foundational feature of productive coaching.

   Feedback should be task aligned

   We assume that feedback is most effective when it's clearly connected to the task's learning objectives or success criteria. In practice, that means comments should reference the skills or content the student was supposed to demonstrate. By anchoring feedback to the original goal, we make sure it's relevant and not off-topic. This addition builds on our existing assumption about accuracy. It emphasizes that being "accurate" isn't just about correct facts, but also about addressing the right skills or content. If learning goals exist and are clear, feedback should explicitly connect to them. If learning goals are unclear or missing, this should be flagged as a context problem that limits feedback quality.

   Not answer giving

   We assume that feedback is most effective when it preserves the student's role as the thinker and writer. High quality feedback may offer a hint, point students back to the text, ask for more specificity, or identify what kind of evidence is missing. However, it should not complete the cognitive work on the student's behalf. If the feedback supplies the exact evidence to use or effectively rewrites the response for the student, it removes the opportunity for productive struggle. Our rubric therefore distinguishes between guidance that supports revision and answer giving that bypasses it.

   Actionability depends on clear next steps

   We assume that feedback only helps if the student can figure out what to do with it. Every piece of feedback should leave the student with a clear idea of what to do next. Sometimes that means the feedback gives a direct suggestion, such as, "Try adding a quote from the text to support your point about X." Other times it poses a question that gets the student thinking, such as, "How could you describe the main character's motive using evidence from the story?" In all cases, the student should not be left guessing what the feedback means. If a comment is too vague or abstract, it is not actionable. Our rubric is therefore built on the idea that a clear, usable next step is a hallmark of high quality feedback.

   Manageable

   We assume that feedback only works if the student can realistically process and use it. More feedback is not always better. When a comment tries to address too many issues at once, it can create confusion, blur priorities, and make revision feel overwhelming. Manageable feedback is focused in both amount and scope. It highlights one or two of the most important next steps so that the student can see what to do first and act on it in a single revision attempt. This is especially important in rapid cycle contexts, where feedback is meant to support immediate revision rather than extended conferencing. At the same time, manageable does not simply mean short. A brief comment can still be unhelpful if it is vague, and a somewhat longer comment can still be manageable if it is focused, clearly prioritized, and realistic for the student to act on. In our rubric, the key question is whether the learner could reasonably process and apply the feedback.

   Feedback should be anchored in the student's response

   Feedback should use student ideas as the foundation for next steps. The best feedback treats the student's ideas as the starting point, then expands or challenges them. If a student made an interesting observation, the feedback might extend that thinking or ask a question about it. If the student took a wrong turn, the feedback references where and how, guiding them back on track. The assumption here is that a feedback conversation should feel like it's rooted in the student's own work, not something more generic. In other words, feedback is most meaningful when it starts from the student's current understanding (even if incorrect), extends or redirects student thinking rather than imposing external logic, and treats errors as windows into student reasoning, not just mistakes to correct. Our rubric thus favors comments that tie new suggestions to the student's existing ideas, so the guidance is relevant and meaningful.

   Acknowledges strengths

   We assume that part of good feedback is genuine acknowledgment of what the student has done so far. This isn't about generic praise like "Good job!" Rather, it's about specific, grounded acknowledgment that shows the teacher or system truly paid attention. For example, a comment might point out a particular strength in a student's reasoning or refer to a specific moment in their writing. By engaging with the real details of a student's work, feedback validates their effort and makes the conversation personal. Further, it increases cognitive clarity so that students learn what concrete actions to repeat or revise; provides emotional validation that helps learners to feel "seen," which increases feedback receptivity (Lipnevich et al., 2021); and contributes to self-assessment development wherein specific feedback models the level of attention students should apply to their own work. Our rubric looks for substantive acknowledgment that shows that the feedback-giver actually read or listened to the student's ideas.

   Feedback must attend to emotional impact

   Feedback is inherently emotional (Lipnevich & Smith, 2022; Goetz et al., 2018). How students feel about feedback influences whether they engage with it, how they interpret it, and what they do with it (Alsaiari et al., 2024). Feedback quality should be evaluated not only in terms of accuracy or specificity, but also in terms of emotional calibration. Effective feedback is neither emotionally flat nor emotionally saturated. Excessive enthusiasm, reassurance, or empathy can feel inauthentic, distracting, or even patronizing, just as emotionally harsh language can be demotivating. High-quality feedback maintains a reasonably neutral, professional tone while remaining supportive and growth-oriented—acknowledging effort and progress when warranted, avoiding language that threatens self-worth, and pairing challenge with clear guidance for improvement.

4. What this rubric does not evaluate

   This rubric is intentionally focused on the content and quality of the feedback itself. To keep the rubric useful for rapid cycle review, there are several adjacent questions that it does not evaluate directly.

   • This rubric does not evaluate the student, the student work, or even the feedback giver. This is not about evaluating the student or the teacher—it is simply focused on the feedback.
   • We are not evaluating whether feedback contains scores. There is research suggesting that when students receive both feedback and a score, they may fixate on the score and ignore the feedback. However, this rubric does not evaluate whether a score is present, absent, or shown separately from the feedback. Whether and how scores are displayed is a product decision, and in some settings an instructional decision, rather than a component of the feedback itself. Our focus is the quality of the feedback message.
   • We are not evaluating questions versus statements. Some feedback tools and teachers prefer questions, while others prefer statements. This rubric does not privilege one form over the other. We are looking for actionable feedback that can take the form of either a question or a statement. What matters is whether the feedback gives the student a clear next step, stays aligned to the task, preserves student thinking, and remains manageable and constructive.
5. Scoring guide: six scoring rules

   Scoring guide

   When using this rubric to evaluate feedback, we follow a set of scoring rules that keep the process fair, consistent, and focused on improvement. These rules ensure that the data we get from scoring is actually helpful for refining feedback over time, rather than just turning into an obscure number. In practice, the approach is a bit different from traditional grading. It's more about looking for patterns and pinpointing specific areas to work on, instead of issuing a blanket judgment.

   1. Score each criterion independently.

      No score should be based on a prior score; all criteria should be treated independently. For example, if a student received feedback but should not have, while the first criterion, "Appropriate Feedback Decision," would not be met, it is still helpful to examine the quality of the feedback that was given to see if it was manageable, actionable, and so on, and could be used as an exemplar of how some parts of the feedback meet criterion.

   2. No single "overall" score for a session.

      One thing we don't do is collapse all the criteria into one final grade for a feedback session. It might be tempting to average all the rubric ratings and declare the session a 6 out of 8 or "75% successful." We avoid that. Each criterion in our rubric measures a distinct aspect of feedback quality. Averaging them together mixes apples and oranges and can obscure important information. For example, it is not unusual to see feedback that meets criterion for being anchored in the student response, but does not meet criterion for acknowledging a student's strength. An overall score would blur those together, whereas looking at each criterion separately lets us see exactly where things went well and where they didn't. The point isn't to give a simplistic thumbs-up or thumbs-down on the whole session, but to diagnose strengths and weaknesses so we know what to keep doing and what to fix.

   3. Focus on patterns, not just averages.

      In addition to not averaging different criteria, we also don't reduce a series of scores to a single mean when summarizing performance on one criterion. Instead, we report the frequency distribution of the ratings. For example, imagine we're looking at the "Alignment" of feedback across ten instances: instead of only saying, "On average, alignment was .30," we'd also note that 3 feedback messages aligned and 7 did not. Looking beyond the average can help reveal whether lower performance is driven by a small number of isolated cases or reflects a more recurring pattern across feedback instances. By looking at the spread of scores, we can spot if there are frequent problems that need addressing or if, say, most feedback is good except for a couple of slips. Those insights are much more actionable. We treat our rubric data as a collection of signals, not as one overall verdict.

   4. Always anchor scoring with a prompt-specific guide.

      Another rule we follow is that scoring should never happen in a vacuum. Before anyone starts rating feedback on a particular task or prompt, we make sure there's a scoring guide or "answer key" specific to that prompt. This guide lays out what good performance on that task looks like. It might include the learning objective, a sample high-quality student response, and examples of strong feedback for that scenario. The reason is simple: context matters for accurate scoring. If a prompt asks students to cite evidence from a text, then what counts as "aligned" or "accurate" feedback will depend on the content of that text and the expected answer. Scorers need to know, for instance, what the key evidence points were, or what a correct solution entails, so they can judge whether the feedback correctly addressed the student's attempt. The prompt-specific guide serves as a reference so that when we evaluate a piece of feedback, we're comparing it against the appropriate expectations for that task. This keeps our scoring consistent and fair. It also mirrors how a teacher or product team would use the rubric in practice—always in tandem with clear success criteria for the given assignment. In short, the rubric isn't one-size-fits-all; it's a framework that gets tailored with examples and notes for each new context, ensuring that "accuracy" or "acknowledgment," for example, are interpreted in light of the actual lesson at hand.

   5. Score what's written, not what you think was meant.

      To keep our evaluation objective, we have a rule against inference. When a piece of feedback is unclear or incomplete, we score it based on what is there on the page (or screen), without trying to infer the giver's intentions. This is especially important when feedback is ambiguous. The reason behind this rule is that students, too, can only respond to what was actually said. If the feedback is so vague that the scorer has to guess its meaning, a student would probably have to guess too, which means the feedback isn't as helpful as it could be. By avoiding inference, we ensure that our ratings reflect the real quality of the feedback from the learner's perspective. This rule also guides us on how to handle ambiguous cases. If a comment could be interpreted in two ways, and it's not clear which one the student would take, that ambiguity itself is treated as a flaw in the feedback. We might note the uncertainty, but we don't fill in the gaps for the scorer. Sticking to the text of the feedback keeps our evaluations grounded and honest.

   6. Calibrate and continually re-calibrate scorers.

      Our final rule is more of a practice—anyone using this rubric should go through calibration exercises to ensure consistency. What this means in plain terms is that two different people using the rubric should ideally come to the same scores when looking at the same piece of feedback. To get there, we often start a scoring session by having all scorers rate a few sample feedback comments independently and then compare notes. If one person labeled a comment as "meets criterion" and another person thought it was "does not meet criterion," they discuss why. They might go back to the underlying assumptions. For example, was the language unclear and therefore not actionable, or did the second scorer infer something that wasn't explicitly stated? Through these conversations, the group comes to a shared understanding of how to apply each criterion. Even for a single scorer working alone (like a teacher self-assessing their own feedback or a product manager reviewing AI outputs), it's useful to periodically check one's consistency by revisiting an earlier score to see if you'd rate it the same way now. We also encourage a "think-aloud" approach in training, or talking through the reasoning for a score on each criterion. This helps illuminate any differences in interpretation. The purpose of all this calibration is to make sure the rubric truly works as intended, highlighting genuine strengths and weaknesses in feedback, and that those insights are reliable. After all, the rubric is a tool for growth. If different people interpret it wildly differently, it won't be effective in guiding improvements. So, we invest time in calibration upfront, and we treat scoring as a skill that gets better with practice and reflection. This way, over multiple rapid cycles of development and use, the rubric remains a steady compass for improving feedback quality.

6. The full rubric

   This rubric represents the final working version used in our rapid-cycle evaluations, following multiple rounds of testing, revision, and refinement.

   Table scrolls sideways on small screens.

   Criterion	Meets criterion	Does not meet criterion
   Appropriate Feedback Decision Did the feedback correctly identify whether the student needed to revise?	Feedback correctly determines that the student response should receive feedback. That is: The response does NOT meet the evidence goal (i.e., does not successfully complete a claim with relevant evidence) If the response meets the evidence goal, but not the grammar goal.	Feedback incorrectly identifies whether the student response should receive feedback. Examples "Great job, you can move on" (when it's actually wrong) OR "This needs revision" (when it's actually right)
   Task Aligned Revision Does the feedback address the learning objective(s)?	Feedback is aligned to the task and supports progress toward completing it. Targets the core requirements of the task (i.e., support a claim by using relevant text-based evidence) Tells students to write one sentence if they wrote more than one initially Feedback may also target other relevant improvements (e.g., clarity or grammar) Reinforces students' effective use of evidence	Feedback is not aligned to the task and/or does not support meaningful progress toward completing the task. Feedback focuses on issues unrelated to the task (i.e., feedback is not related to use of evidence or grammar; e.g., asking someone to quote the text) Feedback introduces a requirement that is not specified in the directions (e.g., asking for a particular amount of evidence such as "2 [or more] pieces")
   Not Answer Giving Does the feedback avoid giving away the answer?	Feedback does not give away the full answer. Feedback does not provide the solution May scaffold for the student (e.g., hints, guiding questions) Student did not need to revise	Feedback provides the solution or completes the task for the student. Supplying the evidence the student should use Writing a complete or near-complete version of the response
   Actionable Revision Does the feedback give a clear next step?	Feedback gives a clear and usable next step for revision. It may ask a question or tell the student what to add, explain, clarify, revise, etc. The student could reasonably act on the feedback without additional clarification If the feedback writer determined the student responses did not need revision, the feedback does not need to include a next step (code this 1 – meeting)	The feedback does not give a clear and usable next step. Is purely evaluative The student could not reasonably act on the feedback Too little to go on.
   Manageable Is the amount of feedback manageable for the student?	The feedback is manageable in amount and scope for the learner. It is focused enough that the student could realistically process. Scope: Feedback addresses no more than two clearly prioritized issues. Length: In general, four sentences or fewer will usually count as manageable. Student did not need to revise (based on feedback writer's decision) Decision tree: Start with length anything over 4 sentences — automatic "not manageable" (in the context of the Quill product/activity) fewer than 4, then weigh the above criteria Then, check scope (see above) If the student did not need to revise (based on feedback writer's decision), then "1" Note: This doesn't necessarily mean the feedback is "good"; the focus of this criterion is to assess the feedback's scope/size/length. Parameters of manageability are dependent on product (task).	Feedback is not manageable in amount or scope for the learner. Task-Specific Requirements: Manageable feedback is defined per task. The feedback is too long, too dense, or includes too many suggestions at once.
   Anchored in Student Response Is the feedback clearly based on the student's specific response?	Feedback is clearly based on the student's specific response. Feedback specifically references, builds on, or responds to the student's specific idea, wording, or use of evidence (e.g., echoes the student's words or is an accurate observation/response to something the student did) Demonstrates an accurate understanding of what the student wrote Re-directing to be task aligned — feedback directs students to be task aligned when they are not, then we consider it anchored: If a student plagiarises, and the feedback tells student to use their own words, this counts as a "1" because it's responsive to the student's wording choice of plagiarizing If a student uses evidence (or opinion) that is OUTSIDE the text and the feedback writer addresses that, re-directing student, this can count as "1" because it's responding specifically to the student's choice of bringing in superfluous info. If a student writes more than one sentence and feedback writer tells student to write one sentence, then that is a "1"	Feedback is not clearly based on the student's specific response. It is generic and could apply to any response (e.g., template-like comments) It is based on a misinterpretation or misunderstanding of the student's response It addresses something the student did not actually say or attempt
   Acknowledges Strength Does the feedback identify what the student did well?	Feedback appropriately acknowledges strength when applicable. Feedback accurately identifies a specific strength in the student's response The strength is relevant to the task (e.g., use and specificity of evidence) Feedback is tied to something particular in this student's response, not a feature that could apply to any response The student response does not contain anything that warrants positive acknowledgement. For example, when a student writes "IDK."	Feedback does not appropriately acknowledge a strength when one is present. Feedback provides only generic or vague praise (e.g., "Good job," "Nice work") without specifying what was done well Feedback includes inaccurate or misleading praise
   Appropriate Emotional Pitch Is the tone appropriate and constructive for the student?	The tone is appropriate and constructive for the student. Feedback uses neutral, professional language throughout Word choice does not carry negative judgment of the student as a person, does not shame or dismiss, and does not use exaggerated positive language that misrepresents the quality of the work Feedback is neutral and/or respectful and/or supportive, even when pointing out areas for improvement Encourages continued effort without overstating or discouraging	The tone is not appropriate or constructive for the student, or evaluates the student rather than the work. Language is evaluative of the student rather than the work ("this is a weak response," "you clearly didn't read the text"), dismissive ("this makes no sense"), or uses praise so inflated it is inaccurate ("this is excellent" when the response is incomplete). Tone is harsh, overly negative, or discouraging (e.g., shaming, dismissive language) Tone is inappropriately exaggerated or misleading (e.g., overly enthusiastic praise that does not match the quality of the response)

7. Works cited

   • Alsaiari, O., Baghaei, N., Lahza, H., Lodge, J. M., Boden, M., & Khosravi, H. (2024). Emotionally enriched feedback via generative AI [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2410.15077
   • Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. https://doi.org/10.1146/annurev-psych-113011-143823
   • Dweck, C. S. (2006). Mindset: The new psychology of success. Random House.
   • Goetz, T., Lipnevich, A. A., Krannich, M., & Gogol, K. (2018). Performance feedback and emotions. In A. A. Lipnevich & J. K. Smith (Eds.), The Cambridge handbook of instructional feedback (pp. 554–574). Cambridge University Press. https://doi.org/10.1017/9781316832134.027
   • Lipnevich, A. A., Murano, D., Krannich, M., & Götz, T. (2021). Should I grade or should I comment? Links among feedback, emotions, and performance. Learning and Individual Differences, 89, 102020. https://doi.org/10.1016/j.lindif.2021.102020
   • Lipnevich, A. A., & Smith, J. K. (2009). "I really need feedback to learn": Students' perspectives on the effectiveness of the differential feedback messages. Educational Assessment, Evaluation and Accountability, 21(4), 347–367. https://doi.org/10.1007/s11092-009-9082-2
   • Lipnevich, A. A., & Smith, J. K. (2022). Student–feedback interaction model: Revised. Studies in Educational Evaluation, 75, 101208. https://doi.org/10.1016/j.stueduc.2022.101208
   • Peterson, E. R., & Irving, S. E. (2008). Secondary school students' conceptions of assessment and feedback. Learning and Instruction, 18(3), 238–250. https://doi.org/10.1016/j.learninstruc.2007.05.001
   • Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (M. Cole, V. John-Steiner, S. Scribner, & E. Souberman, Eds.). Harvard University Press.
   • Warshauer, H. K. (2015). Productive struggle in middle school mathematics classrooms. Journal of Mathematics Teacher Education, 18(4), 375–400. https://doi.org/10.1007/s10857-014-9286-3
   • Winstone, N., & Carless, D. (2019). Designing effective feedback processes in higher education: A learning-focused approach. Routledge. https://doi.org/10.4324/9781351115940
   • Young, J. R., Bevan, D., & Sanders, M. (2024). How productive is the productive struggle? Lessons learned from a scoping review. International Journal of Education in Mathematics, Science and Technology, 12(2), 470–495. https://doi.org/10.46328/ijemst.3364