OpenAI's assessment calibration workflow describes reviewing de-identified assessment material and flagging scoring patterns for human consideration. It separates those flags from grading decisions. The task here audits an existing rubric and its records; if you need to create performance levels, start with the rubric maker guide.
Supply the criteria and the record identities
Provide the rubric version, assignment version, scoring scale, criterion IDs, permitted de-identified excerpts, and the original reviewer records. Explain which submissions were deliberately double-marked. Comparing two unrelated students' totals is a different question from comparing two readings of the same work.
Keep student identifiers and the re-identification key in the authorized course system. Remove identifying material from excerpts, not just the name column. Use only material and accounts permitted for this review under your institution's policy.
This original fictional fixture uses a three-criterion rubric, each criterion scored from 1 to 4. E means evidence, R reasoning, and O organization. All points below are invented calibration records, not grades from actual students.
| Record | Submission / assignment | Reviewer | E / R / O | Total |
|---|---|---|---|---|
| A1 | S01 / V2 | A | 4 / 2 / 3 | 9 |
| B1 | S01 / V2 | B | 2 / 4 / 3 | 9 |
| A2 | S02 / V2 | A | 3 / 3 / 3 | 9 |
| B2 | S02 / V2 | B | 3 / 2 / 3 | 8 |
| A3 | S03 / V1 | A | 3 / 3 / 2 | 8 |
| B3 | S03 / V2 | B | 4 / 3 / 2 | 9 |
The last pair uses different assignment versions. Do not include it in a like-for-like difference summary until someone establishes whether the versions are comparable.
Inspect criterion differences before aggregates
For this exercise, the faculty review rule flags an absolute difference of at least two points on a criterion. It is a supplied local review rule, not a universal grading standard.
S01 triggers two flags: E differs by 2, and R differs by 2. Both totals are 9, so a total-only check would miss them. S02 differs by 1 on R and does not meet this particular threshold. Keep that smaller difference visible in the aligned table; “not flagged” does not mean the judgments have been validated.
Only two submission pairs currently qualify for like-for-like comparison. One of those two pairs has at least one criterion meeting the review threshold. You may describe that count as one of two comparable pairs, but this tiny fictional sample cannot establish that either reviewer is systematically harsher.
Two flagged criteria are also not two different students. State the counting unit beside every summary: criterion differences, submission pairs, or reviewer records.
Ask what evidence could explain the difference
For S01, retrieve the permitted source excerpt and both reviewers' notes for E and R. Check that each note addresses the correct criterion and version. A high evidence score and a low reasoning score can coexist; don't assume the two criteria should move together.
A missing note is an evidence gap. ChatGPT should not write a plausible justification on behalf of the reviewer. Faculty might discover a misplaced score, an omitted passage, or a descriptor boundary needing clarification. The aligned numbers alone cannot choose among those explanations.
Keep any proposed clarification in a separate column with its decision owner. Do not average the original scores and call the result a resolved grade.
Copy an audit-only prompt
Audit these existing rubric scoring records for faculty calibration.
Rubric/assignment versions and criterion IDs: [PASTE]
Original de-identified reviewer rows and record IDs: [ATTACH]
Which records are deliberate double marks: [ENTER]
Supplied comparison/flag rule: [PASTE]
Permitted excerpts and reviewer notes: [ATTACH OR NOT AVAILABLE]
Align like-for-like records by submission, assignment version, rubric,
and criterion. Isolate unmatched, duplicate, and version-mismatched rows.
Preserve originals. Show signed/absolute differences and retain small
differences even when below my flag rule. State the denominator and
counting unit for every summary. Flag missing notes as missing evidence.
For each review lead cite the exact record IDs and available excerpt.
Do not invent a rationale, infer bias from a small sample, assign/change
grades, or average disagreement into a final score. Keep proposed rubric
clarifications separate and leave faculty decisions unresolved.
Keep the review decision outside the original score columns
Bring S01's two criterion flags and S03's version mismatch to the calibration discussion. Record what faculty actually concluded, which evidence they considered, and whether any clarification applies prospectively or needs a separate policy decision.
Save the review workbook beside the authorized assessment records. Preserve the original reviewer entries even when a later decision changes the official outcome through the institution's normal process. The next calibration round should be able to distinguish an original judgment from a reviewed decision.