Agreement you can measure
Annotation work is double-annotated blind: two experts label the same items without seeing each other's work, and inter-annotator agreement (Cohen's kappa) is computed against explicit thresholds. Disagreement is information — it routes items to adjudication rather than into the dataset.
Gold that grows
Quality sampling is seeded with gold items whose answers are established in advance. When human adjudication settles a genuine disagreement, the resolution becomes new gold — the standard sharpens as the work happens.
People close the loop
Automated checks surface candidates for review; humans make the calls that matter. Every verdict is recorded per grader, so agreement between graders — human or model — is measured, not assumed.