DAGL

Know how often the AI agrees with your expert.

Agree the expected answers first. Count where the AI and your expert agree on real cases. When they differ, see which step went wrong.

How do you know whether AI answers are right?

Compare them with your expert's answers on real cases before you rely on them. Put the expert's answer in the next column and count where the two agree. Fix the instructions where they do not. A check that agrees on 46 of 50 cases shows you the 4 to fix first.

Try it on your data
01

Agree the answers first.

Pick real cases and write down the answer your expert expects. These are the cases every version is tested on.

Saved cases · 50
SupplierExpected answerWhy
Northwind FoodsApproveValid, all sites covered
Harbour DairyReviewExpires within 30 days
Alder & SonsRejectExpired
02 · 03

Count the agreement, then find the step that went wrong.

The AI's column sits next to your expert's column. Open a case that differs and follow it step by step to the step that went wrong.

46 of 50 cases agree4 differ
  1. Harbour Dairy · AI said Approve, expert said Review
  2. FetchCertificate found✓
  3. ReadExpires 2 Nov 2026✓
  4. Ask AIThe instructions do not mention the 30-day rule✗
04

Fix the instructions yourself.

Change them in plain words and run the same cases again. No engineer, no code.

Version 4Approve the supplier if the certificate is valid and covers every site.
Version 5 · 49 of 50 agreeApprove the supplier if the certificate is valid and covers every site. Send it to review if it expires within 30 days.
05

When the AI changes, recheck first.

Run your saved cases on the new AI before you switch. Keep the old one until the new one agrees as often.

Current AI49 / 50
New AI50 / 50

How often AI agrees with expert reviewers on six common checks.

We publish the data and the instructions, so you can run the same checks yourself. [Study date and headline result.]

Read the study →

Questions about accuracy.

Guide

Agree the expected answers before you build an AI check

Choose real cases and write down the answer your expert expects. These become the test for every version.

Read more →
Guide

How to measure how often AI agrees with your expert

Put both answers side by side and count the matches. Count by answer type too, so one common case does not hide a rare one.

Read more →
Worked example

How to find the step that gave the wrong answer

Open a case that differs and follow it step by step. The first step whose output looks wrong is where to fix.

Read more →
Guide

How to tell whether a change to your instructions made the answers better

Run the old and the new version on the same saved cases and compare the counts.

Read more →
Checklist

What to recheck when your AI provider updates its AI

Rerun your saved cases on the new AI and compare the counts. Keep the old AI until the new one agrees as often.

Read more →
Guide

How an operations lead can fix AI instructions without an engineer

Write the rule your expert uses in plain words and add it to the instructions. Then run the cases that differed.

Read more →
Checklist

Five things an AI check needs before your team relies on it

Saved cases, an agreement count, a named owner, a budget, and a recheck plan for when the AI changes.

Read more →
Data study

How often AI agrees with expert reviewers on six common checks

Our study, with the data and the instructions published, refreshed twice a year.

Read more →
What does it cost to run AI on every row?Which tool should a team use to run AI on many records?See a check built for your team.

Bring ten cases your expert has already checked.

We will build the check with you and count where it agrees.

Try it on your data

Written by [Founder name], [role] · Updated 24 October 2026: first published.