Wilson score confidence intervals, difference of proportions, and McNemar’s paired test — the statistics behind reporting LLM evaluation accuracy honestly.
Runs entirely in your browser. Nothing is sent anywhere.
Accuracy from k correct out of n trials is a binomial estimate, not a fact. Wilson intervals behave well at small n and extreme accuracies, where the naive (Wald) interval breaks.
| Condition | Correct k | Total n | Accuracy | 95% CI |
|---|
Newcombe’s score interval for p₁ − p₂, built from the Wilson bounds of two rows above. Use this for different samples or models. If both conditions were scored on the same problems, use McNemar below instead — it is strictly more powerful for paired data.
When two conditions are evaluated on the same problems, only the disagreements matter: b = correct in condition 1 but wrong in condition 2; c = the reverse. The gap is real iff b ≫ c. Uses the exact binomial test automatically when b + c < 25.
Chi-square uses χ² = (b−c)²/(b+c) with 1 df. The exact test is a two-sided binomial test of min(b,c) successes in b+c trials at p = ½.