McNemar's Test for NLP

Paired significance test for comparing two models on the same test set. Only the disagreements (b and c) matter — concordant pairs carry no information. Switches automatically to the exact binomial when b+c < 25.

Runs entirely in your browser. Nothing is sent anywhere.

Contingency table

Fill the 2×2 table of per-instance agreements. a = both correct. b = Model A correct, B wrong. c = Model A wrong, B correct. d = both wrong. Only b and c enter the test.

When to use this vs Newcombe: if both models were scored on the same problems, use McNemar (paired). If scored on different samples, use Newcombe's difference-of-proportions instead.
a — both correct
b — A correct, B wrong ★
c — A wrong, B correct ★
d — both wrong
Condition 1 label
Condition 2 label
Significance level (α)
Continuity correction

Results

Paper-ready output
LaTeX