Suppose 100 people take a pass/fail test before a treatment and repeat it afterward. The pass rate rises from 45% to 55%. Does that provide evidence that the probability of passing has changed?
To answer that question, we need to know which people changed their outcomes. McNemar’s test uses that pairing. Fisher’s exact test is appropriate for comparing pass rates between two independent groups, such as untreated people and a different set of treated people.
When does McNemar’s test apply?
Use McNemar’s test when each independent unit contributes two paired binary outcomes. Examples include
- the same person measured before and after treatment
- diagnostic tests administered to the same person
- two classifiers evaluated on the same test cases, recording correct/incorrect predictions.
The observations within a pair may be dependent; the pairs should be independent of one another. The basic test assumes complete pairs and does not handle extra clustering, repeated visits beyond two, or adjustment for covariates.
McNemar’s null hypothesis is that the two marginal probabilities are equal. For the before/after example:
\[ H_0: P(\text{pass before}) = P(\text{pass after}). \]
This is often called marginal homogeneity. See the overview of McNemar’s test for its paired-table formulation.
Same people versus independent groups
Same people: use a table of transitions
For the 100 people measured twice, suppose we observe:
| Before treatment | After: pass | After: fail | Total |
|---|---|---|---|
| Pass | 40 | 5 | 45 |
| Fail | 15 | 40 | 55 |
| Total | 55 | 45 | 100 |
Each person appears in exactly one cell. Forty passed both times, 40 failed both times, five changed from pass to fail, and 15 changed from fail to pass.
The before pass rate is 45/100; the after pass rate is 55/100. The increase is 10 percentage points. The evidence for change comes from the imbalance between the five people who worsened and the 15 who improved.
Different people: use a table of groups and outcomes
Now consider a different study: 100 untreated people and 100 other people who received treatment.
| Group | Pass | Fail | Total |
|---|---|---|---|
| Untreated | 45 | 55 | 100 |
| Treated | 55 | 45 | 100 |
Each person contributes one outcome. There are no individual transitions to count, so McNemar’s test does not apply. Fisher’s exact test can compare the passing odds between these independent groups, conditioning on the observed table margins. For a binary outcome, equality of passing odds is equivalent to equality of passing probabilities. See the SciPy Fisher exact test documentation.
The pass rates happen to match the paired study, but the sampling designs differ:
| Feature | McNemar’s test | Fisher’s exact test for independent groups |
|---|---|---|
| Observations | Two outcomes per person or matched pair | One outcome per person |
| Table cells | Joint outcomes within pairs | Outcomes within separate groups |
| Null hypothesis | Equal marginal passing probabilities | Equal passing odds between groups |
| Main information | Changes in opposite directions | Distribution of outcomes between groups |
Both tests have exact versions. “Exact” describes the calculation under the assumed sampling model; it does not make an unsuitable sampling model appropriate.
McNemar’s formula
Write the paired table as:
| Before treatment | After: pass | After: fail |
|---|---|---|
| Pass | \(a\) | \(b\) |
| Fail | \(c\) | \(d\) |
Here \(b\) counts people who worsened and \(c\) counts people who improved. The observed difference in passing proportions is
\[ \widehat p_{\text{after}}-\widehat p_{\text{before}} =\frac{a+c}{N}-\frac{a+b}{N} =\frac{c-b}{N}, \qquad N=a+b+c+d. \]
The \(a\) and \(d\) cells are concordant: their outcomes agree. The \(b\) and \(c\) cells are discordant: their outcomes differ. Under the null, the two directions of change have equal probability.
Large-sample approximation
The uncorrected McNemar statistic is
\[ X^2=\frac{(b-c)^2}{b+c}. \]
When the number of discordant pairs is sufficiently large, compare this statistic to a chi-square distribution with one degree of freedom:
\[ p=P(\chi^2_1\ge X^2). \]
A common continuity-corrected version is
\[ X^2_{\text{corrected}}=\frac{(|b-c|-1)^2}{b+c}. \]
For \(b=5\) and \(c=15\), the uncorrected statistic is \(100/20=5\), giving \(p\approx0.025347\). With the correction, it is \(81/20=4.05\), giving \(p\approx0.044171\).
The relevant sample size for this approximation is \(b+c\), the number of discordant pairs, rather than the total number of participants.
Exact McNemar test
Conditional on \(m=b+c\) discordant pairs, the null assigns probability \(1/2\) to each direction of change. Thus
\[ B\mid m\sim\operatorname{Binomial}(m,1/2). \]
The conventional two-sided exact p-value is
\[ p_{\text{exact}}= \min\left\{1,\;2\sum_{k=0}^{\min(b,c)}\binom{m}{k}2^{-m}\right\}. \]
In our example, \(m=20\) and \(\min(b,c)=5\), so
\[ p_{\text{exact}}=2\sum_{k=0}^{5}\binom{20}{k}2^{-20} =0.0413894653. \]
This gives evidence against equal marginal pass probabilities at the conventional 5% level. The observed direction is an increase. A before/after comparison alone does not establish that treatment caused it: practice effects and other changes over time could also explain the difference.
The statsmodels documentation distinguishes the exact binomial calculation from the chi-square approximation and its optional continuity correction.
Running the Python examples
This R Markdown document runs Python chunks through the R package reticulate. Install it with install.packages("reticulate") if needed.
The setup selects the same local Python executable used in the Simpson’s paradox post. Replace this path with your own Python executable or virtual environment before running the chunks. If RETICULATE_PYTHON is set, it takes precedence over use_python().
To find python3 on your PATH, run:
Sys.which("python3")Install SciPy and statsmodels in the selected Python environment. Replace python below with that environment’s executable if necessary:
python -m pip install scipy statsmodelsReticulate runs all Python chunks in a shared session, so imports and objects remain available in subsequent chunks.
A Python function
The following function implements the two-sided test from the two discordant counts. It defaults to the exact calculation and also supports the chi-square approximation.
from numbers import Integral
from scipy.stats import binomtest, chi2
def mcnemar_test(b, c, *, exact=True, correction=True):
"""Return a two-sided McNemar p-value.
b: number changing from pass to fail
c: number changing from fail to pass
correction applies only when exact=False.
"""
for name, count in (("b", b), ("c", c)):
if isinstance(count, bool) or not isinstance(count, Integral):
raise TypeError(f"{name} must be an integer count")
if count < 0:
raise ValueError(f"{name} must be nonnegative")
b, c = int(b), int(c)
m = b + c
if m == 0:
# No changes: no evidence against equal marginal probabilities.
return 1.0
if exact:
return float(binomtest(b, n=m, p=0.5,
alternative="two-sided").pvalue)
difference = abs(b - c)
statistic = (difference - int(correction)) ** 2 / m
return float(chi2.sf(statistic, df=1))
print(f"Exact: {mcnemar_test(5, 15):.6f}")## Exact: 0.041389print(f"Uncorrected: {mcnemar_test(5, 15, exact=False, correction=False):.6f}")## Uncorrected: 0.025347print(f"Corrected: {mcnemar_test(5, 15, exact=False):.6f}")## Corrected: 0.044171SciPy’s binomtest supplies the exact binomial p-value. Using a distribution function avoids calculating large binomial coefficients directly. When there are no discordant pairs, the function explicitly returns 1; the chi-square formula itself would divide by zero.
For an existing implementation that accepts the whole paired table:
from statsmodels.stats.contingency_tables import mcnemar
paired_table = [[40, 5], [15, 40]]
result = mcnemar(paired_table, exact=True)
print(f"statsmodels: {result.pvalue:.6f}")## statsmodels: 0.041389What happens if we use Fisher’s test on the paired data?
There are two distinct ways to do this, and they answer different questions.
Mistake 1: discard the pairing
We could collapse the paired observations into a before/after table:
from scipy.stats import fisher_exact
# These rows contain measurements from the SAME 100 people.
before_after = [[45, 55], [55, 45]]
result = fisher_exact(before_after, alternative="two-sided")
print(f"Fisher after discarding pairs: {result.pvalue:.6f}")## Fisher after discarding pairs: 0.202982The calculation treats the rows as independent samples, even though each person contributes to both. It loses the information that only 20 people changed outcomes, with improvements outnumbering declines three to one.
| Analysis of the paired study | Two-sided p-value | Below 0.05? |
|---|---|---|
| Exact McNemar | 0.041389 | Yes |
| Fisher after discarding pairs | 0.202982 | No |
If these counts actually came from the two independent groups described earlier, the Fisher calculation would be appropriate. Its use is inappropriate here because the observations are paired. The numerical example shows how ignoring pairing can change a conclusion; it does not imply that Fisher always produces a larger p-value.
Mistake 2: pass the paired transition table directly to Fisher
Another tempting approach is to give Fisher the original table:
paired_table = [[40, 5], [15, 40]]
result = fisher_exact(paired_table, alternative="two-sided")
print(f"Fisher on transitions: {result.pvalue:.6e}")## Fisher on transitions: 3.436170e-10# Fisher on transitions: 3.436170e-10Here each person contributes once, so Fisher can test an association between that person’s before and after outcomes, assuming people are independent. But that is a different hypothesis from equal before/after pass probabilities.
Its odds ratio is
\[ \frac{ad}{bc}=\frac{40\times40}{5\times15}=21.3333. \]
The tiny p-value indicates strong evidence of association: people who passed before were more likely to pass afterward than people who failed before. It does not test whether the overall pass rate changed.
The hypotheses make the distinction explicit:
\[ \begin{aligned} \text{McNemar:}\quad &P(\text{pass before, fail after}) =P(\text{fail before, pass after}),\\ \text{Fisher on transitions:}\quad &\text{before outcome and after outcome are independent}. \end{aligned} \]
Strong association can coexist with identical marginal pass probabilities. For example, the transition table [[45, 5], [5, 45]] has 50% passing at each time and an exact McNemar p-value of 1, despite strong agreement between measurements.
Choose the table and test from the study design and the question: **for a change in binary test.
Last modified on 2026-11-13