The exact one-sided binomial test, the Jeffreys variant supervisors use, the normal shortcut that misleads on low-default grades, and what default correlation does to all three.
For each rating grade, compute the probability of seeing at least the observed number of defaults if the grade's PD were correct: one minus the binomial cumulative distribution at defaults minus one. A grade with a 3 per cent PD, 1,200 loans and 52 defaults against 36 expected has a p-value of 0.0063, so the PD is rejected as too low at the 1 per cent level. The Jeffreys test, the Bayesian variant European supervisors use, gives 0.0051.
Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →
A rating model assigns each borrower to one of seven grades, each with a PD. One year later the defaults are counted. The data are illustrative. Each grade is tested on its own, one-sided, because the question that matters to capital is whether the PD is too low.
| Grade | PD | Loans | Expected defaults | Observed | Observed rate |
|---|---|---|---|---|---|
| 1 | 0.10% | 1,800 | 1.8 | 5 | 0.28% |
| 2 | 0.30% | 2,400 | 7.2 | 9 | 0.38% |
| 3 | 0.75% | 2,600 | 19.5 | 22 | 0.85% |
| 4 | 1.50% | 1,900 | 28.5 | 37 | 1.95% |
| 5 | 3.00% | 1,200 | 36.0 | 52 | 4.33% |
| 6 | 7.00% | 600 | 42.0 | 46 | 7.67% |
| 7 | 15.00% | 300 | 45.0 | 49 | 16.33% |
| All | 1.67% | 10,800 | 180.0 | 220 | 2.04% |
If grade 5's PD of 3 per cent is right and defaults are independent, the number of defaults among 1,200 loans is binomial with mean 36.0.
p-value = P(X ≥ D | N, PD) = 1 − BINOM.DIST(D − 1, N, PD, TRUE)
Grade 5: 1 − BINOM.DIST(51, 1200, 0.03, TRUE) = 0.0063
Critical count at 95 per cent, the smallest D with p ≤ 5%: =BINOM.INV(N, PD, 0.95)+1 = 47; at 99 per cent, 51
52 defaults is past both critical counts: the grade's PD is too low. The observed rate of 4.33 per cent is not a surprise in a single loan; it is a surprise in 1,200 of them.
The ECB's validation reporting uses the Jeffreys test. With a Jeffreys prior, the posterior distribution of the grade's true default rate is a Beta distribution, and the p-value is the posterior probability that the true rate is at or below the PD.
Jeffreys p-value = BETA.DIST(PD, D + 0.5, N − D + 0.5, TRUE)
Grade 5: BETA.DIST(0.03, 52.5, 1148.5, TRUE) = 0.0051
On this sample the two tests reject the same grades. The Jeffreys p-value is lower on every grade, so it rejects a little more readily, and it behaves better with zero defaults, where the binomial p-value is exactly 1 and says nothing.
| Grade | Expected | Observed | Binomial p | Jeffreys p | Critical at 95% | Critical at 99% | Verdict at 5% |
|---|---|---|---|---|---|---|---|
| 1 | 1.8 | 5 | 0.0363 | 0.0197 | 5 | 7 | Reject |
| 2 | 7.2 | 9 | 0.2972 | 0.2399 | 13 | 15 | Pass |
| 3 | 19.5 | 22 | 0.3142 | 0.2764 | 28 | 31 | Pass |
| 4 | 28.5 | 37 | 0.0699 | 0.0590 | 38 | 42 | Pass |
| 5 | 36.0 | 52 | 0.0063 | 0.0051 | 47 | 51 | Reject |
| 6 | 42.0 | 46 | 0.2824 | 0.2567 | 54 | 58 | Pass |
| 7 | 45.0 | 49 | 0.2814 | 0.2554 | 56 | 61 | Pass |
Grade 4 is the one to watch: one more default and it would have reached its critical count of 38. Grade 1 rejects on five defaults, which shows how little room a low-default grade has: at a 0.1 per cent PD on 1,800 loans, the fifth default is enough.
Many spreadsheets use z = (D − N × PD) / √(N × PD × (1 − PD)). For grade 5, z is (52 − 36.0) / 5.909 = 2.708 and the one-sided p-value 0.0034, not far from the exact 0.0063. For grade 1, with 1.8 defaults expected, z is 2.39 and the normal p-value 0.0085, against an exact 0.0363. The normal curve has thin tails where the binomial with a tiny mean is lumpy and skewed, so the shortcut overstates the evidence about fourfold in exactly the grades where defaults are rare. Use the exact test or Jeffreys whenever expected defaults are in single figures.
| Loans in grade | Expected defaults | Critical count | Critical rate | Times the PD |
|---|---|---|---|---|
| 200 | 3.0 | 7 | 3.50% | 2.33 |
| 500 | 7.5 | 13 | 2.60% | 1.73 |
| 1,000 | 15.0 | 23 | 2.30% | 1.53 |
| 1,900 | 28.5 | 38 | 2.00% | 1.33 |
| 5,000 | 75.0 | 90 | 1.80% | 1.20 |
On 200 loans a grade's observed default rate must reach 2.33 times its PD before the test notices. Small grades pass almost anything, which is why a passing binomial test on a thin grade is weak evidence of calibration.
The calibration table on fixed bands, and the out-of-time sample it is measured on, are live in the free workbooks for this book.
A Bayesian version of the binomial test: with a Jeffreys prior, the posterior for the default rate is a Beta distribution with parameters D + 0.5 and N - D + 0.5, and the p-value is that distribution's probability below the PD. For 52 defaults on 1,200 loans at a 3 per cent PD it is BETA.DIST(0.03, 52.5, 1148.5, TRUE) = 0.0051, close to the exact binomial 0.0063.
Because it is poor when expected defaults are few. In an illustrative grade with a 0.1 per cent PD, 1,800 loans and 5 defaults against 1.8 expected, the normal approximation gives a p-value of 0.0085 while the exact binomial gives 0.0363. The shortcut overstates the evidence fourfold in exactly the low-default grades where it matters.
The binomial test assumes defaults are independent, so in a bad year it rejects correct PDs. With a Vasicek one-factor model and 12 per cent asset correlation, the 95 per cent critical default rate for a 3 per cent PD grade is 8.11 per cent, or 97 defaults on 1,200 loans, against 47 under independence. The truth for most portfolios lies between the two.
This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.
Get the book on Amazon →Free companion files
Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada
Reading guide: corporate finance, valuation and markets → · All 1340 articles →
If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.