The calculation takes ten rows. Reading it takes one more test, because the statistic can pass a model whose overall default rate is plainly wrong.
The Hosmer-Lemeshow test sorts borrowers into groups by predicted PD, usually ten, and sums (observed defaults minus expected defaults) squared over N × PD × (1 − PD) for each group; the total is compared with a chi-square distribution on 8 degrees of freedom. On an illustrative validation sample of 10,000 loans the statistic is 10.98, a p-value of 0.2028, so the model passes. Yet it under-predicts defaults by 10.7 per cent, and a one-line test of the overall rate rejects it at 0.43 per cent.
Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →
A PD model has scored 10,000 loans, sorted into ten groups of 1,000 by predicted PD. A year later, 692 have defaulted against 625 expected. The data are illustrative.
| Mean PD | Loans | Expected defaults | Observed defaults | Observed rate |
|---|---|---|---|---|
| 0.5% | 1,000 | 5.0 | 4 | 0.4% |
| 1.0% | 1,000 | 10.0 | 12 | 1.2% |
| 1.5% | 1,000 | 15.0 | 19 | 1.9% |
| 2.5% | 1,000 | 25.0 | 30 | 3.0% |
| 3.5% | 1,000 | 35.0 | 41 | 4.1% |
| 5.0% | 1,000 | 50.0 | 57 | 5.7% |
| 6.5% | 1,000 | 65.0 | 76 | 7.6% |
| 9.0% | 1,000 | 90.0 | 102 | 10.2% |
| 13.0% | 1,000 | 130.0 | 146 | 14.6% |
| 20.0% | 1,000 | 200.0 | 205 | 20.5% |
| 6.25% | 10,000 | 625 | 692 | 6.92% |
Termg = (Og − Ng × PDg)2 / (Ng × PDg × (1 − PDg))
Group at 6.5%: (76 − 65.0)2 / (1,000 × 6.5% × 93.5%) = 121 / 60.775 = 1.991
Excel: =(Obs-N*PD)^2/(N*PD*(1-PD)), then =SUM() over the ten rows
| Mean PD | Term | z = (O − E) / sd |
|---|---|---|
| 0.5% | 0.201 | -0.45 |
| 1.0% | 0.404 | 0.64 |
| 1.5% | 1.083 | 1.04 |
| 2.5% | 1.026 | 1.01 |
| 3.5% | 1.066 | 1.03 |
| 5.0% | 1.032 | 1.02 |
| 6.5% | 1.991 | 1.41 |
| 9.0% | 1.758 | 1.33 |
| 13.0% | 2.263 | 1.50 |
| 20.0% | 0.156 | 0.40 |
| HL statistic | 10.98 |
Degrees of freedom = groups − 2 = 8; 5 per cent critical value = 15.507
p-value = CHISQ.DIST.RT(10.98, 8) = 0.2028
At 10.98 against a critical value of 15.507, the test does not reject: by its standard, predicted and observed defaults agree. The g − 2 convention comes from the development sample, where the model's parameters were fitted to the same data. On an independent out-of-time sample, such as this one, many validators use 10 degrees of freedom instead; this article keeps the stricter 8. Ten would be more lenient here: a p-value of 0.3591 against a critical value of 18.307.
Look at the z column. Nine groups out of ten have more defaults than predicted. No single group is far off, so no single term is large, and the chi-square spreads the evidence over eight degrees of freedom. Tested on the total instead:
z = (692 − 625) / √(Σ N × PD × (1 − PD)) = 67 / √551.08 = 67 / 23.47 = 2.85
Two-sided p-value = 0.0043, a rejection at the 1 per cent level
The model ranks well and is wrong on level: it predicts 6.25 per cent and the sample runs at 6.92, an under-prediction of 10.7 per cent. Scale every PD by 692 / 625 = 1.107 and the Hosmer-Lemeshow statistic falls to 3.15, a p-value of 0.9245. The test that passed was not detecting that the level was off.
With the same default rates in every group, the statistic grows in proportion to sample size.
| Loans per group | Total loans | HL statistic | p-value |
|---|---|---|---|
| 250 | 2,500 | 2.74 | 0.9493 |
| 500 | 5,000 | 5.49 | 0.7041 |
| 1,000 | 10,000 | 10.98 | 0.2028 |
| 2,000 | 20,000 | 21.96 | 0.0050 |
| 4,000 | 40,000 | 43.92 | 0.0000 |
The calibration is identical in every row. A sample 1.41 times larger, 14,123 loans, would reject it at 5 per cent. A small portfolio passes almost anything and a retail book of a million loans fails almost everything, so the p-value says as much about sample size as about the model.
Reporting "Hosmer-Lemeshow passed" as proof of calibration. Read it with the observed-against-expected table, the sign of each group's error and a test of the overall default rate. Defaults are also correlated through the economic cycle, which the test's binomial variance ignores, so in a downturn year it rejects good models. The fix for a level error is in how to recalibrate a PD model to a new default rate; ranking is measured separately, in Gini and KS.
The calibration table on fixed bands, and the intercept shift that repairs it, are live in the free workbook for this book.
Groups minus two, so 8 for the usual ten groups, on the sample the model was fitted to; the 5 per cent critical value is then 15.507. On an independent validation sample many validators use the number of groups, 10, with a critical value of 18.307. An illustrative statistic of 10.98 gives p-values of 0.2028 and 0.3591 under the two conventions.
The statistic is proportional to sample size for a given pattern of errors. In the illustrative case the same default rates give 2.74 on 2,500 loans, 10.98 on 10,000 and 43.92 on 40,000. A modest, practically irrelevant miscalibration becomes significant on a large retail book, and a serious one can pass on a small portfolio.
The observed and expected defaults by group, the sign of each group's error, and a test of the overall default rate. In the illustrative sample HL passes at 10.98, but nine of ten groups under-predict and the overall test gives z of 2.85, a p-value of 0.0043. Scaling every PD by 1.107 brings HL down to 3.15.
This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.
Get the book on Amazon →Free companion files
Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada
Reading guide: corporate finance, valuation and markets → · All 1340 articles →
If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.