Binary cross-entropy computed from band counts, read against the base-rate model, and the reason an overconfident PD model can score worse than no model at all.
For each loan, take the log of the predicted PD if it defaulted, or the log of one minus the PD if it did not, average across the sample and change the sign. On an illustrative validation sample of 10,000 loans in ten PD bands, log loss is 0.2066. On its own that number means little: a model that gives every loan the portfolio rate of 6.25 per cent scores 0.2338, so the PD model improves on the base rate by 11.6 per cent.
Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →
A PD model has scored 10,000 loans, grouped into ten bands of 1,000 by predicted PD. Within a band every loan carries the band's PD, so the calculation can be done from counts. The data are illustrative, and chosen so that expected and observed defaults agree in total: 625 each.
| PD | Loans | Defaults | Contribution to −log-likelihood | Per loan |
|---|---|---|---|---|
| 0.5% | 1,000 | 6 | 36.77 | 0.0368 |
| 1.0% | 1,000 | 9 | 51.41 | 0.0514 |
| 1.5% | 1,000 | 16 | 82.07 | 0.0821 |
| 2.5% | 1,000 | 24 | 113.24 | 0.1132 |
| 3.5% | 1,000 | 37 | 158.35 | 0.1583 |
| 5.0% | 1,000 | 49 | 195.57 | 0.1956 |
| 6.5% | 1,000 | 68 | 248.51 | 0.2485 |
| 9.0% | 1,000 | 88 | 297.91 | 0.2979 |
| 13.0% | 1,000 | 127 | 380.68 | 0.3807 |
| 20.0% | 1,000 | 201 | 501.79 | 0.5018 |
| 6.25% | 10,000 | 625 | 2,066.30 | 0.2066 |
Log loss = −(1/N) × Σ [ y × ln(PD) + (1 − y) × ln(1 − PD) ]
Band at 6.5%: 68 defaults and 932 non-defaults
Defaults: −68 × ln(0.065) = −68 × (−2.7334) = 185.87
Non-defaults: −932 × ln(0.935) = −932 × (−0.0672) = 62.64
Band total = 248.51
Excel, per band: =-(D*LN(PD)+(N-D)*LN(1-PD)); loan by loan: =-AVERAGE(Y*LN(P)+(1-Y)*LN(1-P))
The 68 defaults in that band cost three times as much as the 932 loans that repaid. That asymmetry runs through the whole sample: defaults are 6.25 per cent of the loans and 71.6 per cent of the log loss.
Sum of the ten bands = 2,066.30; divided by 10,000 = 0.2066
Base-rate model: PD = 625 / 10,000 = 6.25% for everyone, log loss = 0.2338
McFadden pseudo R2 = 1 − 0.2066 / 0.2338 = 11.6%
Log loss has no natural scale. It rises with the default rate, so a card book at 8 per cent and a mortgage book at 1 per cent cannot be compared on raw log loss. The base-rate model is the yardstick: it knows the default rate and nothing else. A PD model that cannot beat it has learnt nothing about who defaults. An 11.6 per cent pseudo R2 looks modest beside the R2 of a linear regression, and is ordinary for credit data, where most of what decides a default is not known at application.
| PD | If it defaults | If it repays |
|---|---|---|
| 20.0% | 1.6094 | 0.2231 |
| 1.0% | 4.6052 | 0.0101 |
| 0.1% | 6.9078 | 0.0010 |
At a PD of 1 per cent, one default costs as much as 458 loans that repay. Push the lowest band from 0.5 to 0.1 per cent and its six defaults raise that band's contribution from 36.77 to 42.44, lifting the whole sample to 0.2072. This is why a PD floor protects a model's log loss, and why a model that assigns near-zero PDs to a segment that then defaults is punished hard.
| Change to the PDs | Log loss | Pseudo R2 |
|---|---|---|
| All PDs halved | 0.2197 | 6.0% |
| All PDs × 0.8 | 0.2082 | 10.9% |
| As fitted | 0.2066 | 11.6% |
| All PDs × 1.25 | 0.2086 | 10.8% |
| All PDs doubled | 0.2309 | 1.2% |
| Log-odds spread × 0.5 (underconfident: 1.797% to 11.4%) | 0.2128 | |
| Log-odds spread × 1.5 (overconfident: 0.138% to 32.6%) | 0.2133 | |
| Log-odds spread × 2.0 (overconfident: 0.038% to 48.4%) | 0.2340 |
The ranking is identical in every row, so Gini and AUC would not move at all. Log loss does, because it is a proper scoring rule: it is minimised by the true probabilities, here at a scale of exactly 1.000. Doubling the spread of the log-odds keeps the ranking and destroys the probabilities: at 0.2340 the model scores worse than the base rate, despite ranking as well as before. An unregularised boosted tree can drift in exactly that direction.
The calibration sheet that solves an intercept shift for a mis-levelled card is in the free workbooks for this book.
There is no universal threshold, because log loss depends on the default rate. Compare it with the base-rate model that predicts the portfolio default rate for everyone. On an illustrative book defaulting at 6.25 per cent, that benchmark scores 0.2338; a model at 0.2066 improves on it by 11.6 per cent in McFadden terms. A model at or above the benchmark adds nothing.
The cost of a default is minus the log of its PD, which has no ceiling. A default on a loan scored at 20 per cent costs 1.6094; at 1 per cent it costs 4.6052, and at 0.1 per cent 6.9078. One default at 1 per cent weighs as much as 458 non-defaults at the same PD. It is one practical reason to floor predicted PDs before scoring.
Both are proper scoring rules, minimised by the true probabilities. Log loss weights rare surprises far more heavily, so it is sensitive to very low PDs on loans that default; the Brier score is bounded and gentler. On the illustrative sample log loss is 0.2066 and Brier 0.05514. Report one consistently, beside its base-rate benchmark.
This article is one calculation from Machine Learning for Finance. The book takes the same case from first principles to the decision, chapter by chapter, and every figure it prints is a live formula in the free companion workbooks.
Get the book on Amazon →Free companion files
Also on Amazon UK · Amazon Germany · Amazon France · Amazon Canada
Reading guide: corporate finance, valuation and markets → · All 1340 articles →
If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.