Articles

How do you calculate the Brier score for a PD model?

The Brier score measures probability accuracy, level as well as ranking, but its size depends on the default rate, so it has to be decomposed before anyone concludes the model got worse.

Julian R. Sterling · from Machine Learning for Finance

The Brier score is the average squared difference between each loan's predicted PD and what happened, 1 for a default and 0 otherwise: BS = Σ(PD − outcome)² / N, and lower is better. On the development sample of the credit scorecard in Machine Learning for Finance it is 0.05588; out of time it nearly doubles to 0.10505. Decomposed, almost all of that rise is the population's default rate doubling, not the model getting worse: miscalibration adds only 0.00285.

Worked in full in Machine Learning for Finance by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →

Why a validator asks for it

Gini and KS measure ranking only: multiply every PD by two and they do not move. The Brier score is one of the few single numbers that punishes a wrong level as well as a wrong order, which is why it sits next to Gini in model validation reports, and why it is the usual loss for comparing a logistic card with a boosted tree on probability, not just on rank. Its weakness is that its size depends on the default rate of the sample, so a raw Brier score can only be compared with another one on the same population.

The data

The book's fictional lender scores small-business loans with a seven-characteristic card. The development sample has 6,000 loans and 379 bads, a 6.32 per cent bad rate; the out-of-time sample has 2,400 loans and 303 bads, 12.625 per cent. Both are cut into the same ten score bands, fixed on development, with the average predicted PD in each. The calculation below is done band by band, treating every loan in a band as carrying the band's average PD; with loan-level PDs the formula is identical, only longer.

The calculation, step by step

Development sample: each band's contribution

Band	Loans	Bads	Predicted	Observed	Squared error
1 (worst)	600	133	21.59%	22.17%	103.538
2	599	57	10.78%	9.52%	51.672
3	601	39	7.72%	6.49%	36.561
4	600	36	5.91%	6.00%	33.840
5	600	35	4.72%	5.83%	33.033
6	598	24	3.85%	4.01%	23.038
7	602	20	3.13%	3.32%	19.338
8	600	13	2.47%	2.17%	12.724
9	600	16	1.85%	2.67%	15.613
10 (best)	600	6	1.15%	1.00%	5.941
Total	6,000	379	6.32%	6.32%	335.3

335.3 / 6,000 = 0.05588. The worst band supplies almost a third of it on a tenth of the loans, and the bads supply most of it: a default predicted at 5.56 per cent costs 0.8919, a good loan at the same PD 0.0031.

Reading it: the decomposition

Murphy's decomposition splits the score into three parts, exactly when the PD is constant within each group, as here:

Development, out of time, and out of time after recalibration

Sample	Bad rate	Brier	Reliability	Resolution	Uncertainty	Skill score
Development	6.32%	0.05588	0.00006	0.00335	0.05918	5.57%
Out of time	12.625%	0.10505	0.00290	0.00816	0.11031	4.77%
Out of time, recalibrated	12.625%	0.10227	0.00012	0.00816	0.11031	7.29%

The rise of 0.04917 from development to out of time breaks down into +0.05113 from uncertainty, +0.00285 from reliability and −0.00481 from resolution, which actually improved because the bands spread further apart when defaults rose. The card did not lose its ability to separate; it lost its level, predicting 7.90 per cent on average against 12.625 observed. Shifting every log-odds by one constant, as in how to recalibrate a PD model to a new default rate, removes 0.00278 of reliability error and lifts the skill score from 4.77 to 7.29 per cent. Resolution does not move: recalibration repairs level, never ranking.

Skill scores of 5 to 7 per cent look poor beside a Gini of about 0.47 on the same card. They are not contradictory: with a bad rate in single figures most of the squared error is irreducible, and the Brier score is measuring that, not the ranking. Compare Brier scores between models on the same loans, never across portfolios.

The common mistakes

• Comparing raw Brier scores across samples. 0.05588 against 0.10505 reads as a model that halved in quality. The skill score and the decomposition show a population that doubled its default rate.
• Using a score or a band rank instead of a probability. The Brier score needs PDs between 0 and 1. A score of 568.85 has to be converted first.
• Ignoring the naive benchmark. Giving every out-of-time loan the development rate of 6.32 per cent scores 0.11429, worse than predicting the true average and worse than the model's 0.10505. If a model cannot beat that line, it is not adding information about level.

Takeaway

• Brier = mean of (PD − outcome)²: 0.05588 in development, 0.10505 out of time.
• Decompose before concluding: here uncertainty is +0.05113 of the 0.04917 rise.
• Reliability is the part a recalibration can fix; resolution is the part only a better model can.
• Report the skill score beside the raw score so a reader can compare across years.

The ten bands, their predicted and observed rates and the recalibration shift solved on the calibration sheet are in the free workbooks for this case; the ranking side of the same card is in how to calculate Gini and KS for a credit scorecard.

Questions readers ask

What is a good Brier score for a credit model?

There is no universal threshold, because the score depends on the default rate. Predicting the average rate for everyone scores the rate times one minus the rate: 0.05918 at a 6.32 per cent bad rate. A useful model beats that; the illustrative card scores 0.05588, a skill score of 5.57 per cent. Judge models against each other on the same loans.

What is the difference between the Brier score and AUC?

AUC, and Gini, measure ranking only and ignore the level of the PDs; the Brier score penalises both. Doubling every PD leaves AUC unchanged but changes the Brier score. On the illustrative card the out-of-time bands still rank almost in order while reliability error rises from 0.00006 to 0.00290, which only the Brier decomposition shows.

How do you calculate the Brier skill score?

Brier skill score = 1 minus the model's Brier score divided by the score of a reference forecast, usually the sample default rate given to every loan, which equals rate times one minus rate. Out of time the illustrative card scores 0.10505 against 0.11031, a skill score of 4.77 per cent, rising to 7.29 per cent after recalibration.

Also on this site

• How do you run a Hosmer-Lemeshow test on a PD model?Sum (observed minus expected defaults) squared over N x PD x (1 minus PD) in ten groups: 10.98, p of 0.2028. It passes a model that under-predicts by 10.7%.
• How do you recalibrate a PD model to a new default rate?Shift every applicant's log-odds by one constant, solved so mean PD hits the new rate: 7.90% to 12.625% takes 0.5623, or 16.22 scorecard points.
• How do you calculate Gini and KS for a credit scorecard?Gini = 2 x AUC minus 1, with tied bads counted half; KS is the widest gap in cumulative shares. Ten score bands give 45.61% and 32.47%, worked step by step.
• How do you calculate PD from a logistic regression model?Log-odds z = intercept + coefficient x WoE summed; PD = 1 / (1 + e^-z). An illustrative applicant: z of -2.8324, a PD of 5.56%, a score of 568.85.
• How do you calculate the population stability index (PSI)?Sum (new share minus old share) x ln(new / old) over fixed score bands: 0.090517 for a card whose bad rate doubled. Under 0.10 is not proof of stability.
• How do you calculate scorecard points from PDO, factor and offset?Factor = PDO / ln 2, offset = anchor score minus factor x ln(odds): 20 points to double the odds and 600 at 50:1 give 28.8539 and 487.1229.

Reading guide: corporate finance, valuation and markets → · All 1340 articles →

If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.