Articles

How do you calculate Cohen's kappa for two analysts' ratings?

Measuring whether two due diligence analysts rate managers the same way, with the chance correction that raw agreement leaves out.

Julian R. Sterling · from Operational Due Diligence in Private Equity

Cohen's kappa measures how far two reviewers agree beyond what chance would produce: κ = (p_o − p_e) ÷ (1 − p_e), where p_o is the share of items they rate the same and p_e the share they would match on by chance given their own rating habits. Two operational due diligence analysts who agree on 82.5 per cent of 40 domain ratings, against 45.9 per cent expected by chance, have a kappa of 0.68: substantial, but not the near-certainty the raw figure suggests.

Worked in full in Operational Due Diligence in Private Equity by Julian R. Sterling, with every figure reproduced in a free workbook.See the book on Amazon →

An ODD programme that relies on one analyst's rating is relying on that analyst's calibration. The usual defence is a second reviewer. Whether the second reviewer adds anything depends on a number most programmes never compute: how often two analysts, looking at the same evidence, reach the same rating. Raw agreement overstates it, because if most domains are rated satisfactory, two analysts will agree most of the time without trying. Kappa removes that.

The assumptions

Two analysts independently rate the same five managers on eight domains each, 40 ratings in all, on a three-point scale. Rows are the first analyst, columns the second.

Illustrative ratings, analyst 1 (rows) against analyst 2 (columns)

Analyst 1 \ Analyst 2	Satisfactory	Watch	Concern	Total
Satisfactory	22	3	0	25
Watch	2	7	1	10
Concern	0	1	4	5
Total	24	11	5	40

The calculation, step by step

Step 1: observed agreement. The diagonal.

Step 2: agreement expected by chance. If each analyst kept their own mix of ratings but assigned them at random, the chance that both say Satisfactory is 25/40 × 24/40, and so on for each category.

Step 3: kappa.

On the bands most often quoted, from Landis and Koch, 0.61 to 0.80 is substantial agreement and 0.81 to 1.00 almost perfect. To reach 0.80 with the same rating mix, the pair would have to agree on 89.2 per cent of ratings: 36 of 40 instead of 33.

The result

The two analysts agree well beyond chance, and the disagreements are all one notch apart: no domain is Satisfactory for one and Concern for the other. But kappa over the whole scale is not the number an investment committee cares most about. The decision that matters is whether a domain is a Concern, because that is where a veto applies. Collapsing the table to Concern against everything else, the analysts agree on 4 Concerns, each flags 1 that the other does not, and they agree on 34 non-Concerns: 95.0 per cent observed, 78.1 per cent by chance, a kappa of 0.77. Of the 6 domains either analyst flagged as a Concern, 2 were flagged by one analyst only. With a single reviewer, one of those would have gone unnoticed.

What if: most ratings are satisfactory

Two programmes compared

Measure	Base programme	Skewed programme
Ratings agreed, of 40	33	36
Observed agreement	82.5%	90.0%
Chance agreement	45.9%	81.6%
Kappa	0.68	0.46
Linear weighted kappa	0.75	0.56

In the skewed programme both analysts rate 36 of 40 domains Satisfactory, 3 Watch and 1 Concern, and they agree on 36. Raw agreement is higher than in the base case, yet kappa falls to 0.46, moderate, because with that mix chance alone gives 81.6 per cent. This is the trap of a programme that rates nearly everything satisfactory: the agreement figure looks excellent precisely because the ratings carry little information. Kappa tells the head of ODD that the second review, in that programme, is mostly confirming defaults.

Weighted kappa gives half credit to one-notch disagreements. It raises the base case from 0.68 to 0.75 and is the right measure for an ordered scale, but report the Concern-only kappa beside it: in operational due diligence the near miss between Watch and Concern is the costly one.

The common mistake

Reporting percentage agreement and calling the rating process consistent. 90.0 per cent sounds reassuring and can mean less than 82.5. The second mistake is computing kappa on reviews that were not independent: if the second analyst has seen the first one's draft, agreement measures deference, not calibration. Run the exercise on a sample of reviews rated blind, at least once a year, and keep the confusion table, because the off-diagonal cells show which domains and which rating boundaries need a clearer definition.

Takeaway

• κ = (p_o − p_e) ÷ (1 − p_e), with p_e from each analyst's own rating mix.
• On the illustrative 40 ratings, 82.5 per cent agreement is a kappa of 0.68; 90.0 per cent in a skewed programme is only 0.46.
• Compute it separately for the Concern category, where a disagreement changes a decision.

The book asks whether two analysts would draw the line in the same place and builds a scorecard that records the second reviewer's view; the free workbook for this case measures how many managers sit one notch from a different recommendation. Why adding domains to a rubric weakens it is worked in why refining an ODD rubric makes it weaker.

Questions readers ask

What is a good Cohen's kappa for due diligence ratings?

The bands most often quoted, from Landis and Koch, call 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect. The illustrative pair of ODD analysts scores 0.68. To reach 0.80 with the same rating mix they would need to agree on 89.2 per cent of ratings, 36 of 40 rather than 33.

Why can 90 per cent agreement give a low kappa?

Because when nearly every rating is the same, agreement is easy by chance. In the illustrative skewed programme both analysts rate 36 of 40 domains satisfactory, so chance alone produces 81.6 per cent agreement. Their 90.0 per cent beats that by little, and kappa is 0.46, lower than the 0.68 of a pair agreeing only 82.5 per cent of the time.

What is weighted kappa and when should ODD teams use it?

Weighted kappa gives partial credit to near misses, so a Satisfactory against a Watch counts as half an agreement while Satisfactory against Concern counts as none. On the illustrative ratings it rises from 0.68 to 0.75. Use it for ordered scales, but report the plain figure for the Concern category, where a near miss is the one that matters.

Also on this site

• Why refining an operational due diligence rubric makes it weakerThree of twelve managers carry a terminal flaw and the averaging model approves all three. Splitting eight domains into sixteen makes it easier to hide, not harder.
• How many managers can one ODD analyst cover a year?Size the team on hours, not headcount ratios. 12 new reviews and 60 monitored managers take 2,144 hours: 1.79 analysts, 40.3 managers but 6.7 new reviews each.
• How is audit materiality calculated for a private equity fund?A percentage of net assets: at 1% of 800.0, materiality is 8.0 and performance materiality 6.0, enough to hide a 6.7% mis-mark on the largest holding.
• How long can a GP survive if its next fund does not close?Project fees through the step-down with no successor fund. Here fees fall 30.1 per cent in 2029, cash runs out that year, and break-even needs a 38.3 per cent cut.
• How much should a GP commit to its own fund?Measure own cash, not the headline. A 2 per cent, $15.0m GP commitment shrinks to $3.0m once fee waiver and guaranteed loans are removed: 2.4 months of fees.
• How should broken deal expenses be allocated to co-investors?By their share of closed-deal equity. At 30 per cent co-investment and 3.0m of dead deals a year, charging the fund costs its LPs 4.5m over five years.

Reading guide: private equity and private markets → · All 1340 articles →

If this book helped, or didn’t, a few lines on Amazon are worth more than they look: they are what the next reader goes on. Write a review. The workbook stays free either way.