Raw classifier scores are not probabilities. A fraud model outputting 0.9 does not mean 90% chance of fraud — it means "ranked high relative to training data," which is a different thing. calibrate-ml turns those scores into actual probabilities with Platt scaling (parametric sigmoid fit) or isotonic regression (non-parametric step function), and measures what's left with ECE and Brier score.
pip install -e . # from sourcefrom calibrateml import platt_scale, isotonic_calibrate, ece, brier_score, reliability_diagram
# Fit on a held-out calibration set (not the training set)
cal = platt_scale(y_cal_true, y_cal_score) # or: isotonic_calibrate(...)
# Apply to new scores
probs = cal(y_test_score)
# Measure residual miscalibration
print(f"ECE: {ece(y_test_true, probs):.4f}")
print(f"Brier: {brier_score(y_test_true, probs):.4f}")
# ASCII reliability diagram
print(reliability_diagram(y_test_true, probs))| Function | Returns |
|---|---|
platt_scale(y_true, y_score) |
Callable calibrate(scores); fits sigmoid A, B via gradient descent with Platt correction |
isotonic_calibrate(y_true, y_score) |
Callable calibrate(scores); fits PAVA step function, output interpolated |
brier_score(y_true, y_prob) |
Mean squared error; 0 = perfect, 1 = worst |
ece(y_true, y_prob, n_bins=10) |
Bin-weighted mean |
reliability_diagram(y_true, y_prob) |
Multiline ASCII string; diagonal = perfect calibration |
Platt scaling fits one sigmoid (two parameters: A, B). Works well when raw scores are already roughly monotone with a sigmoid-shaped relationship to true probability — common with SVM outputs and gradient-boosted trees. Works from small calibration sets (≥100 examples).
Isotonic regression fits an unconstrained step function via PAVA. Flexible enough to handle arbitrary score distributions, but needs larger calibration sets (≥1000) to avoid overfitting the steps. Preferred for neural nets with non-sigmoid output shapes.
- Calibrate on a held-out set, never the training data.
- ECE < 0.05 is well-calibrated for most risk/fraud applications.
- Brier score combines calibration and discrimination; ECE isolates calibration quality alone.
- Dependencies: NumPy only.