calibration

js/ml/metrics/calibration.ts

Calibration metrics: whether predicted probabilities mean what they claim.

A model can rank perfectly and still be badly calibrated — if everything it calls "90% likely" happens 60% of the time, rocAuc will not notice but every downstream decision built on that number will be wrong.

Types

type BinStrategy = 'uniform' | 'quantile'

How predictions are grouped into calibration bins.

uniform splits [0, 1] into equal-width bins, which keeps the bin edges interpretable. quantile splits into bins holding equal numbers of predictions, which keeps each estimate equally reliable when predictions cluster.

Interfaces

interface CalibrationOptions {

Options for binned calibration measurements.

Properties

bins?: number

Number of bins. Defaults to 10.

strategy?: BinStrategy

Binning strategy. Defaults to uniform.

positiveLabel?: Label

Which label the probabilities refer to. Inferred as 1 or true when the labels allow it.

interface CalibrationBin {

One bin of a calibration curve.

Properties

lowerEdge: number

Inclusive lower edge of the bin's predicted-probability range.

upperEdge: number

Upper edge, exclusive except in the final bin.

count: number

Number of predictions that fell in the bin.

meanPredicted: number

Mean predicted probability among them.

fractionPositive: number

Observed fraction that were actually positive.

Functions

function brierScore( yTrue: ArrayLike<Label>, probabilities: ArrayLike<number>, options: { positiveLabel?: Label } = {}, ): number

Mean squared error between predicted probabilities and outcomes.

0 is perfect and 0.25 is what always guessing 0.5 earns. Unlike logLoss it stays finite for a confidently wrong prediction, so a single bad call cannot dominate the average.

import { brierScore } from 'fino:ml/metrics';

console.log(brierScore([1, 0, 1], [0.9, 0.1, 0.8]).toFixed(4)); // 0.0200

function calibrationCurve( yTrue: ArrayLike<Label>, probabilities: ArrayLike<number>, options: CalibrationOptions = {}, ): CalibrationBin[]

Group predictions into bins and report predicted versus observed rates.

A well-calibrated model produces bins where meanPredicted tracks fractionPositive. Empty bins are omitted.

import { calibrationCurve } from 'fino:ml/metrics';

const bins = calibrationCurve([0, 0, 1, 1], [0.1, 0.2, 0.8, 0.9], { bins: 2 });
console.log(bins.map((bin) => bin.fractionPositive)); // [0, 1]

function expectedCalibrationError( yTrue: ArrayLike<Label>, probabilities: ArrayLike<number>, options: CalibrationOptions = {}, ): number

Expected calibration error: mean gap between confidence and reality, weighted by how many predictions land in each bin.

0 is perfectly calibrated. This is the single number to watch when a downstream system thresholds on the probability itself.

import { expectedCalibrationError } from 'fino:ml/metrics';

console.log(expectedCalibrationError([0, 0, 1, 1], [0.1, 0.2, 0.8, 0.9], { bins: 2 }));

function maximumCalibrationError( yTrue: ArrayLike<Label>, probabilities: ArrayLike<number>, options: CalibrationOptions = {}, ): number

Largest calibration gap in any populated bin — the worst case rather than the average.

import { maximumCalibrationError } from 'fino:ml/metrics';

console.log(maximumCalibrationError([0, 1], [0.5, 0.5], { bins: 2 })); // 0