Post-processing: `keras_proba.postprocessing`¶

Post-hoc calibration and conformal prediction for trained models.

Temperature scaling¶

TemperatureScaling ¶

TemperatureScaling(temperature=1.0, **kwargs)

Bases: keras.src.layers.layer.Layer

Divides logits by a learned temperature.

The temperature is stored as a trainable log_temperature weight, so it stays positive. Fit it in closed form on held-out logits with fit, or train it with model.fit on a model whose other layers are frozen.

Parameters:

Name	Type	Description	Default
`temperature`		Initial temperature, a positive float.	`1.0`
`**kwargs`		Keyword arguments for `keras.layers.Layer`.	required

Input shape

Logits of any shape.

Output shape

Same as the input.

doc `class-attribute` ¶

__doc__ = 'Divides logits by a learned temperature.\n\n    The temperature is stored as a trainable `log_temperature` weight, so it\n    stays positive. Fit it in closed form on held-out logits with `fit`, or\n    train it with `model.fit` on a model whose other layers are frozen.\n\n    Args:\n        temperature: Initial temperature, a positive float.\n        **kwargs: Keyword arguments for `keras.layers.Layer`.\n\n    Input shape:\n        Logits of any shape.\n\n    Output shape:\n        Same as the input.\n    '

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

module `class-attribute` ¶

__module__ = 'keras_proba.postprocessing.temperature_scaling'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

temperature `property` ¶

temperature

The current temperature, exp(log_temperature).

fit ¶

fit(logits, labels, **kwargs)

Fits the temperature to held-out logits and labels; see fit_temperature.

Returns:

Type	Description
	The fitted temperature as a Python float.

fit_temperature ¶

fit_temperature(logits, labels, sample_weight=None, temperature_range=(0.05, 20.0), tolerance=1e-07, max_iterations=200)

Finds the temperature that minimizes the NLL of softmax(logits / T).

The NLL is convex in beta = 1 / T, so the minimizer is the root of its non-decreasing derivative, found by bisection in log(beta). This is exact, deterministic, needs no learning rate, and is computed in float64 NumPy, so it is the same on every backend.

Parameters:

Name	Description	Default
`logits`	Held-out logits of shape `(n, num_classes)`, or `(n,)` / `(n, 1)` for binary sigmoid logits.	required
`labels`	Integer labels of shape `(n,)` or one-hot labels.	required
`sample_weight`	Optional weights of shape `(n,)`.	`None`
`temperature_range`	`(min, max)` temperatures searched. If the optimum lies outside, the nearest bound is returned; e.g. on perfectly separable data the NLL keeps decreasing as `T -> 0`.	`(0.05, 20.0)`
`tolerance`	Relative tolerance on the temperature.	`1e-07`
`max_iterations`	Maximum number of bisection steps.	`200`

Returns:

Type	Description
	The fitted temperature as a Python float.

temperature_scaled_nll ¶

temperature_scaled_nll(logits, labels, temperature, sample_weight=None)

Mean negative log-likelihood of softmax(logits / temperature).

Parameters:

Name	Description	Default
`logits`	Array of shape `(n, num_classes)`, or `(n,)` / `(n, 1)` for binary sigmoid logits.	required
`labels`	Integer labels of shape `(n,)` or one-hot labels.	required
`temperature`	Positive temperature.	required
`sample_weight`	Optional weights of shape `(n,)`.	`None`

Returns:

Type	Description
	The (weighted) mean NLL as a Python float.

Split conformal prediction¶

SplitConformalRegressor ¶

SplitConformalRegressor(alpha=0.1)

Split conformal prediction intervals for regression.

Without scale, the score is the absolute residual |y - y_pred| and intervals have a constant width 2 q_hat. With a per-example scale (e.g. the standard deviation of a GaussianHead, MixtureGaussianHead or Monte Carlo samples), the score is |y - y_pred| / scale and intervals y_pred +- q_hat * scale adapt to each input [2]. Each output dimension gets its own threshold, so coverage holds for each dimension separately.

Example:

mean, scale = split_gaussian_params(model.predict(x_cal))
conformal = SplitConformalRegressor(alpha=0.1).fit(y_cal, mean, scale=scale)
mean, scale = split_gaussian_params(model.predict(x_test))
lower, upper = conformal.predict_interval(mean, scale=scale)

Parameters:

Name	Type	Description	Default
`alpha`		Miscoverage level in `(0, 1)`; intervals cover with probability at least `1 - alpha`.	`0.1`

doc `class-attribute` ¶

__doc__ = 'Split conformal prediction intervals for regression.\n\n    Without `scale`, the score is the absolute residual `|y - y_pred|` and\n    intervals have a constant width `2 q_hat`. With a per-example `scale`\n    (e.g. the standard deviation of a `GaussianHead`, `MixtureGaussianHead` or\n    Monte Carlo samples), the score is `|y - y_pred| / scale` and intervals\n    `y_pred +- q_hat * scale` adapt to each input [2]. Each output dimension\n    gets its own threshold, so coverage holds for each dimension separately.\n\n    Example:\n\n    ```python\n    mean, scale = split_gaussian_params(model.predict(x_cal))\n    conformal = SplitConformalRegressor(alpha=0.1).fit(y_cal, mean, scale=scale)\n    mean, scale = split_gaussian_params(model.predict(x_test))\n    lower, upper = conformal.predict_interval(mean, scale=scale)\n    ```\n\n    Args:\n        alpha: Miscoverage level in `(0, 1)`; intervals cover with probability\n            at least `1 - alpha`.\n    '

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

module `class-attribute` ¶

__module__ = 'keras_proba.postprocessing.conformal'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

weakref `property` ¶

__weakref__

list of weak references to the object

fit ¶

fit(y_true, y_pred, scale=None)

Computes the threshold from held-out calibration data.

Parameters:

Name	Description	Default
`y_true`	Calibration targets, of the same shape as `y_pred`.	required
`y_pred`	Point predictions of shape `(n,)` or `(n, units)`.	required
`scale`	Optional positive per-example scales, of the same shape.	`None`

Returns:

Type	Description
	`self`.

predict_interval ¶

predict_interval(y_pred, scale=None)

Returns the prediction intervals (lower, upper) for y_pred.

Parameters:

Name	Type	Description	Default
`y_pred`		Point predictions of shape `(n,)` or `(n, units)`.	required
`scale`		Per-example scales, required if and only if `fit` was given scales.	`None`

Returns:

Type	Description
	Tuple of `lower` and `upper` bounds, each of the shape of `y_pred`.

SplitConformalClassifier ¶

SplitConformalClassifier(alpha=0.1, score='lac', from_logits=False, randomized=True, seed=None)

Split conformal prediction sets for classification.

Scores:

"lac" (least ambiguous set-valued classifier): 1 - p_label. Sets are {k : p_k >= 1 - q_hat}, with exact coverage and the smallest average size [2]; they can be empty for ambiguous inputs.
"aps" (adaptive prediction sets) [3]: the probability mass of the classes ranked before the label, plus u * p_label. Sets grow with the model's uncertainty about each input. With randomized=True (default), u ~ Uniform(0, 1) and sets {k : mass before k + u * p_k <= q_hat} have exact coverage. With randomized=False, u = 1 and the sets {k : mass before k < q_hat} are never empty but conservative: on hard tasks they can cover far more than 1 - alpha.

Example:

conformal = SplitConformalClassifier(alpha=0.1, score="aps").fit(y_cal, model.predict(x_cal))
sets = conformal.predict_set(model.predict(x_test))  # Boolean, (n, num_classes).

Parameters:

Name	Description	Default
`alpha`	Miscoverage level in `(0, 1)`; sets contain the label with probability at least `1 - alpha`.	`0.1`
`score`	`"lac"` or `"aps"`.	`'lac'`
`from_logits`	Whether predictions are logits instead of probabilities. A single output unit is treated as a binary sigmoid output.	`False`
`randomized`	Whether to randomize APS scores and sets for exact coverage. Ignored for `"lac"`.	`True`
`seed`	Seed for the APS randomization.	`None`

doc `class-attribute` ¶

__doc__ = 'Split conformal prediction sets for classification.\n\n    Scores:\n\n    - `"lac"` (least ambiguous set-valued classifier): `1 - p_label`. Sets are\n      `{k : p_k >= 1 - q_hat}`, with exact coverage and the smallest average\n      size [2]; they can be empty for ambiguous inputs.\n    - `"aps"` (adaptive prediction sets) [3]: the probability mass of the\n      classes ranked before the label, plus `u * p_label`. Sets grow with the\n      model\'s uncertainty about each input. With `randomized=True` (default),\n      `u ~ Uniform(0, 1)` and sets `{k : mass before k + u * p_k <= q_hat}`\n      have exact coverage. With `randomized=False`, `u = 1` and the sets\n      `{k : mass before k < q_hat}` are never empty but conservative: on hard\n      tasks they can cover far more than `1 - alpha`.\n\n    Example:\n\n    ```python\n    conformal = SplitConformalClassifier(alpha=0.1, score="aps").fit(y_cal, model.predict(x_cal))\n    sets = conformal.predict_set(model.predict(x_test))  # Boolean, (n, num_classes).\n    ```\n\n    Args:\n        alpha: Miscoverage level in `(0, 1)`; sets contain the label with\n            probability at least `1 - alpha`.\n        score: `"lac"` or `"aps"`.\n        from_logits: Whether predictions are logits instead of probabilities.\n            A single output unit is treated as a binary sigmoid output.\n        randomized: Whether to randomize APS scores and sets for exact\n            coverage. Ignored for `"lac"`.\n        seed: Seed for the APS randomization.\n    '

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

module `class-attribute` ¶

__module__ = 'keras_proba.postprocessing.conformal'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

weakref `property` ¶

__weakref__

list of weak references to the object

fit ¶

fit(y_true, y_pred)

Computes the threshold from held-out calibration labels and predictions.

Parameters:

Name	Type	Description	Default
`y_true`		Integer labels of shape `(n,)` or `(n, 1)`, or one-hot labels.	required
`y_pred`		Probabilities or logits of shape `(n, num_classes)`.	required

Returns:

Type	Description
	`self`.

predict_set ¶

predict_set(y_pred)

Returns boolean prediction sets of shape (n, num_classes).

scores ¶

scores(y_true, y_pred)

Nonconformity scores of the labels y_true, of shape (n,).

conformal_quantile ¶

conformal_quantile(scores, alpha)

The split conformal threshold: the ceil((n + 1) (1 - alpha))-th smallest score.

Parameters:

Name	Type	Description	Default
`scores`		Calibration scores of shape `(n, ...)`; thresholds are computed independently for each trailing position.	required
`alpha`		Miscoverage level in `(0, 1)`.	required

Returns:

Type	Description
	Tensor of shape `scores.shape[1:]`. If `n` is too small for the
	requested coverage (`k > n`), the threshold is `inf`, i.e. every label
	is included.

interval_coverage ¶

interval_coverage(y_true, lower, upper)

Fraction of targets inside [lower, upper], per output dimension for 2-D inputs.

set_coverage ¶

set_coverage(y_true, prediction_sets)

Fraction of labels contained in their boolean prediction sets.

average_set_size ¶

average_set_size(prediction_sets)

Average number of classes in the boolean prediction sets.

Post-processing: keras_proba.postprocessing¶

Temperature scaling¶

TemperatureScaling ¶

__doc__ class-attribute ¶

__module__ class-attribute ¶

temperature property ¶

fit ¶

fit_temperature ¶

temperature_scaled_nll ¶

Split conformal prediction¶

SplitConformalRegressor ¶

__doc__ class-attribute ¶

__module__ class-attribute ¶

__weakref__ property ¶

fit ¶

predict_interval ¶

SplitConformalClassifier ¶

__doc__ class-attribute ¶

__module__ class-attribute ¶

__weakref__ property ¶

fit ¶

predict_set ¶

scores ¶

conformal_quantile ¶

interval_coverage ¶

set_coverage ¶

average_set_size ¶

Post-processing: `keras_proba.postprocessing`¶

doc `class-attribute` ¶

module `class-attribute` ¶

temperature `property` ¶

doc `class-attribute` ¶

module `class-attribute` ¶

weakref `property` ¶

doc `class-attribute` ¶

module `class-attribute` ¶

weakref `property` ¶