Metrics: `keras_proba.metrics`¶

Calibration metrics. The classes accumulate over batches for model.compile(metrics=...); the functions evaluate a full set of predictions.

Streaming metrics¶

ExpectedCalibrationError ¶

ExpectedCalibrationError(num_bins=15, from_logits=False, name='ece', dtype=None)

Bases: keras.src.metrics.metric.Metric

Streaming Expected Calibration Error over equal-width bins [1].

Accumulates per-bin counts, correct counts and confidence sums over batches, so the result equals expected_calibration_error on all the data seen since the last reset.

Example:

model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",
              metrics=[ExpectedCalibrationError(num_bins=15)])

Parameters:

Name	Description	Default
`num_bins`	Number of equal-width probability bins.	`15`
`from_logits`	Whether `y_pred` contains logits instead of probabilities. A single output unit is treated as a binary sigmoid output.	`False`
`name`	Name of the metric.	`'ece'`
`dtype`	Dtype of the metric state.	`None`

Call arguments

y_true: Integer labels of shape (batch,) or (batch, 1), or one-hot labels of shape (batch, num_classes). y_pred: Probabilities or logits of shape (batch, num_classes). sample_weight: Optional weights of shape (batch,).

doc `class-attribute` ¶

__doc__ = 'Streaming Expected Calibration Error over equal-width bins [1].\n\n    Accumulates per-bin counts, correct counts and confidence sums over\n    batches, so the result equals `expected_calibration_error` on all the data\n    seen since the last reset.\n\n    Example:\n\n    ```python\n    model.compile(optimizer="adam", loss="sparse_categorical_crossentropy",\n                  metrics=[ExpectedCalibrationError(num_bins=15)])\n    ```\n\n    Args:\n        num_bins: Number of equal-width probability bins.\n        from_logits: Whether `y_pred` contains logits instead of probabilities.\n            A single output unit is treated as a binary sigmoid output.\n        name: Name of the metric.\n        dtype: Dtype of the metric state.\n\n    Call arguments:\n        y_true: Integer labels of shape `(batch,)` or `(batch, 1)`, or one-hot\n            labels of shape `(batch, num_classes)`.\n        y_pred: Probabilities or logits of shape `(batch, num_classes)`.\n        sample_weight: Optional weights of shape `(batch,)`.\n    '

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

module `class-attribute` ¶

__module__ = 'keras_proba.metrics.calibration'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

BrierScore ¶

BrierScore(from_logits=False, name='brier_score', dtype=None)

Bases: keras.src.metrics.reduction_metrics.Mean

Streaming mean Brier score, sum_i p[i]^2 - 2 p[label] (see brier_score).

Parameters:

Name	Description	Default
`from_logits`	Whether `y_pred` contains logits instead of probabilities. A single output unit is treated as a binary sigmoid output.	`False`
`name`	Name of the metric.	`'brier_score'`
`dtype`	Dtype of the metric state.	`None`

Call arguments

y_true: Integer labels of shape (batch,) or (batch, 1), or one-hot labels of shape (batch, num_classes). y_pred: Probabilities or logits of shape (batch, num_classes). sample_weight: Optional weights of shape (batch,).

doc `class-attribute` ¶

__doc__ = 'Streaming mean Brier score, `sum_i p[i]^2 - 2 p[label]` (see `brier_score`).\n\n    Args:\n        from_logits: Whether `y_pred` contains logits instead of probabilities.\n            A single output unit is treated as a binary sigmoid output.\n        name: Name of the metric.\n        dtype: Dtype of the metric state.\n\n    Call arguments:\n        y_true: Integer labels of shape `(batch,)` or `(batch, 1)`, or one-hot\n            labels of shape `(batch, num_classes)`.\n        y_pred: Probabilities or logits of shape `(batch, num_classes)`.\n        sample_weight: Optional weights of shape `(batch,)`.\n    '

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

module `class-attribute` ¶

__module__ = 'keras_proba.metrics.calibration'

str(object='') -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to 'strict'.

Functions¶

expected_calibration_error ¶

expected_calibration_error(num_bins, logits=None, labels_true=None, labels_predicted=None)

Computes the Expected Calibration Error (ECE).

Implements equation (3) of [1], using the probability of the decided label being correct. ECE estimates are unreliable when some of the num_bins equal-width bins contain few samples; see also expected_calibration_error_quantiles.

Parameters:

Name	Description	Default
`num_bins`	Number of equal-width probability bins, e.g. 10.	required
`logits`	Tensor of shape `(n, num_classes)`.	`None`
`labels_true`	Integer tensor of shape `(n,)` with class labels.	`None`
`labels_predicted`	Integer tensor of shape `(n,)` with the decisions of the predictive system. If `None`, the argmax of `logits` is used.	`None`

Returns:

Type	Description
	Scalar ECE.

expected_calibration_error_quantiles ¶

expected_calibration_error_quantiles(hit, pred_log_prob, num_buckets=20, axis=0, log_space_buckets=False)

Expected calibration error over equal-count buckets of predicted probability.

A model is calibrated if buckets of predicted probabilities have the same within-bucket average accuracy. The ECE is the average absolute difference between predicted probability and bucket accuracy:

bucket_weight = bucket_count / sum(bucket_count)
bucket_error = abs(bucket_accuracy - bucket_confidence)
ece = sum(bucket_weight * bucket_error)

where buckets are num_buckets-quantiles of exp(pred_log_prob).

Example:

hit = [False, False, True, False, True, True]
log_pred = np.log([0.1, 0.05, 0.5, 0.2, 0.99, 0.99])
ece, acc, conf, cnt, edges, bucket = expected_calibration_error_quantiles(hit, log_pred, num_buckets=3)
# ece  ==> 0.145
# acc  ==> [0., 0., 1.]
# conf ==> [0.075, 0.2, 0.826665]
# cnt  ==> [2., 1., 3.]

Parameters:

Name	Description	Default
`hit`	Boolean tensor, `True` where the model prediction was correct. Must broadcast with `pred_log_prob`.	required
`pred_log_prob`	The model's predicted log probability for each `hit`.	required
`num_buckets`	Number of quantile buckets.	`20`
`axis`	Axis over which to compute buckets and aggregate statistics.	`0`
`log_space_buckets`	Whether to compute bucket edges from `pred_log_prob` instead of `exp(pred_log_prob)`.	`False`

Returns:

Name	Type	Description
`ece`		Expected calibration error, with the shape of `pred_log_prob` reduced over `axis`.
`bucket_accuracy`		Within-bucket average hits, of shape `(num_buckets, ...)`.
`bucket_confidence`		Within-bucket average probability, of shape `(num_buckets, ...)`.
`bucket_count`		Number of observations per bucket, of shape `(num_buckets, ...)`.
`bucket_pred_log_prob`		Bucket edges in log space, of shape `(num_buckets + 1, ...)`.
`bucket`		Bucket of each observation, with the shape of `pred_log_prob`.

brier_score ¶

brier_score(labels, logits)

Computes the Brier score for a probabilistic prediction.

For a probability vector p and a realized outcome k, the Brier score is sum_i p[i]^2 - 2 p[k]. Smaller values are better. It is a strictly proper scoring rule, and can be negative. This form differs from sum_i (p[i] - onehot(k)[i])^2 by a constant of 1.

Parameters:

Name	Type	Description	Default
`labels`		Integer tensor of shape `(N1, ..., Nk)` with class labels.	required
`logits`		Tensor of shape `(N1, ..., Nk, num_classes)`.	required

Returns:

Type	Description
	Tensor of shape `(N1, ..., Nk)` with elementwise Brier scores; reduce
	with a mean over examples.

brier_decomposition ¶

brier_decomposition(labels, logits)

Decomposes the Brier score into uncertainty, resolution, and reliability.

Any proper scoring rule admits a unique decomposition as Score = Uncertainty - Resolution + Reliability, where:

Uncertainty is a generalized entropy of the average predictive distribution; it can be positive or negative.
Resolution is a generalized variance of individual predictive distributions; it is always non-negative, and larger is better.
Reliability measures calibration of predictions against the true frequency of events; it is always non-negative, and lower is better.

The space of predictive distributions is discretized by their argmax. The estimation error of each component is O(k/n) for n instances and k labels.

Note: TFP's implementation computes the resolution as sum_k w_k sum_j (pbar[k] - dist_mean[k, j])^2, indexing the average label distribution by the predicted class. Here it is sum_k w_k sum_j (pbar[j] - dist_mean[k, j])^2, for which the decomposition reproduces the Brier score exactly when all predictions with the same argmax are identical. Uncertainty and reliability match TFP.

Parameters:

Name	Type	Description	Default
`labels`		Integer tensor of shape `(..., n)` with class labels.	required
`logits`		Tensor of shape `(..., n, num_classes)`.	required

Returns:

Type	Description
	Tuple of `uncertainty`, `resolution` and `reliability`, each of shape
	`(...)`.

Metrics: keras_proba.metrics¶

Streaming metrics¶

ExpectedCalibrationError ¶

__doc__ class-attribute ¶

__module__ class-attribute ¶

BrierScore ¶

__doc__ class-attribute ¶

__module__ class-attribute ¶

Functions¶

expected_calibration_error ¶

expected_calibration_error_quantiles ¶

brier_score ¶

brier_decomposition ¶

Metrics: `keras_proba.metrics`¶

doc `class-attribute` ¶

module `class-attribute` ¶

doc `class-attribute` ¶

module `class-attribute` ¶