expected_calibration_error

bayesflow.diagnostics.metrics.expected_calibration_error(estimates: ndarray, targets: ndarray, model_names: Sequence[str] = None, num_bins: int = 10, return_probs: bool = False) → dict[str, Any]

Estimates the expected calibration error (ECE) of a model comparison network according to [1].

[1] Naeini, M. P., Cooper, G., & Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 29, No. 1).

Parameters:

estimatesarray of shape (num_sim, num_models)

The predicted posterior model probabilities.

targetsarray of shape (num_sim, num_models)

The one-hot-encoded true model indices.

model_namesSequence[str], optional (default = None)

Optional model names to show in the output. By default, models are called “M_” + model index.

num_binsint, optional, default: 10

The number of bins to use for the calibration curves (and marginal histograms). Passed into bayesflow.utils.calibration_curve().

return_probsbool (default = False)

Do you want to obtain the output of bayesflow.utils.calibration_curve()?

Returns:

resultdict

Dictionary containing: - “values” : np.ndarray

The expected calibration error per model

• “metric_name”str

  The name of the metric (“Expected Calibration Error”).

• “model_names”str

  The (inferred) variable names.

• “probs_true”: (optional) list[np.ndarray]:

  Outputs of bayesflow.utils.calibration.calibration_curve() per model

• “probs_pred”: (optional) list[np.ndarray]:

  Outputs of bayesflow.utils.calibration.calibration_curve() per model

Notes

Make sure that targets are one-hot encoded classes (i.e., model indices)!