calibration_ecdf_from_quantiles#

bayesflow.diagnostics.calibration_ecdf_from_quantiles(estimates: Mapping[str, Mapping[str, ndarray]], targets: Mapping[str, ndarray], quantile_levels: Sequence[float], quantiles_key: str = 'quantiles', variable_keys: Sequence[str] = None, variable_names: Sequence[str] = None, difference: bool = False, stacked: bool = False, figsize: Sequence[float] = None, label_fontsize: int = 16, legend_fontsize: int = 14, legend_location: str = 'upper right', title_fontsize: int = 18, tick_fontsize: int = 12, rank_ecdf_color: str = '#132a70', fill_color: str = 'grey', num_row: int = None, num_col: int = None, **kwargs) → Figure[source]#

Creates the empirical CDFs for each marginal rank distribution and plots it against a uniform ECDF.

For models with many parameters, use stacked=True to obtain an idea of the overall calibration of a posterior approximator.

Note: In contrast to the related calibration_ecdf() function, this does not use simultaneous confidence bands. Confidence bands apply to each quantile level separately.

[1] Säilynoja, T., Bürkner, P. C., & Vehtari, A. (2022). Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison. Statistics and Computing, 32(2), 1-21. https://arxiv.org/abs/2103.10522

[2] Lemos, Pablo, et al. “Sampling-based accuracy testing of posterior estimators

for general inference.” International Conference on Machine Learning. PMLR, 2023. https://proceedings.mlr.press/v202/lemos23a.html

Parameters:
estimatesdict[str, dict[str, np.ndarray]]

The model-generated estimates in a nested dictionary, e.g. as returned by approximator.estimate(conditions=…). - The outer keys identify the inference variable. - The inner keys identify point estimates.

Select which one to treat as quantile predictions with argument quantiles_key.

  • The inner value is an ndarray of shape (num_datasets, point_estimate_size, variable_block_size)

targetsdict[str, np.ndarray]

The prior draws (true parameters) used for generating the num_datasets

quantile_levelslist of floats

The target quantile levels that the quantile predictions should be tested against.

quantiles_keystr, optional, default: “quantiles”

Selects which estimate to treat as quantile predictions with argument quantiles_key.

variable_keyslist or None, optional, default: None

Select keys from the dictionaries provided in estimates and targets. By default, select all keys.

variable_nameslist or None, optional, default: None

The parameter names for nice plot titles. Inferred if None. Only relevant if stacked=False.

differencebool, optional, default: False

If True, plots the ECDF difference. Enables a more dynamic visualization range.

stackedbool, optional, default: False

If True, all ECDFs will be plotted on the same plot. If False, each ECDF will have its own subplot, similar to the behavior of calibration_histogram.

figsizetuple or None, optional, default: None

The figure size passed to the matplotlib constructor. Inferred if None.

label_fontsizeint, optional, default: 16

The font size of the y-label and y-label texts

legend_fontsizeint, optional, default: 14

The font size of the legend text

title_fontsizeint, optional, default: 18

The font size of the title text. Only relevant if stacked=False

tick_fontsizeint, optional, default: 12

The font size of the axis ticklabels

rank_ecdf_colorstr, optional, default: ‘#a34f4f’

The color to use for the rank ECDFs

fill_colorstr, optional, default: ‘grey’

The color of the fill arguments.

num_rowint, optional, default: None

The number of rows for the subplots. Dynamically determined if None.

num_colint, optional, default: None

The number of columns for the subplots. Dynamically determined if None.

**kwargsdict, optional, default: {}

Keyword arguments can be passed to control the behavior of ECDF simultaneous band computation through the ecdf_bands_kwargs dictionary. See pointwise_ecdf_bands for keyword arguments.

Returns:
fplt.Figure - the figure instance for optional saving
Raises:
ShapeError

If there is a deviation form the expected shapes of estimates and targets.

ValueError

If an unknown rank_type is passed.