accuracy_random_points#

bayesflow.diagnostics.metrics.accuracy_random_points(estimates: Mapping[str, ndarray] | ndarray, targets: Mapping[str, ndarray] | ndarray, variable_keys: Sequence[str] = None, variable_names: Sequence[str] = None, references: ndarray = None, resolution: int = 20, standardize: bool = True, distance: Callable = None, rng: Generator = None) → dict[source]#

Computes the TARP (Test of Accuracy with Random Points) diagnostic.

TARP is a global diagnostic which checks whether the estimated posteriors are calibrated by measuring expected coverage probabilities (ECP) across credibility levels [1].

When no reference points are provided, they are generated as a derangement of the target parameters via a random cyclic shift (a pure permutation with no fixed points). For full control, e.g. to draw references from an independent prior sample, pass explicit references.

References: [1] Lemos et al., 2023, https://proceedings.mlr.press/v202/lemos23a/lemos23a.pdf

Parameters:
estimatesnp.ndarray or dict[str, np.ndarray]

The random draws from the approximate posteriors, either as a NumPy array of shape (num_datasets, num_draws, num_variables) or as a dictionary mapping variable names to arrays, over num_datasets.

targetsnp.ndarray or dict[str, np.ndarray]

The corresponding ground-truth values sampled from the prior, either as a NumPy array of shape (num_datasets, num_variables) or as a dictionary mapping variable names to arrays.

variable_keysSequence[str], optional (default = None)

Select keys from the dictionaries provided in estimates and targets. By default, select all keys.

variable_namesSequence[str], optional (default = None)

Optional variable names to show in the output.

referencesnp.ndarray, optional (default = None)

Reference points of shape (num_datasets, num_variables). If None, reference points are generated automatically as a derangement of targets via a random cyclic shift (i.e. a pure permutation with no fixed points): a random offset in [1, num_datasets) is drawn and np.roll is applied along the dataset axis, so each dataset is assigned a reference from a different dataset.

resolutionint, optional, default: 20

The number of credibility intervals (CIs) to consider

standardizebool, optional (default = True)

Whether to normalize parameters to [0, 1] before computing distances.

distancecallable, optional (default = L2)

Distance function accepting two arrays of shape (..., D) and returning distances of shape (...,). Defaults to Euclidean (L2).

rngnp.random.Generator, optional (default = None)
Returns:
resultdict

Dictionary containing:

  • “atc”float

    Area to curve: signed difference between ECP and the diagonal for alpha > 0.5. Values near 0 indicate calibration; positive values indicate overdispersion; negative values indicate underdispersion.

  • “ks_pvalue”float

    p-value of a two-sample Kolmogorov-Smirnov test between ECP and alpha. Values close to 1 indicate calibration.

  • “metric_name”str

    The name of the metric (“TARP”).

  • “variable_names”str

    The (inferred) variable names.