metrics

Numerical diagnostics for evaluating trained Workflows.

Functions

accuracy(predictions, true_models[, ...])	Compute classification accuracy for model comparison networks.
accuracy_random_points(estimates, targets[, ...])	Computes the TARP (Test of Accuracy with Random Points) diagnostic.
bootstrap_comparison(observed_samples, ...)	Computes the distance between observed and reference samples and generates a distribution of null sample distances by bootstrapping for hypothesis testing.
brier_score(estimates, targets[, model_names])	Computes the (multi-class) Brier score of a model comparison network [1].
calibration_error(estimates, ...)	Computes an aggregate score for the marginal calibration error over an ensemble of approximate posteriors.
calibration_log_gamma(estimates, targets[, ...])	Compute the log gamma discrepancy statistic to test posterior calibration, see [1] for additional information.
classifier_two_sample_test(estimates, targets)	C2ST metric [1] between samples from two distributions computed using a neural classifier.
correlation(estimates, ...)	Computes the Pearson correlation between estimates and targets for each random draw from the posterior distribution across datasets, separately for each variable.
expected_calibration_error(estimates, targets)	Estimates the expected calibration error (ECE) of a model comparison network according to [1].
gamma_discrepancy(ranks[, num_post_draws])	Quantifies deviation from uniformity by the likelihood of observing the most extreme point on the empirical CDF of the given rank distribution according to [1] (equation 7).
gamma_null_distribution(num_ranks[, ...])	Computes the distribution of expected gamma values under uniformity of ranks.
posterior_contraction(estimates, ...)	Computes the posterior contraction (PC) from prior to posterior for the given samples.
posterior_z_score(estimates, ...)	Computes the posterior z-score from prior to posterior for the given samples according to [1]:
root_mean_squared_error(estimates, ...)	Computes the (Normalized) Root Mean Squared Error (RMSE/NRMSE) for the given posterior and prior samples.
summary_space_comparison(observed_data, ...)	Computes the distance between observed and reference data in the summary space and generates a distribution of distance values under the null hypothesis to assess model misspecification.