bayesflow.computational_utilities module

bayesflow.computational_utilities.posterior_calibration_error(posterior_samples, prior_samples, alpha_resolution=20, aggregator_fun=<function median>, min_quantile=0.005, max_quantile=0.995)

Computes an aggregate score for the marginal calibration error over an ensemble of approximate posteriors. The calibration error is given as the aggregate (e.g., median) of the absolute deviation between an alpha-CI and the relative number of inliers from prior_samples over multiple alphas in (0, 1).

Note: The function will use posterior quantiles for determining the credibility intervals. An alternative definition of the calibration error is possible using highest density intervals (HDIs).

Parameters:

posterior_samplesnp.ndarray of shape (num_datasets, num_draws, num_params)

The random draws from the approximate posteriors over num_datasets

prior_samplesnp.ndarray of shape (num_datasets, num_params)

The corresponding ground-truth values sampled from the prior

alpha_resolutionint, optional, default: 20

The number of credibility intervals (CIs) to consider

aggregator_funcallable or None, optional, default: np.median

The function used to aggregate the marginal calibration errors. If None provided, the per-alpha calibration errors will be returned.

min_quantilefloat in (0, 1), optional, default: 0.005

The minimum posterior quantile to consider

max_quantilefloat in (0, 1), optional, default: 0.995

The maximum posterior quantile to consider

Returns:

calibration_errorsnp.ndarray of shape (num_params, ) or (alpha_resolution, num_params),

if aggregator_fun is None. The aggregated calibration error per marginal posterior.

bayesflow.computational_utilities.compute_jacobian_trace(function, inputs, **kwargs)

Computes the exact Jacobian Trace of function with respect to inputs. Suitable for low dimensions (<32)

Parameters:

functioncallable

The function whose Jacobian at inputs will be computed

inputstf.Tensor of shape (batch_size, …)

The tensor with respect to which we are computing the Jacobian

Returns:

outputstf.Tensor of shape (batch_size, …)

The outputs of function

tracetf.Tensor of shape (batch_size, …)

The trace of the Jacobian at inputs.

bayesflow.computational_utilities.gaussian_kernel_matrix(x, y, sigmas=None)

Computes a Gaussian radial basis functions (RBFs) between the samples of x and y.

We create a sum of multiple Gaussian kernels each having a width \(\sigma_i\).

Parameters:

xtf.Tensor of shape (num_draws_x, num_features)

Comprises num_draws_x Random draws from the “source” distribution P.

ytf.Tensor of shape (num_draws_y, num_features)

Comprises num_draws_y Random draws from the “source” distribution Q.

sigmaslist(float), optional, default: None

List which denotes the widths of each of the gaussians in the kernel. If sigmas is None, a default range will be used, contained in bayesflow.default_settings.MMD_BANDWIDTH_LIST

Returns:

kerneltf.Tensor of shape (num_draws_x, num_draws_y)

The kernel matrix between pairs from x and y.

bayesflow.computational_utilities.inverse_multiquadratic_kernel_matrix(x, y, sigmas=None)

Computes an inverse multiquadratic RBF between the samples of x and y.

We create a sum of multiple IM-RBF kernels each having a width \(\sigma_i\).

Parameters:

xtf.Tensor of shape (num_draws_x, num_features)

Comprises num_draws_x Random draws from the “source” distribution P.

ytf.Tensor of shape (num_draws_y, num_features)

Comprises num_draws_y Random draws from the “source” distribution Q.

sigmaslist(float), optional, default: None

List which denotes the widths of each of the gaussians in the kernel. If sigmas is None, a default range will be used, contained in bayesflow.default_settings.MMD_BANDWIDTH_LIST

Returns:

kerneltf.Tensor of shape (num_draws_x, num_draws_y)

The kernel matrix between pairs from x and y.

bayesflow.computational_utilities.mmd_kernel(x, y, kernel)

Computes the estimator of the Maximum Mean Discrepancy (MMD) between two samples: x and y.

Maximum Mean Discrepancy (MMD) is a distance-measure between random draws from the distributions x ~ P and y ~ Q.

Parameters:

xtf.Tensor of shape (N, num_features)

An array of N random draws from the “source” distribution x ~ P.

ytf.Tensor of shape (M, num_features)

An array of M random draws from the “target” distribution y ~ Q.

kernelcallable

A function which computes the distance between pairs of samples.

Returns:

losstf.Tensor of shape (,)

The statistically biased squared maximum mean discrepancy (MMD) value.

bayesflow.computational_utilities.mmd_kernel_unbiased(x, y, kernel)

Computes the unbiased estimator of the Maximum Mean Discrepancy (MMD) between two samples: x and y. Maximum Mean Discrepancy (MMD) is a distance-measure between the samples of the distributions x ~ P and y ~ Q.

Parameters:

xtf.Tensor of shape (N, num_features)

An array of N random draws from the “source” distribution x ~ P.

ytf.Tensor of shape (M, num_features)

An array of M random draws from the “target” distribution y ~ Q.

kernelcallable

A function which computes the distance between pairs of random draws from x and y.

Returns:

losstf.Tensor of shape (,)

The statistically unbiased squared maximum mean discrepancy (MMD) value.

bayesflow.computational_utilities.expected_calibration_error(m_true, m_pred, num_bins=10)

Estimates the expected calibration error (ECE) of a model comparison network according to [1].

[1] Naeini, M. P., Cooper, G., & Hauskrecht, M. (2015).

Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 29, No. 1).

Parameters:

m_truenp.ndarray of shape (num_sim, num_models)

The one-hot-encoded true model indices.

m_predtf.tensor of shape (num_sim, num_models)

The predicted posterior model probabilities.

num_binsint, optional, default: 10

The number of bins to use for the calibration curves (and marginal histograms).

Returns:

cal_errslist of length (num_models)

The ECEs for each model.

probslist of length (num_models)

The bin information for constructing the calibration curves. Each list contains two arrays of length (num_bins) with the predicted and true probabilities for each bin.

Notes

Make sure that m_true are one-hot encoded classes!

bayesflow.computational_utilities.maximum_mean_discrepancy(source_samples, target_samples, kernel='gaussian', mmd_weight=1.0, minimum=0.0)

Computes the MMD given a particular choice of kernel.

For details, consult Gretton et al. (2012): https://www.jmlr.org/papers/volume13/gretton12a/gretton12a.pdf

Parameters:

source_samplestf.Tensor of shape (N, num_features)

An array of N random draws from the “source” distribution.

target_samplestf.Tensor of shape (M, num_features)

An array of M random draws from the “target” distribution.

kernelstr in (‘gaussian’, ‘inverse_multiquadratic’), optional, default: ‘gaussian’

The kernel to use for computing the distance between pairs of random draws.

mmd_weightfloat, optional, default: 1.0

The weight of the MMD value.

minimumfloat, optional, default: 0.0

The lower bound of the MMD value.

Returns:

loss_valuetf.Tensor

A scalar Maximum Mean Discrepancy, shape (,)

bayesflow.computational_utilities.get_coverage_probs(z, u)

Vectorized function to compute the minimal coverage probability for uniform ECDFs given evaluation points z and a sample of samples u.

Parameters:

znp.ndarray of shape (num_points, )

The vector of evaluation points.

unp.ndarray of shape (num_simulations, num_samples)

The matrix of simulated draws (samples) from U(0, 1)

bayesflow.computational_utilities.simultaneous_ecdf_bands(num_samples, num_points=None, num_simulations=1000, confidence=0.95, eps=1e-05, max_num_points=1000)

Computes the simultaneous ECDF bands through simulation according to the algorithm described in Section 2.2:

https://link.springer.com/content/pdf/10.1007/s11222-022-10090-6.pdf

Depends on the vectorized utility function get_coverage_probs(z, u).

Parameters:

num_samplesint

The sample size used for computing the ECDF. Will equal to the number of posterior samples when used for calibrarion. Corresponds to N in the paper above.

num_pointsint, optional, default: None

The number of evaluation points on the interval (0, 1). Defaults to num_points = num_samples if not explicitly specified. Correspond to K in the paper above.

num_simulationsint, optional, default: 1000

The number of samples of size n_samples to simulate for determining the simultaneous CIs.

confidencefloat in (0, 1), optional, default: 0.95

The confidence level, confidence = 1 - alpha specifies the width of the confidence interval.

epsfloat, optional, default: 1e-5

Small number to add to the lower and subtract from the upper bound of the interval [0, 1] to avoid edge artefacts. No need to touch this.

max_num_pointsint, optional, default: 1000

Upper bound on num_points. Saves computation time when num_samples is large.

Returns:

(alpha, z, L, U) - tuple of scalar and three arrays of size (num_samples,) containing the confidence level as well as

the evaluation points, the lower, and the upper confidence bands, respectively.

bayesflow.computational_utilities.mean_squared_error(x_true, x_pred)

Computes the mean squared error between a single true value and M estimates thereof.

Parameters:

x_truefloat or np.ndarray

true values, shape ()

x_prednp.ndarray

predicted values, shape (M, )

Returns:

outfloat

The MSE between x_true and x_pred

bayesflow.computational_utilities.root_mean_squared_error(x_true, x_pred)

Computes the root mean squared error (RMSE) between a single true value and M estimates thereof.

Parameters:

x_truefloat or np.ndarray

true values, shape ()

x_prednp.ndarray

predicted values, shape (M, )

Returns:

outfloat

The RMSE between x_true and x_pred

bayesflow.computational_utilities.aggregated_error(x_true, x_pred, inner_error_fun=<function root_mean_squared_error>, outer_aggregation_fun=<function mean>)

Computes the aggregated error between a vector of N true values and M estimates of each true value.

x_true : np.ndarray true values, shape (N)

x_pred : np.ndarray predicted values, shape (M, N)

inner_error_fun: callable, default: root_mean_squared_error computes the error between one true value and M estimates thereof

outer_aggregation_fun: callable, default: np.mean aggregates N errors to a single aggregated error value

bayesflow.computational_utilities.aggregated_rmse(x_true, x_pred)

Computes the aggregated RMSE for a matrix of predictions.

Parameters:

x_truenp.ndarray

true values, shape (N)

x_prednp.ndarray

predicted values, shape (M, N)

Returns:

aggregated RMSE

bayesflow.computational_utilities.c2st(source_samples, target_samples, n_folds=5, scoring='accuracy', normalize=True, seed=123, hidden_units_per_dim=16, aggregate_output=True)

C2ST metric [1] using an sklearn neural network classifier (i.e., MLP). Code adapted from sbi-benchmark/sbibm

[1] Lopez-Paz, D., & Oquab, M. (2016). Revisiting classifier two-sample tests. arXiv:1610.06545.

Parameters:

source_samplesnp.ndarray or tf.Tensor

Source samples (e.g., approximate posterior samples)

target_samplesnp.ndarray or tf.Tensor

Target samples (e.g., samples from a reference posterior)

n_foldsint, optional, default: 5

Number of folds in k-fold cross-validation for the classifier evaluation

scoringstr, optional, default: “accuracy”

Evaluation score of the sklearn MLP classifier

normalizebool, optional, default: True

Whether the data shall be z-standardized relative to source_samples

seedint, optional, default: 123

RNG seed for the MLP and k-fold CV

hidden_units_per_dimint, optional, default: 16

Number of hidden units in the MLP, relative to the input dimensions. Example: source samples are 5D, hidden_units_per_dim=16 -> 80 hidden units per layer

aggregate_outputbool, optional, default: True

Whether to return a single value aggregated over all cross-validation runs or all values from all runs. If left at default, the empirical mean will be returned

Returns:

c2st_scorefloat

The resulting C2ST score