classifier_two_sample_test

bayesflow.diagnostics.metrics.classifier_two_sample_test(estimates: ndarray, targets: ndarray, metric: str = 'accuracy', patience: int = 5, max_epochs: int = 1000, batch_size: int = 128, return_metric_only: bool = True, cross_validation_splits: int = 5, validation_split: float = 0.5, standardize: bool = True, mlp_widths: Sequence | Literal['auto'] = 'auto', **kwargs) → float | Mapping[str, Any]

C2ST metric [1] between samples from two distributions computed using a neural classifier. Can be computationally expensive if called in a loop[, since it needs to train the model for each set of samples.

Note: works best for large numbers of samples and averaged across different posteriors.

[1] Lopez-Paz, D., & Oquab, M. (2016). Revisiting classifier two-sample tests. arXiv:1610.06545.

Parameters:

estimatesnp.ndarray

Array of shape (num_samples_est, num_variables) containing samples representing estimated quantities (e.g., approximate posterior samples).

targetsnp.ndarray

Array of shape (num_samples_tar, num_variables) containing target samples (e.g., samples from a reference posterior).

metricstr, optional

Metric to evaluate the classifier performance. Default is “accuracy”.

patienceint, optional

Number of epochs with no improvement after which training will be stopped. Default is 5.

max_epochsint, optional

Maximum number of epochs to train the classifier. Default is 1000.

batch_sizeint, optional

Number of samples per batch during training. Default is 64.

return_metric_onlybool, optional

If True, only the final validation metric is returned. Otherwise, a dictionary with the score, classifier, and full training history is returned. Default is True.

cross_validation_splitsint, optional

Number of cross-validation splits to perform. Default is 5.

validation_splitfloat, optional

Fraction of the training data to be used as validation data, for single hold-out split. Default is 0.5.

standardizebool, optional

If True, both estimates and targets will be standardized using the mean and standard deviation of estimates. Default is True.

mlp_widthsSequence[int], optional

Sequence specifying the number of units in each hidden layer of the MLP classifier. If set to ‘auto’, defaults to two hidden layers with widths such that width is larger than 10 times the number of variables and a power of two. Default is ‘auto’.

**kwargs

Additional keyword arguments. Recognized keyword:

mlp_kwargsdict

Dictionary of additional parameters to pass to the MLP constructor.

Returns:

resultsfloat or dict

If return_metric_only is True, returns the final validation metric (e.g., accuracy) as a float. Otherwise, returns a dictionary with keys “score”, “classifier”, and “history”, where “score” is the final validation metric, “classifier” is the trained Keras model, and “history” contains the full training history.