classifier_two_sample_test

bayesflow.diagnostics.metrics.classifier_two_sample_test(estimates: ndarray, targets: ndarray, metric: str = 'accuracy', patience: int = 5, min_epochs: int = 5, max_epochs: int = 100, batch_size: int = 128, initial_learning_rate: float = 0.0005, return_metric_only: bool = True, cross_validation_splits: int = 5, validation_split: float = 0.5, early_stopping_split: float = 0.2, standardize: bool = True, mlp_widths: Sequence | Literal['auto'] = 'auto', classifier: Model | Callable[[], Model] | None = None, conformal: bool | None = None, num_permutations: int = 0, seed: int | None = None, **kwargs) → float | Mapping[str, Any]

C2ST metric [1, 4] between samples from two distributions, computed with a neural classifier. Can be expensive in a loop, since each call trains at least one classifier.

Besides the classification metric, two statistics are computed from the held-out classifier scores: the global regression statistic mean((m(x) - pi)^2) of [2], which is only interpretable relative to its permutation null since classifier variance inflates it, and the AUC, which is rank-based and therefore robust to weak, miscalibrated or overfit classifiers [3]. P-values are available from a label permutation test (num_permutations) and, for the AUC, from the conformal test of [3] (conformal).

[1] Lopez-Paz, D., & Oquab, M. (2016). Revisiting classifier two-sample tests. arXiv:1610.06545.

[2] Kim, I., Lee, A. B., & Lei, J. (2019). Global and local two-sample tests via regression. arXiv:1812.08927.

[3] Bansal, V., Chen, T., & Scott, J. G. (2026). Conformal C2ST: Turning weak classifiers into strong two-sample tests. ICML 2026. arXiv:2507.17026.

[4] Yao, Y., & Domke, J. (2023). Discriminative calibration: Check Bayesian computation from simulations and flexible classifier. NeurIPS 2023. arXiv:2305.14593.

Parameters:

estimatesnp.ndarray

Array of shape (num_samples_est, num_variables), e.g., approximate posterior samples.

targetsnp.ndarray

Array of shape (num_samples_tar, num_variables), e.g., samples from a reference posterior.

metricstr

Classifier metric in [0, 1] where larger is better; mapped to >= 0.5. Default is “accuracy”.

patienceint

Number of epochs without improvement after which training stops. Default is 5.

min_epochsint

Number of warm-up epochs during which early stopping is disabled. Default is 5.

max_epochsint

Maximum number of epochs to train the classifier. Default is 100.

batch_sizeint

Number of samples per batch during training. Default is 128.

initial_learning_ratefloat

Peak learning rate of the default AdamW optimizer. Ignored if classifier is an already compiled model. Default is 5e-4.

return_metric_onlybool

If True, only the validation metric is returned; otherwise, also the other statistics, classifiers and histories. Ignored if conformal is True or num_permutations is set. Default is True.

cross_validation_splitsint

Number of cross-validation splits. Default is 5.

validation_splitfloat

Fraction of the data used as validation data for a single hold-out split (cross_validation_splits=1). Default is 0.5.

early_stopping_splitfloat

Fraction of the training data held out for early stopping. Set to 0 to disable early stopping. Default is 0.2.

standardizebool

If True, the pooled samples are standardized. Default is True.

mlp_widthsSequence[int] or “auto”

Hidden layer widths of the default MLP. ‘auto’ uses two layers of the smallest power of two above 10 times the number of variables. Ignored if classifier is passed. Default is ‘auto’.

classifierkeras.Model or callable, optional

Classifier to use instead of the default MLP: a Keras model (cloned) or a callable returning one. It must map (num_samples, num_variables) to a probability in [0, 1]; uncompiled models are compiled with AdamW, binary cross-entropy and metric`. If it ends in a dense sigmoid unit (like the default MLP), its pre-sigmoid log-odds are used as scores for the rank-based statistics. Default is None.

conformalbool, optional

Whether to additionally compute the conformal two-sample test of [3] (their budget-matched “multiple” variant), which calibrates the ranks of the held-out scores without re-training and therefore stays powerful for weak, biased or overfit classifiers. Default is None: the test is included whenever a dictionary is returned anyway, since it costs no extra fits.

num_permutationsint

Number of label permutations for permutation p-values of the classification metric, the regression statistic and the AUC, with resolution 1 / (num_permutations + 1). Default is 0 (no test).

seedint, optional

Seed for reproduciblity. Default is None (non-deterministic).

**kwargs

Additional keyword arguments. Recognized keyword:

mlp_kwargsdict

Dictionary of additional parameters to pass to the MLP constructor. The default MLP is regularized with dropout=0.1.

Returns:

resultsfloat or dict