Evaluate Command#

The evaluate command applies trained models to new data and computes metrics against ground truth.

Usage#

respredai evaluate --models-dir <path> --data <csv_file> --output <output_dir>

Options#

Required#

  • --models-dir, -m - Directory containing trained models and training_metadata.json

    • Must be output from respredai train command

  • --data, -d - Path to new data CSV file

    • Must contain all feature columns from training

    • Must contain target columns (ground truth required)

  • --output, -o - Output directory for evaluation results

Optional#

  • --quiet, -q - Suppress progress output

Data Requirements#

The new data CSV must have:

  1. All feature columns from the training data (same names)

  2. All target columns for ground truth comparison

  3. Same categorical values (new categories will be ignored)

The command validates columns before evaluation and provides clear error messages for missing columns.

Output Structure#

output_dir/
├── metrics/
│   ├── Target1/
│   │   ├── LR_metrics.csv
│   │   └── RF_metrics.csv
│   └── Target2/
│       └── ...
├── predictions/
│   ├── LR_Target1_predictions.csv
│   ├── LR_Target2_predictions.csv
│   └── ...
└── evaluation_summary.csv

predictions CSV Format#

sample_id,y_true,y_pred,y_prob,prediction_set_size,is_uncertain
0,1,1,0.73,1,False
1,0,0,0.21,1,False
2,1,0,0.48,2,True
  • prediction_set_size: Number of classes in the conformal prediction set (0 = empty, 1 = certain, 2 = both classes included)

  • is_uncertain: True if prediction set size is not exactly 1 (ambiguous prediction)

The prediction sets come from the split-conformal model stored in the bundle (fitted on the proper part of the training data and calibrated on the held-out part), while y_prob and y_pred come from the model fit on the whole training data, so a prediction can occasionally fall outside its set. The nominal coverage 1 - alpha per class is guaranteed only for data exchangeable with the calibration split; on an external cohort it is a target, and the metrics CSV reports the empirical coverage observed on it (empirical_coverage_overall, empirical_coverage_class_0, empirical_coverage_class_1, nominal_coverage, set sizes and the calibration sizes conformal_n_calib_class_*).

  • Bundles trained with earlier versions have no conformal model: their sets are computed from the stored q_hat on the main model, with a warning that they carry no coverage guarantee; retrain to obtain calibrated sets.

  • Bundles trained with calibration_fraction = 0 (or whose calibration split was not feasible) have no prediction sets: the two columns are empty.

metrics CSV Format#

Metric,Value,CI95_lower,CI95_upper
Precision (0),0.82,0.77,0.87
Precision (1),0.51,0.38,0.63
Recall (0),0.91,0.87,0.94
Recall (1),0.33,0.24,0.44
F1 (0),0.86,0.83,0.89
F1 (1),0.40,0.31,0.49
F1 (weighted),0.79,0.75,0.83
MCC,0.28,0.17,0.39
Balanced Acc,0.62,0.57,0.67
AUROC,0.71,0.64,0.77
VME,0.67,0.56,0.76
ME,0.09,0.06,0.13
FOR,0.18,0.13,0.23
  • Value: Metric on the new cohort

  • CI{n}_lower / CI{n}_upper: Bootstrap confidence interval for the external cohort (“bootstrap CI (external cohort)”), with the trained model held fixed: it describes the sampling uncertainty of the estimate on this cohort. The percentage, the number of resamples and the ECE/MCE bin count are those recorded in training_metadata.json (confidence_level, n_bootstrap, calibration_bins); models trained with earlier versions lack some of these keys, which then fall back to the defaults (95%, 1,000, 10) with a warning.

  • The bootstrap is stratified by class, so a cohort with a single class still gets intervals for the metrics defined on that class (e.g. ME and Recall (0) for an all-susceptible cohort); metrics that need the missing class are empty.

  • When training used a group_column and the new cohort has that column, whole patients are resampled (cluster bootstrap), as in cross-validation. If the column is missing or has missing values, a warning is raised and the rows are resampled.

  • VME (Very Major Error): Rate of false susceptible predictions (predicted 0 when actually 1)

  • ME (Major Error): Rate of false resistant predictions (predicted 1 when actually 0)

  • FOR (False Omission Rate): Fraction of predicted-susceptible that are actually resistant = FN / (FN + TN)

evaluation_summary.csv#

Aggregated view of all model-target combinations:

Model,Target,Precision (0),Precision (1),...,AUROC
LR,Target1,0.82,0.51,...,0.71
RF,Target1,0.85,0.48,...,0.73
LR,Target2,0.79,0.55,...,0.69

Example Usage#

# Basic evaluation
respredai evaluate \
    --models-dir ./output/trained_models \
    --data ./new_data.csv \
    --output ./evaluation/

# Quiet mode
respredai evaluate \
    --models-dir ./output/trained_models \
    --data ./new_data.csv \
    --output ./evaluation/ \
    --quiet

Handling Different Data#

Missing Categorical Values#

If new data has different categorical values than training:

  • Missing dummy columns are added with value 0

  • Extra categories in new data are ignored (encoded as all zeros)

Column Order#

Column order doesn’t matter - the command reorders columns to match training.

Error Scenarios#

Missing features:

Validation Error: Missing feature columns: {'age', 'bmi'}

Missing targets:

Validation Error: Missing target columns (ground truth required): {'Target1'}

Invalid models directory:

Error: Training metadata not found: ./invalid/training_metadata.json
Ensure this directory was created by 'respredai train'

See Also#