Evaluate Command#
The evaluate command applies trained models to new data and computes metrics against ground truth.
Usage#
respredai evaluate --models-dir <path> --data <csv_file> --output <output_dir>
Options#
Required#
--models-dir, -m- Directory containing trained models andtraining_metadata.jsonMust be output from
respredai traincommand
--data, -d- Path to new data CSV fileMust contain all feature columns from training
Must contain target columns (ground truth required)
--output, -o- Output directory for evaluation results
Optional#
--quiet, -q- Suppress progress output
Data Requirements#
The new data CSV must have:
All feature columns from the training data (same names)
All target columns for ground truth comparison
Same categorical values (new categories will be ignored)
The command validates columns before evaluation and provides clear error messages for missing columns.
Output Structure#
output_dir/
├── metrics/
│ ├── Target1/
│ │ ├── LR_metrics.csv
│ │ └── RF_metrics.csv
│ └── Target2/
│ └── ...
├── predictions/
│ ├── LR_Target1_predictions.csv
│ ├── LR_Target2_predictions.csv
│ └── ...
└── evaluation_summary.csv
predictions CSV Format#
sample_id,y_true,y_pred,y_prob,prediction_set_size,is_uncertain
0,1,1,0.73,1,False
1,0,0,0.21,1,False
2,1,0,0.48,2,True
prediction_set_size: Number of classes in the conformal prediction set (0 = empty, 1 = certain, 2 = both classes included)
is_uncertain: True if prediction set size is not exactly 1 (ambiguous prediction)
The prediction sets come from the split-conformal model stored in the bundle (fitted on the
proper part of the training data and calibrated on the held-out part), while y_prob and
y_pred come from the model fit on the whole training data, so a prediction can occasionally
fall outside its set. The nominal coverage 1 - alpha per class is guaranteed only for data
exchangeable with the calibration split; on an external cohort it is a target, and the
metrics CSV reports the empirical coverage observed on it (empirical_coverage_overall,
empirical_coverage_class_0, empirical_coverage_class_1, nominal_coverage, set sizes
and the calibration sizes conformal_n_calib_class_*).
Bundles trained with earlier versions have no conformal model: their sets are computed from the stored
q_haton the main model, with a warning that they carry no coverage guarantee; retrain to obtain calibrated sets.Bundles trained with
calibration_fraction = 0(or whose calibration split was not feasible) have no prediction sets: the two columns are empty.
metrics CSV Format#
Metric,Value,CI95_lower,CI95_upper
Precision (0),0.82,0.77,0.87
Precision (1),0.51,0.38,0.63
Recall (0),0.91,0.87,0.94
Recall (1),0.33,0.24,0.44
F1 (0),0.86,0.83,0.89
F1 (1),0.40,0.31,0.49
F1 (weighted),0.79,0.75,0.83
MCC,0.28,0.17,0.39
Balanced Acc,0.62,0.57,0.67
AUROC,0.71,0.64,0.77
VME,0.67,0.56,0.76
ME,0.09,0.06,0.13
FOR,0.18,0.13,0.23
Value: Metric on the new cohort
CI{n}_lower / CI{n}_upper: Bootstrap confidence interval for the external cohort (“bootstrap CI (external cohort)”), with the trained model held fixed: it describes the sampling uncertainty of the estimate on this cohort. The percentage, the number of resamples and the ECE/MCE bin count are those recorded in
training_metadata.json(confidence_level,n_bootstrap,calibration_bins); models trained with earlier versions lack some of these keys, which then fall back to the defaults (95%, 1,000, 10) with a warning.The bootstrap is stratified by class, so a cohort with a single class still gets intervals for the metrics defined on that class (e.g.
MEandRecall (0)for an all-susceptible cohort); metrics that need the missing class are empty.When training used a
group_columnand the new cohort has that column, whole patients are resampled (cluster bootstrap), as in cross-validation. If the column is missing or has missing values, a warning is raised and the rows are resampled.VME (Very Major Error): Rate of false susceptible predictions (predicted 0 when actually 1)
ME (Major Error): Rate of false resistant predictions (predicted 1 when actually 0)
FOR (False Omission Rate): Fraction of predicted-susceptible that are actually resistant = FN / (FN + TN)
evaluation_summary.csv#
Aggregated view of all model-target combinations:
Model,Target,Precision (0),Precision (1),...,AUROC
LR,Target1,0.82,0.51,...,0.71
RF,Target1,0.85,0.48,...,0.73
LR,Target2,0.79,0.55,...,0.69
Example Usage#
# Basic evaluation
respredai evaluate \
--models-dir ./output/trained_models \
--data ./new_data.csv \
--output ./evaluation/
# Quiet mode
respredai evaluate \
--models-dir ./output/trained_models \
--data ./new_data.csv \
--output ./evaluation/ \
--quiet
Handling Different Data#
Missing Categorical Values#
If new data has different categorical values than training:
Missing dummy columns are added with value 0
Extra categories in new data are ignored (encoded as all zeros)
Column Order#
Column order doesn’t matter - the command reorders columns to match training.
Error Scenarios#
Missing features:
Validation Error: Missing feature columns: {'age', 'bmi'}
Missing targets:
Validation Error: Missing target columns (ground truth required): {'Target1'}
Invalid models directory:
Error: Training metadata not found: ./invalid/training_metadata.json
Ensure this directory was created by 'respredai train'
See Also#
Train Command - Train models (required before evaluate)
Run Command - Full nested CV pipeline for model evaluation
Feature Importance Command - Extract feature importance from trained models