Run Command#
The run command executes the full machine learning pipeline with nested cross-validation for antimicrobial resistance prediction.
Usage#
respredai run --config <path_to_config.ini> [options]
Options#
Required#
--config, -c- Path to the configuration file (INI format)Must be a valid, readable file
See Configuration File section for details
Optional#
--quiet, -q- Suppress banner and progress outputDoes not suppress error messages or logs
CLI Overrides#
Override configuration file parameters without editing the file:
--models, -m- Override models (comma-separated)Example:
--models LR,RF,XGB
--targets, -t- Override targets (comma-separated)Example:
--targets Target1,Target2
--output, -o- Override output folderExample:
--output ./new_results/
--seed, -s- Override random seedExample:
--seed 123
--validation-strategy- Override validation strategyValues:
cv,temporal, orbothExample:
--validation-strategy temporal
Examples with overrides:
# Run with different models
respredai run --config my_config.ini --models LR,RF
# Run only specific targets with a different output folder
respredai run --config my_config.ini --targets Target1 --output ./experiment1/
# Quick experiment with different seed
respredai run --config my_config.ini --seed 42 --quiet
# Run with temporal validation
respredai run --config my_config.ini --validation-strategy temporal
Configuration File#
The configuration file uses INI format with the following sections.
Note
Optional parameters can be disabled by commenting out the line with #.
Empty values (e.g., group_column =) are treated as absent.
[Data] Section#
Defines the input data and features.
[Data]
data_path = ./data/my_data.csv
targets = Target1,Target2,Target3
continuous_features = Age,Weight,Temperature
Parameters:
data_path- Path to CSV file containing the datasetMust include all features and target columns
First column is assumed to be the sample ID
targets- Comma-separated list of target column namesEach target will be trained separately
Must exist in the CSV file
continuous_features- Comma-separated list of continuous feature namesThese features will be scaled using StandardScaler
All other features are treated as categorical and one-hot encoded
May be left empty (or omitted) when every feature is categorical
A categorical feature with high cardinality (at least 20 distinct values and more than half as many as rows) triggers a warning, since it is one-hot encoded into one column per level
[Metadata] Section#
Defines metadata columns used for grouping, temporal splitting, and subgroup analysis.
[Metadata]
group_column = patient_id
temporal_column = collection_date
# subgroup_columns = ward, sex
Parameters:
group_column(optional) - Column name for grouping related samplesUse when you have multiple samples from the same patient/subject
Prevents data leakage by keeping all samples from the same group in the same fold
Enables
StratifiedGroupKFoldfor both outer and inner cross-validation (if not specified, standardStratifiedKFoldis used)See details in Create Config Command
temporal_column(optional) - Name of the date/time column for temporal splittingRequired when
validation_strategyistemporalorbothValues are parsed as dates
subgroup_columns(optional) - Comma-separated list of columns for subgroup analysisPerformance metrics are computed separately for each subgroup, each with its bootstrap interval (same stratified, patient-level scheme as the main metrics)
Descriptive only: no test of between-subgroup differences is performed, and overlapping or non-overlapping intervals are not a test
The
Unreliablecolumn flags subgroups with fewer than 10 units (patients whengroup_columnis set, samples otherwise) containing one of the two classesUseful for describing model performance across demographic or clinical categories
[Pipeline] Section#
Controls the machine learning pipeline configuration.
[Pipeline]
models = LR,RF,XGB,CatBoost
outer_folds = 5
inner_folds = 3
outer_cv_repeats = 1
calibrate_threshold = false
threshold_method = auto
calibrate_probabilities = false
probability_calibration_method = sigmoid
probability_calibration_cv = 5
confidence_level = 0.95
n_bootstrap = 1000
calibration_bins = 10
compute_feature_direction = false
Parameters:
models- Comma-separated list of models to trainAvailable models:
LR,MLP,XGB,RF,CatBoost,TabPFN,RBF_SVC,Linear_SVC,KNNUse
respredai list-modelsto see all available models with descriptionsNames are case-insensitive (
lrselectsLR); unknown names are rejected when the configuration is loaded
outer_folds- Number of folds for outer cross-validationUsed for model evaluation (must be at least 2)
inner_folds- Number of folds for inner cross-validationUsed for hyperparameter tuning with GridSearchCV (must be at least 2)
calibrate_threshold- Enable decision threshold optimization (optional, default:false)true: Optimize threshold using Youden’s J statistic (Sensitivity + Specificity - 1)false: Use default threshold of 0.5Threshold optimization uses
inner_foldsfor cross-validationHyperparameters are tuned first (optimizing ROC-AUC), then threshold is optimized
threshold_method- Method for threshold optimization (optional, default:auto)auto: Automatically choose based on sample size (OOF if n < 1000, CV otherwise)oof: Out-of-fold predictions method - aggregates predictions from all CV folds into a single set, then finds one global threshold maximizing Youden’s J across all concatenated samplescv:TunedThresholdClassifierCVmethod - scores every candidate threshold on each inner CV fold, averages the fold curves and keeps the threshold that maximizes the averaged objectiveKey difference:
ooffinds one threshold on all concatenated OOF predictions (global optimization), whilecvaverages the per-fold objective curves before choosing (fold-wise scoring then aggregation)With either method the saved fold model is the classifier fit on the whole training fold (with group-aware probability calibration if enabled) and the threshold is stored beside it. With a
group_columnboth searches are group-aware, including the probability calibrator refitted inside them (cvroutes the groups through scikit-learn metadata routing). A calibrator fit whose group-aware split would leave a training part with a single class uses plain stratified folds instead, with one warning per model and targetOnly used when
calibrate_threshold = true
outer_cv_repeats- Number of repetitions for outer cross-validation (optional, default:1)1: Standard (non-repeated) cross-validation>1: Repeated stratified cross-validation with different random shufflesProvides more robust performance estimates by averaging over multiple CV runs
calibrate_probabilities- Enable post-hoc probability calibration (optional, default:false)true: Apply CalibratedClassifierCV to the best estimator from GridSearchCVfalse: Use uncalibrated probability predictionsApplied after hyper-parameters tuning and before threshold tuning
probability_calibration_method- Method for probability calibration (optional, default:sigmoid)sigmoid: Platt scaling - fits a logistic regression on the classifier outputsisotonic: Isotonic regression - non-parametric, monotonic transformationOnly used when
calibrate_probabilities = true
probability_calibration_cv- Number of folds for probability calibration (optional, default:5)CV folds used internally by CalibratedClassifierCV
Must be at least 2
Only used when
calibrate_probabilities = true
confidence_level- Confidence level for bootstrap confidence intervals (optional, default:0.95)Must be between 0.5 and 1.0
Controls the width of the reported CI bounds (e.g., 0.95 gives the
CI95_lowerandCI95_uppercolumns)
n_bootstrap- Number of bootstrap resamples for confidence intervals (optional, default:1000)Must be at least 100
Higher values give more stable CI estimates at the cost of computation time; with repeated CV the cost also grows with
outer_cv_repeats
calibration_bins- Number of bins for the ECE/MCE calibration metrics (optional, default:10)Must be at least 2
compute_feature_direction- Compute the direction of feature effects (optional, default:false)true: Determine whether each feature increases or decreases the predicted probabilityfalse: Skip feature direction computationUseful for interpretability alongside feature importance scores
[Reproducibility] Section#
Ensures reproducible results.
[Reproducibility]
seed = 42
Parameters:
seed- Random seed for reproducibilitySame seed ensures identical results across runs
Affects data splitting and model initialization
[Log] Section#
Controls logging behavior.
[Log]
verbosity = 1
log_basename = respredai.log
Parameters:
verbosity- Logging level0: No logging to file1: Log major events (model start/end, target completion)2: Verbose logging (includes fold-level details)Any other value is rejected when the configuration is loaded
log_basename- Name of the log fileCreated in the output folder
Contains detailed execution information
[Resources] Section#
Controls computational resources.
[Resources]
n_jobs = -1
Parameters:
n_jobs- Number of parallel jobs-1: Use all available CPU cores1: No parallelizationN: Use N cores
[ModelSaving] Section#
Enables saving trained models for resumption.
[ModelSaving]
enable = true
compression = 3
Parameters:
enable- Enable saving trained modelstrue: Save models after each fold (enables resumption)false: No model saving (faster but no resumption)
compression- Compression level for saved model filesRange: 1-9
1: Minimal compression (fastest, largest files)3: Balanced compression (recommended)9: Maximum compression (slowest, smallest files)
[Imputation] Section#
Controls missing data imputation (optional).
[Imputation]
method = none
strategy = mean
n_neighbors = 5
estimator = bayesian_ridge
Parameters:
method- Imputation methodnone: No imputation (default, requires complete data)simple: SimpleImputer from scikit-learnknn: KNNImputer for k-nearest neighbors imputationiterative: IterativeImputer (MissForest-style)
strategy- Strategy for SimpleImputer (only used whenmethod = simple)mean: Replace missing values with column mean (default)median: Replace with column medianmost_frequent: Replace with most frequent value
n_neighbors- Number of neighbors for KNNImputer (only used whenmethod = knn)Default:
5
estimator- Estimator for IterativeImputer (only used whenmethod = iterative)bayesian_ridge: BayesianRidge estimator (default)random_forest: RandomForestRegressor (MissForest-style)
[Output] Section#
Specifies output location.
[Output]
out_folder = ./output/
Parameters:
out_folder- Path to output directoryWill be created if it doesn’t exist
Contains all results, metrics, and saved models
[Preprocessing] Section#
Controls categorical feature encoding.
[Preprocessing]
ohe_min_frequency = 0.05
Parameters:
ohe_min_frequency(optional) - Minimum frequency for categorical values in OneHotEncoderCategories appearing below this threshold are grouped into an “infrequent” category
Values in (0, 1): proportion of samples (e.g., 0.05 = at least 5% of samples)
Values >= 1: absolute count (e.g., 10 = at least 10 occurrences)
Set to 0 or omit to disable (keep all categories)
[Uncertainty] Section#
Controls split-conformal prediction (Mondrian, class-conditional) for uncertainty quantification.
[Uncertainty]
alpha = 0.1
calibration_fraction = 0.25
Parameters:
alpha- Miscoverage rate for Mondrian conformal prediction (default:0.1)Range: 0 to 0.5 (exclusive)
Default 0.1 gives 90% nominal coverage per class
Prediction sets:
{S},{R}, or{S, R}; in cross-validation the expected coverage of each class is at least1 - alphafor exchangeable data (finite-sample guarantee of split conformal, on average over calibration and test samples; approximate when patients contribute several rows)Conformal diagnostics (coverage, set sizes, calibration sizes,
nominal_coverageandconformal_status) are appended to the metrics CSV
calibration_fraction- Fraction of each training fold held out to calibrate the prediction sets (default:0.25;0or between 0.1 and 0.5)The hyperparameters are tuned on the rest of the fold and the fold model is refit on the whole fold with them;
0disables conformal prediction and tunes on the whole foldA model and target whose calibration split would leave a class out of a fold used by the hyperparameter search or the probability calibration runs without conformal prediction, with a warning
See Create Config Command (section 8) for the method and the minimum calibration size
[Validation] Section#
Controls validation strategy (optional, defaults to standard cross-validation).
[Validation]
validation_strategy = cv
# temporal_split_date = 2023-01-01
# temporal_split_ratio = 0.8
[Metadata]
temporal_column = collection_date
# subgroup_columns = ward, sex
Parameters:
validation_strategy- Validation approach (default:cv)cv: Standard nested cross-validation onlytemporal: Temporal (prospective-style) validation onlyboth: Run both CV and temporal validation
temporal_column- Name of the date/time column for temporal splitting (configured in[Metadata]section)Required when
validation_strategyistemporalorbothValues are parsed as dates
temporal_split_date- Cutoff date in ISO format (e.g.,2023-01-01)Train set: dates before cutoff; test set: dates on or after cutoff
Mutually exclusive with
temporal_split_ratio
temporal_split_ratio- Fraction of data for training by sorted date orderMust be between 0 and 1 (exclusive)
Mutually exclusive with
temporal_split_date
Note: When group_column is configured in the [Metadata] section, temporal splitting assigns entire groups based on the group’s latest date to prevent data leakage.
Pipeline Workflow#
The run command executes the following steps for each model and target:
Configuration and data loading - Parse and validate the configuration file and the dataset
Split - Group-aware outer CV folds (optionally repeated) and/or a temporal split
Each training set:
Preprocessing: one-hot encoding, imputation and scaling fitted on the training data only
Conformal calibration split: part of the training set held out to calibrate the prediction sets (
calibration_fraction > 0)Tuning: inner-CV grid search on the part not held out, then refit on the whole training set
Calibration and threshold: optional probability calibration and decision-threshold tuning
Prediction: labels, probabilities and conformal prediction sets for the held-out data
Save Models: fold models saved for resumption and feature importance (if enabled)
Across folds - Fold mean, std and Nadeau-Bengio SE; pooled out-of-fold metrics with patient-level bootstrap CIs; conformal coverage; subgroup tables (if configured)
Output generation - HTML report, metrics CSVs, confusion matrices, reliability curves and the reproducibility manifest
Output Files#
The pipeline generates the following output structure:
output_folder/
├── models/ # Trained models (if model saving enabled)
│ ├── {Model}_{Target}_models.joblib # Saved models for resumption
│ └── ...
├── metrics/ # Detailed metrics
│ ├── {target}/
│ │ ├── {model}_metrics_detailed.csv # Comprehensive metrics with CI
│ │ ├── {model}_temporal_metrics.csv # Temporal validation metrics (temporal or both)
│ │ └── summary.csv # Summary across all models for this target
│ └── summary_all.csv # Global summary across all models and targets
├── confusion_matrices/ # Confusion matrix heatmaps
│ └── Confusion_matrix_{model}_{target}.png # One PNG per model-target combination
├── calibration/ # Calibration diagnostics
│ └── reliability_curve_{model}_{target}.png # Reliability curves per fold + aggregate
├── subgroup_analysis/ # Subgroup metrics (if subgroup_columns configured)
│ └── {target}/
│ ├── {model}_{subgroup_col}_subgroup.csv # Per-subgroup metrics, intervals, Unreliable flag
│ └── {model}_{subgroup_col}_temporal_subgroup.csv # Same, on the temporal test split
├── report.html # Comprehensive HTML report
├── reproducibility.json # Reproducibility manifest
└── respredai.log # Execution log (if verbosity > 0)
Metrics Files#
Each {model}_metrics_detailed.csv contains:
Metric: Name of the metric (Precision, Recall, F1, MCC, Balanced Acc, AUROC, VME, ME, Brier Score, ECE, MCE)
Mean: Mean value across folds
Std: Standard deviation across folds
SE: Nadeau-Bengio corrected standard error, accounting for training set overlap in k-fold CV
Pooled: Metric on the pooled out-of-fold predictions; with repeated CV, the metric of each repeat (each prediction labelled with its own fold threshold) averaged over repeats
CI{n}_lower: Lower bound of the bootstrap confidence interval of
PooledCI{n}_upper: Upper bound of the bootstrap confidence interval of
Pooled
The CI percentage and number of bootstrap resamples are controlled by confidence_level and n_bootstrap in the [Pipeline] section (defaults: 95%, 1,000 resamples).
What the interval estimates. The CI{n} columns are an out-of-fold (OOF) bootstrap
interval: the evaluated samples are resampled while the fold models are held fixed. The
interval describes the sampling uncertainty of the estimate on this cohort; it is not a
confidence interval for the performance of a model retrained on new data, whose variability
across training sets is what SE reflects. Report Mean with SE, or Pooled with
its interval, and do not mix the two.
How it is computed. Stratified (cluster) BCa bootstrap:
The resampling unit is the patient when
group_columnis set and the sample otherwise; with repeated CV the predictions of a sample in different repeats are resampled together.Units are stratified by the labels they contain (only susceptible, mixed, only resistant) and resampled with replacement within each stratum, so every resample keeps both classes. The interval is therefore conditional on the observed class counts, which makes it slightly narrower than an unconditional bootstrap for prevalence-dependent metrics (Precision, FOR, F1). A stratum with fewer than 10 units is merged into the largest one: a single patient in its own stratum would enter every resample exactly once and add no uncertainty.
The interval is not computed (empty) when fewer than 2 units contain a class the metric needs (for example
Recall (1)andVMEneed resistant samples), and a warning flags intervals based on fewer than 10 such units as coarse.No decision threshold is ever chosen on the test labels: every prediction keeps the label given by the threshold tuned on its own training fold.
Calibration Metrics (always computed, independent of probability calibration setting):
Brier Score: Mean squared error of probability predictions (lower is better, range 0-1)
ECE (Expected Calibration Error): Weighted average of calibration error across probability bins
MCE (Maximum Calibration Error): Maximum calibration error across any probability bin
Confusion Matrix Plots#
Each Confusion_matrix_{model}_{target}.png shows:
Normalized confusion matrix for a single model-target combination
Mean F1, MCC, and AUROC scores with standard deviations
Color-coded heatmap (0.0 = poor, 1.0 = perfect)
HTML Report#
The report.html file provides a comprehensive, self-contained summary:
Metadata: Configuration settings, data path, timestamp
Framework Summary: Pipeline parameters, models, targets, and calibration settings
Results Tables: Per-target pooled out-of-fold metrics with their OOF bootstrap intervals (labelled with the configured
confidence_level) for each modelConfusion Matrices: Embedded visualizations in a responsive grid layout
Calibration Diagnostics: Brier Score, ECE, MCE metrics with their OOF bootstrap intervals, plus reliability curve plots
The report can be opened in any web browser and shared without additional dependencies.
Model Saving System#
Each {Model}_{Target}_models.joblib file contains all data from the outer cross-validation in a single file:
fold_models: A list containing one trained model per outer CV fold
fold_transformers: A list containing one fitted transformer (scaler) per fold
fold_ohe_transformers: A list containing one fitted OneHotEncoder per fold
fold_thresholds: A list containing one calibrated threshold per fold
fold_hyperparams: A list containing the best hyperparameters per fold
fold_test_data: Optional list of (X_test_scaled, feature_names) tuples for SHAP computation
metrics: All metrics (precision, recall, F1, MCC, AUROC, confusion matrices) for every fold
completed_folds: Number of completed folds
timestamp: When the file was saved
For example, with outer_folds=5, each joblib file will contain 5 trained models and their corresponding transformers and metrics.
Examples#
Basic Usage#
respredai run --config my_config.ini
Quiet Mode (for scripts)#
respredai run --config my_config.ini --quiet
See Also#
Train Command - Train models on entire dataset for cross-dataset validation
Evaluate Command - Apply trained models to new data
Feature Importance Command - Extract and visualize feature importance
Validate Config Command - Validate configuration files
Create Config Command - How to create the configuration file