Run Command#

The run command executes the full machine learning pipeline with nested cross-validation for antimicrobial resistance prediction.

Usage#

respredai run --config <path_to_config.ini> [options]

Options#

Required#

  • --config, -c - Path to the configuration file (INI format)

Optional#

  • --quiet, -q - Suppress banner and progress output

    • Does not suppress error messages or logs

CLI Overrides#

Override configuration file parameters without editing the file:

  • --models, -m - Override models (comma-separated)

    • Example: --models LR,RF,XGB

  • --targets, -t - Override targets (comma-separated)

    • Example: --targets Target1,Target2

  • --output, -o - Override output folder

    • Example: --output ./new_results/

  • --seed, -s - Override random seed

    • Example: --seed 123

  • --validation-strategy - Override validation strategy

    • Values: cv, temporal, or both

    • Example: --validation-strategy temporal

Examples with overrides:

# Run with different models
respredai run --config my_config.ini --models LR,RF

# Run only specific targets with a different output folder
respredai run --config my_config.ini --targets Target1 --output ./experiment1/

# Quick experiment with different seed
respredai run --config my_config.ini --seed 42 --quiet

# Run with temporal validation
respredai run --config my_config.ini --validation-strategy temporal

Configuration File#

The configuration file uses INI format with the following sections.

Note

Optional parameters can be disabled by commenting out the line with #. Empty values (e.g., group_column =) are treated as absent.

[Data] Section#

Defines the input data and features.

[Data]
data_path = ./data/my_data.csv
targets = Target1,Target2,Target3
continuous_features = Age,Weight,Temperature

Parameters:

  • data_path - Path to CSV file containing the dataset

    • Must include all features and target columns

    • First column is assumed to be the sample ID

  • targets - Comma-separated list of target column names

    • Each target will be trained separately

    • Must exist in the CSV file

  • continuous_features - Comma-separated list of continuous feature names

    • These features will be scaled using StandardScaler

    • All other features are treated as categorical and one-hot encoded

    • May be left empty (or omitted) when every feature is categorical

    • A categorical feature with high cardinality (at least 20 distinct values and more than half as many as rows) triggers a warning, since it is one-hot encoded into one column per level

[Metadata] Section#

Defines metadata columns used for grouping, temporal splitting, and subgroup analysis.

[Metadata]
group_column = patient_id
temporal_column = collection_date
# subgroup_columns = ward, sex

Parameters:

  • group_column (optional) - Column name for grouping related samples

    • Use when you have multiple samples from the same patient/subject

    • Prevents data leakage by keeping all samples from the same group in the same fold

    • Enables StratifiedGroupKFold for both outer and inner cross-validation (if not specified, standard StratifiedKFold is used)

    • See details in Create Config Command

  • temporal_column (optional) - Name of the date/time column for temporal splitting

    • Required when validation_strategy is temporal or both

    • Values are parsed as dates

  • subgroup_columns (optional) - Comma-separated list of columns for subgroup analysis

    • Performance metrics are computed separately for each subgroup, each with its bootstrap interval (same stratified, patient-level scheme as the main metrics)

    • Descriptive only: no test of between-subgroup differences is performed, and overlapping or non-overlapping intervals are not a test

    • The Unreliable column flags subgroups with fewer than 10 units (patients when group_column is set, samples otherwise) containing one of the two classes

    • Useful for describing model performance across demographic or clinical categories

[Pipeline] Section#

Controls the machine learning pipeline configuration.

[Pipeline]
models = LR,RF,XGB,CatBoost
outer_folds = 5
inner_folds = 3
outer_cv_repeats = 1
calibrate_threshold = false
threshold_method = auto
calibrate_probabilities = false
probability_calibration_method = sigmoid
probability_calibration_cv = 5
confidence_level = 0.95
n_bootstrap = 1000
calibration_bins = 10
compute_feature_direction = false

Parameters:

  • models - Comma-separated list of models to train

    • Available models: LR, MLP, XGB, RF, CatBoost, TabPFN, RBF_SVC, Linear_SVC, KNN

    • Use respredai list-models to see all available models with descriptions

    • Names are case-insensitive (lr selects LR); unknown names are rejected when the configuration is loaded

  • outer_folds - Number of folds for outer cross-validation

    • Used for model evaluation (must be at least 2)

  • inner_folds - Number of folds for inner cross-validation

    • Used for hyperparameter tuning with GridSearchCV (must be at least 2)

  • calibrate_threshold - Enable decision threshold optimization (optional, default: false)

    • true: Optimize threshold using Youden’s J statistic (Sensitivity + Specificity - 1)

    • false: Use default threshold of 0.5

    • Threshold optimization uses inner_folds for cross-validation

    • Hyperparameters are tuned first (optimizing ROC-AUC), then threshold is optimized

  • threshold_method - Method for threshold optimization (optional, default: auto)

    • auto: Automatically choose based on sample size (OOF if n < 1000, CV otherwise)

    • oof: Out-of-fold predictions method - aggregates predictions from all CV folds into a single set, then finds one global threshold maximizing Youden’s J across all concatenated samples

    • cv: TunedThresholdClassifierCV method - scores every candidate threshold on each inner CV fold, averages the fold curves and keeps the threshold that maximizes the averaged objective

    • Key difference: oof finds one threshold on all concatenated OOF predictions (global optimization), while cv averages the per-fold objective curves before choosing (fold-wise scoring then aggregation)

    • With either method the saved fold model is the classifier fit on the whole training fold (with group-aware probability calibration if enabled) and the threshold is stored beside it. With a group_column both searches are group-aware, including the probability calibrator refitted inside them (cv routes the groups through scikit-learn metadata routing). A calibrator fit whose group-aware split would leave a training part with a single class uses plain stratified folds instead, with one warning per model and target

    • Only used when calibrate_threshold = true

  • outer_cv_repeats - Number of repetitions for outer cross-validation (optional, default: 1)

    • 1: Standard (non-repeated) cross-validation

    • >1: Repeated stratified cross-validation with different random shuffles

    • Provides more robust performance estimates by averaging over multiple CV runs

  • calibrate_probabilities - Enable post-hoc probability calibration (optional, default: false)

    • true: Apply CalibratedClassifierCV to the best estimator from GridSearchCV

    • false: Use uncalibrated probability predictions

    • Applied after hyper-parameters tuning and before threshold tuning

  • probability_calibration_method - Method for probability calibration (optional, default: sigmoid)

    • sigmoid: Platt scaling - fits a logistic regression on the classifier outputs

    • isotonic: Isotonic regression - non-parametric, monotonic transformation

    • Only used when calibrate_probabilities = true

  • probability_calibration_cv - Number of folds for probability calibration (optional, default: 5)

    • CV folds used internally by CalibratedClassifierCV

    • Must be at least 2

    • Only used when calibrate_probabilities = true

  • confidence_level - Confidence level for bootstrap confidence intervals (optional, default: 0.95)

    • Must be between 0.5 and 1.0

    • Controls the width of the reported CI bounds (e.g., 0.95 gives the CI95_lower and CI95_upper columns)

  • n_bootstrap - Number of bootstrap resamples for confidence intervals (optional, default: 1000)

    • Must be at least 100

    • Higher values give more stable CI estimates at the cost of computation time; with repeated CV the cost also grows with outer_cv_repeats

  • calibration_bins - Number of bins for the ECE/MCE calibration metrics (optional, default: 10)

    • Must be at least 2

  • compute_feature_direction - Compute the direction of feature effects (optional, default: false)

    • true: Determine whether each feature increases or decreases the predicted probability

    • false: Skip feature direction computation

    • Useful for interpretability alongside feature importance scores

[Reproducibility] Section#

Ensures reproducible results.

[Reproducibility]
seed = 42

Parameters:

  • seed - Random seed for reproducibility

    • Same seed ensures identical results across runs

    • Affects data splitting and model initialization

[Log] Section#

Controls logging behavior.

[Log]
verbosity = 1
log_basename = respredai.log

Parameters:

  • verbosity - Logging level

    • 0: No logging to file

    • 1: Log major events (model start/end, target completion)

    • 2: Verbose logging (includes fold-level details)

    • Any other value is rejected when the configuration is loaded

  • log_basename - Name of the log file

    • Created in the output folder

    • Contains detailed execution information

[Resources] Section#

Controls computational resources.

[Resources]
n_jobs = -1

Parameters:

  • n_jobs - Number of parallel jobs

    • -1: Use all available CPU cores

    • 1: No parallelization

    • N: Use N cores

[ModelSaving] Section#

Enables saving trained models for resumption.

[ModelSaving]
enable = true
compression = 3

Parameters:

  • enable - Enable saving trained models

    • true: Save models after each fold (enables resumption)

    • false: No model saving (faster but no resumption)

  • compression - Compression level for saved model files

    • Range: 1-9

    • 1: Minimal compression (fastest, largest files)

    • 3: Balanced compression (recommended)

    • 9: Maximum compression (slowest, smallest files)

[Imputation] Section#

Controls missing data imputation (optional).

[Imputation]
method = none
strategy = mean
n_neighbors = 5
estimator = bayesian_ridge

Parameters:

  • method - Imputation method

    • none: No imputation (default, requires complete data)

    • simple: SimpleImputer from scikit-learn

    • knn: KNNImputer for k-nearest neighbors imputation

    • iterative: IterativeImputer (MissForest-style)

  • strategy - Strategy for SimpleImputer (only used when method = simple)

    • mean: Replace missing values with column mean (default)

    • median: Replace with column median

    • most_frequent: Replace with most frequent value

  • n_neighbors - Number of neighbors for KNNImputer (only used when method = knn)

    • Default: 5

  • estimator - Estimator for IterativeImputer (only used when method = iterative)

    • bayesian_ridge: BayesianRidge estimator (default)

    • random_forest: RandomForestRegressor (MissForest-style)

[Output] Section#

Specifies output location.

[Output]
out_folder = ./output/

Parameters:

  • out_folder - Path to output directory

    • Will be created if it doesn’t exist

    • Contains all results, metrics, and saved models

[Preprocessing] Section#

Controls categorical feature encoding.

[Preprocessing]
ohe_min_frequency = 0.05

Parameters:

  • ohe_min_frequency (optional) - Minimum frequency for categorical values in OneHotEncoder

    • Categories appearing below this threshold are grouped into an “infrequent” category

    • Values in (0, 1): proportion of samples (e.g., 0.05 = at least 5% of samples)

    • Values >= 1: absolute count (e.g., 10 = at least 10 occurrences)

    • Set to 0 or omit to disable (keep all categories)

[Uncertainty] Section#

Controls split-conformal prediction (Mondrian, class-conditional) for uncertainty quantification.

[Uncertainty]
alpha = 0.1
calibration_fraction = 0.25

Parameters:

  • alpha - Miscoverage rate for Mondrian conformal prediction (default: 0.1)

    • Range: 0 to 0.5 (exclusive)

    • Default 0.1 gives 90% nominal coverage per class

    • Prediction sets: {S}, {R}, or {S, R}; in cross-validation the expected coverage of each class is at least 1 - alpha for exchangeable data (finite-sample guarantee of split conformal, on average over calibration and test samples; approximate when patients contribute several rows)

    • Conformal diagnostics (coverage, set sizes, calibration sizes, nominal_coverage and conformal_status) are appended to the metrics CSV

  • calibration_fraction - Fraction of each training fold held out to calibrate the prediction sets (default: 0.25; 0 or between 0.1 and 0.5)

    • The hyperparameters are tuned on the rest of the fold and the fold model is refit on the whole fold with them; 0 disables conformal prediction and tunes on the whole fold

    • A model and target whose calibration split would leave a class out of a fold used by the hyperparameter search or the probability calibration runs without conformal prediction, with a warning

    • See Create Config Command (section 8) for the method and the minimum calibration size

[Validation] Section#

Controls validation strategy (optional, defaults to standard cross-validation).

[Validation]
validation_strategy = cv
# temporal_split_date = 2023-01-01
# temporal_split_ratio = 0.8

[Metadata]
temporal_column = collection_date
# subgroup_columns = ward, sex

Parameters:

  • validation_strategy - Validation approach (default: cv)

    • cv: Standard nested cross-validation only

    • temporal: Temporal (prospective-style) validation only

    • both: Run both CV and temporal validation

  • temporal_column - Name of the date/time column for temporal splitting (configured in [Metadata] section)

    • Required when validation_strategy is temporal or both

    • Values are parsed as dates

  • temporal_split_date - Cutoff date in ISO format (e.g., 2023-01-01)

    • Train set: dates before cutoff; test set: dates on or after cutoff

    • Mutually exclusive with temporal_split_ratio

  • temporal_split_ratio - Fraction of data for training by sorted date order

    • Must be between 0 and 1 (exclusive)

    • Mutually exclusive with temporal_split_date

Note: When group_column is configured in the [Metadata] section, temporal splitting assigns entire groups based on the group’s latest date to prevent data leakage.

Pipeline Workflow#

ResPredAI pipeline: split the data, fit on each training set, aggregate across folds and write the report

The run command executes the following steps for each model and target:

  1. Configuration and data loading - Parse and validate the configuration file and the dataset

  2. Split - Group-aware outer CV folds (optionally repeated) and/or a temporal split

  3. Each training set:

    • Preprocessing: one-hot encoding, imputation and scaling fitted on the training data only

    • Conformal calibration split: part of the training set held out to calibrate the prediction sets (calibration_fraction > 0)

    • Tuning: inner-CV grid search on the part not held out, then refit on the whole training set

    • Calibration and threshold: optional probability calibration and decision-threshold tuning

    • Prediction: labels, probabilities and conformal prediction sets for the held-out data

    • Save Models: fold models saved for resumption and feature importance (if enabled)

  4. Across folds - Fold mean, std and Nadeau-Bengio SE; pooled out-of-fold metrics with patient-level bootstrap CIs; conformal coverage; subgroup tables (if configured)

  5. Output generation - HTML report, metrics CSVs, confusion matrices, reliability curves and the reproducibility manifest

Output Files#

The pipeline generates the following output structure:

output_folder/
├── models/                                       # Trained models (if model saving enabled)
│   ├── {Model}_{Target}_models.joblib            # Saved models for resumption
│   └── ...
├── metrics/                                      # Detailed metrics
│   ├── {target}/
│   │   ├── {model}_metrics_detailed.csv          # Comprehensive metrics with CI
│   │   ├── {model}_temporal_metrics.csv          # Temporal validation metrics (temporal or both)
│   │   └── summary.csv                           # Summary across all models for this target
│   └── summary_all.csv                           # Global summary across all models and targets
├── confusion_matrices/                           # Confusion matrix heatmaps
│   └── Confusion_matrix_{model}_{target}.png     # One PNG per model-target combination
├── calibration/                                  # Calibration diagnostics
│   └── reliability_curve_{model}_{target}.png    # Reliability curves per fold + aggregate
├── subgroup_analysis/                             # Subgroup metrics (if subgroup_columns configured)
│   └── {target}/
│       ├── {model}_{subgroup_col}_subgroup.csv   # Per-subgroup metrics, intervals, Unreliable flag
│       └── {model}_{subgroup_col}_temporal_subgroup.csv  # Same, on the temporal test split
├── report.html                                   # Comprehensive HTML report
├── reproducibility.json                          # Reproducibility manifest
└── respredai.log                                 # Execution log (if verbosity > 0)

Metrics Files#

Each {model}_metrics_detailed.csv contains:

  • Metric: Name of the metric (Precision, Recall, F1, MCC, Balanced Acc, AUROC, VME, ME, Brier Score, ECE, MCE)

  • Mean: Mean value across folds

  • Std: Standard deviation across folds

  • SE: Nadeau-Bengio corrected standard error, accounting for training set overlap in k-fold CV

  • Pooled: Metric on the pooled out-of-fold predictions; with repeated CV, the metric of each repeat (each prediction labelled with its own fold threshold) averaged over repeats

  • CI{n}_lower: Lower bound of the bootstrap confidence interval of Pooled

  • CI{n}_upper: Upper bound of the bootstrap confidence interval of Pooled

The CI percentage and number of bootstrap resamples are controlled by confidence_level and n_bootstrap in the [Pipeline] section (defaults: 95%, 1,000 resamples).

What the interval estimates. The CI{n} columns are an out-of-fold (OOF) bootstrap interval: the evaluated samples are resampled while the fold models are held fixed. The interval describes the sampling uncertainty of the estimate on this cohort; it is not a confidence interval for the performance of a model retrained on new data, whose variability across training sets is what SE reflects. Report Mean with SE, or Pooled with its interval, and do not mix the two.

How it is computed. Stratified (cluster) BCa bootstrap:

  • The resampling unit is the patient when group_column is set and the sample otherwise; with repeated CV the predictions of a sample in different repeats are resampled together.

  • Units are stratified by the labels they contain (only susceptible, mixed, only resistant) and resampled with replacement within each stratum, so every resample keeps both classes. The interval is therefore conditional on the observed class counts, which makes it slightly narrower than an unconditional bootstrap for prevalence-dependent metrics (Precision, FOR, F1). A stratum with fewer than 10 units is merged into the largest one: a single patient in its own stratum would enter every resample exactly once and add no uncertainty.

  • The interval is not computed (empty) when fewer than 2 units contain a class the metric needs (for example Recall (1) and VME need resistant samples), and a warning flags intervals based on fewer than 10 such units as coarse.

  • No decision threshold is ever chosen on the test labels: every prediction keeps the label given by the threshold tuned on its own training fold.

Calibration Metrics (always computed, independent of probability calibration setting):

  • Brier Score: Mean squared error of probability predictions (lower is better, range 0-1)

  • ECE (Expected Calibration Error): Weighted average of calibration error across probability bins

  • MCE (Maximum Calibration Error): Maximum calibration error across any probability bin

Confusion Matrix Plots#

Each Confusion_matrix_{model}_{target}.png shows:

  • Normalized confusion matrix for a single model-target combination

  • Mean F1, MCC, and AUROC scores with standard deviations

  • Color-coded heatmap (0.0 = poor, 1.0 = perfect)

HTML Report#

The report.html file provides a comprehensive, self-contained summary:

  • Metadata: Configuration settings, data path, timestamp

  • Framework Summary: Pipeline parameters, models, targets, and calibration settings

  • Results Tables: Per-target pooled out-of-fold metrics with their OOF bootstrap intervals (labelled with the configured confidence_level) for each model

  • Confusion Matrices: Embedded visualizations in a responsive grid layout

  • Calibration Diagnostics: Brier Score, ECE, MCE metrics with their OOF bootstrap intervals, plus reliability curve plots

The report can be opened in any web browser and shared without additional dependencies.

Model Saving System#

Each {Model}_{Target}_models.joblib file contains all data from the outer cross-validation in a single file:

  • fold_models: A list containing one trained model per outer CV fold

  • fold_transformers: A list containing one fitted transformer (scaler) per fold

  • fold_ohe_transformers: A list containing one fitted OneHotEncoder per fold

  • fold_thresholds: A list containing one calibrated threshold per fold

  • fold_hyperparams: A list containing the best hyperparameters per fold

  • fold_test_data: Optional list of (X_test_scaled, feature_names) tuples for SHAP computation

  • metrics: All metrics (precision, recall, F1, MCC, AUROC, confusion matrices) for every fold

  • completed_folds: Number of completed folds

  • timestamp: When the file was saved

For example, with outer_folds=5, each joblib file will contain 5 trained models and their corresponding transformers and metrics.

Examples#

Basic Usage#

respredai run --config my_config.ini

Quiet Mode (for scripts)#

respredai run --config my_config.ini --quiet

See Also#