Create Config Command#
The create-config command generates a template configuration file that you can customize for your data.
Usage#
respredai create-config <output_path.ini>
Options#
Required#
output_path- Path where the template configuration file will be createdMust end with
.iniextensionParent directory must exist or be creatable
File will be overwritten if it already exists
Description#
This command creates a ready-to-use configuration template with all required sections pre-populated and inline comments explaining each parameter.
The generated template follows the INI format required by the run command.
Generated Template#
The command creates a file with the following structure:
[Data]
data_path = ./data/my_data.csv
targets = Target1,Target2
continuous_features = Feature1,Feature2
[Pipeline]
# Available models: LR, MLP, XGB, RF, CatBoost, TabPFN, RBF_SVC, Linear_SVC, KNN
models = LR,XGB,RF
outer_folds = 5
inner_folds = 3
outer_cv_repeats = 1
calibrate_threshold = false
threshold_method = auto
calibrate_probabilities = false
probability_calibration_method = sigmoid
probability_calibration_cv = 5
# confidence_level = 0.95
# n_bootstrap = 1000
calibration_bins = 10
[Reproducibility]
seed = 42
[Log]
# Verbosity levels: 0 = no log, 1 = basic logging, 2 = detailed logging
verbosity = 1
log_basename = respredai.log
[Resources]
# Number of parallel jobs (-1 uses all available cores)
n_jobs = -1
# [Uncertainty]
# Miscoverage rate for conformal prediction (default 0.1 = 90% coverage)
# alpha = 0.1
# Fraction of each training set held out to calibrate the conformal prediction sets
# (0, or between 0.1 and 0.5; default 0.25). 0 disables conformal prediction and tunes
# the hyperparameters on the whole training set
# calibration_fraction = 0.25
[Preprocessing]
ohe_min_frequency = 0.05
# [Imputation]
# method = none # none, simple, knn, or iterative
# strategy = mean # For simple: mean, median, most_frequent
# n_neighbors = 5 # For knn
# estimator = bayesian_ridge # For iterative: bayesian_ridge or random_forest
[ModelSaving]
# Enable model saving for resuming interrupted runs
enable = true
# Compression level for saved models (1-9, higher = more compression but slower)
compression = 3
[Output]
out_folder = ./output/
# [Metadata]
# temporal_column = collection_date
# group_column = PatientID
# subgroup_columns = Ward,Specimen
# [Validation]
# Validation strategy: cv (default), temporal (prospective-style), or both
# validation_strategy = cv
# temporal_split_date = 2023-01-01
# temporal_split_ratio = 0.8
Customization Steps#
After generating the template, customize it for your data.
Note
Optional parameters can be disabled by commenting out the line with #.
Empty values (e.g., group_column =) are treated as absent.
1. Update Data Section#
[Data]
data_path = ./path/to/your/data.csv
targets = AntibioticA,AntibioticB
continuous_features = Feature1,Feature3,Feature4
[Metadata]
# group_column = PatientID # Optional
# subgroup_columns = Ward,Specimen # Optional
data_path: Path to your CSV file
targets: Comma-separated list of target columns (binary classification)
continuous_features: Features to scale with StandardScaler (all others are one-hot encoded)
The [Metadata] section holds columns that describe sample context but are not used as features:
group_column (optional): Column name for grouping multiple samples from the same patient/subject to prevent data leakage
subgroup_columns (optional): Comma-separated column names for defining subgroups for stratified performance evaluation
temporal_column (optional): Column with dates for temporal (prospective-style) validation
Configuring Subgroup Analysis#
Subgroup analysis evaluates model performance separately for each distinct value of one or more categorical columns, helping identify disparities across clinical subgroups (e.g., ward, specimen type, species).
[Metadata]
group_column = PatientID
subgroup_columns = Ward,Specimen
Each column listed in
subgroup_columnsmust exist in the input CSV.Subgroup columns are automatically removed from the feature matrix - they are used for stratified evaluation only, not as predictive features.
Multiple columns can be specified (comma-separated); each is analyzed independently.
What subgroup analysis produces:
For each model-target-subgroup combination, a CSV is saved under
<out_folder>/subgroup_analysis/<target>/<model>_<subgroup_column>_subgroup.csv
containing:
One row per unique subgroup value
Columns:
Subgroup,N(sample count; a sample predicted in several CV repeats counts once),Prevalence(class 1 rate),Unreliable, all standard metrics (Precision, Recall, F1, MCC, AUROC, VME, ME, FOR, etc.) and, for each metric, the bootstrap interval bounds<metric>_CI95_lowerand<metric>_CI95_upper(the percentage followsconfidence_level)UnreliableisTruewhen fewer than 10 units contain one of the two classes (units are patients whengroup_columnis set, samples otherwise); such subgroups are also reported with a warningThe table is descriptive only: no test of between-subgroup differences is performed, and overlapping or non-overlapping intervals are not a test
Subgroup,N,Prevalence,Unreliable,Precision (0),...,AUROC,...,AUROC_CI95_lower,AUROC_CI95_upper,...
ICU,142,0.35,False,0.81,...,0.78,...,0.70,0.85,...
General,310,0.18,False,0.88,...,0.72,...,0.65,0.79,...
ER,89,0.28,True,0.79,...,0.74,...,0.61,0.86,...
Note
group_column and subgroup_columns serve different purposes:
group_column controls cross-validation splitting (keeping all samples from
the same patient in the same fold to prevent data leakage), while
subgroup_columns only affects post-hoc metric stratification. They can
overlap - for instance, group by PatientID while analyzing performance
by Ward.
2. Select Models#
[Pipeline]
models = LR,RF,XGB,CatBoost
Use respredai list-models to see all available models.
3. Configure Cross-Validation#
outer_folds = 5 # For model evaluation
inner_folds = 3 # For hyperparameter tuning
outer_folds: Number of folds for performance evaluation
inner_folds: Number of folds for GridSearchCV hyperparameter tuning
4. Configure Threshold Optimization (Optional)#
calibrate_threshold = true
threshold_method = auto
threshold_objective = youden
vme_cost = 1.0
me_cost = 1.0
calibrate_threshold: Enable decision threshold optimization
true: Calibrate threshold using the specified objectivefalse: Use default threshold of 0.5
threshold_method: Method for threshold optimization (only used when
calibrate_threshold = true)auto: Automatically choose based on sample size (OOF if n < 1000, CV otherwise)oof: Out-of-fold predictions method - aggregates predictions from all CV folds into a single set, then finds one global threshold across all concatenated samplescv:TunedThresholdClassifierCVmethod - scores every candidate threshold on each inner CV fold, averages the fold curves and keeps the threshold that maximizes the averaged objectiveKey difference:
ooffinds one threshold on all concatenated OOF predictions (global optimization), whilecvaverages the per-fold objective curves before choosing (fold-wise scoring then aggregation)With either method the saved fold model is the classifier fit on the whole training fold (with group-aware probability calibration if enabled) and the threshold is stored beside it. With a
group_columnboth searches are group-aware, including the probability calibrator refitted inside them (cvroutes the groups through scikit-learn metadata routing). A calibrator fit whose group-aware split would leave a training part with a single class uses plain stratified folds instead, with one warning per model and target
threshold_objective: Objective function for threshold optimization
youden: Maximize Youden’s J statistic (Sensitivity + Specificity - 1) - balanced approachf1: Maximize F1 score - balances precision and recallf2: Maximize F2 score - prioritizes recall over precision (reduces VME at potential cost of increased ME)cost_sensitive: Minimize weighted error cost usingvme_costandme_cost
vme_cost / me_cost: Cost weights for cost-sensitive threshold optimization
VME (Very Major Error): Predicted susceptible when actually resistant
ME (Major Error): Predicted resistant when actually susceptible
Higher
vme_costrelative tome_costwill shift threshold to reduce false susceptible predictions
5. Configure Repeated Cross-Validation (Optional)#
outer_cv_repeats = 3
outer_cv_repeats: Number of times to repeat outer cross-validation (default: 1)
1: Standard (non-repeated) cross-validation>1: Repeated CV with different random shuffles for more robust estimatesTotal iterations =
outer_folds×outer_cv_repeatsExample: 5 folds × 3 repeats = 15 total train/test iterations
6. Configure Probability Calibration (Optional)#
calibrate_probabilities = true
probability_calibration_method = sigmoid
probability_calibration_cv = 5
confidence_level = 0.95
n_bootstrap = 1000
calibrate_probabilities: Enable post-hoc probability calibration
true: Apply CalibratedClassifierCV to calibrate predicted probabilitiesfalse: Use uncalibrated probabilities (default)Applied after hyper-parameters tuning and before threshold tuning
probability_calibration_method: Calibration method
sigmoid: Platt scaling - fits logistic regression (default, works well for most cases)isotonic: Isotonic regression - non-parametric (requires more data)
probability_calibration_cv: CV folds for calibration (default: 5)
Internal cross-validation used by CalibratedClassifierCV
Must be at least 2
confidence_level: Confidence level for bootstrap CIs (default: 0.95)
Must be between 0.5 and 1.0
n_bootstrap: Number of bootstrap resamples for CIs (default: 1000)
Must be at least 100
calibration_bins: Number of bins for the ECE/MCE calibration metrics (default: 10)
Must be at least 2
Note: Calibration diagnostics (Brier Score, ECE, MCE, reliability curves) are always computed regardless of this setting.
7. Configure Imputation (Optional)#
[Imputation]
method = none
strategy = mean
n_neighbors = 5
estimator = bayesian_ridge
method: Imputation method
none: No imputation (default, requires complete data)simple: SimpleImputer from scikit-learnknn: KNNImputer for k-nearest neighbors imputationiterative: IterativeImputer (MissForest-style)
strategy: Strategy for SimpleImputer (only used when
method = simple)mean,median, ormost_frequent
n_neighbors: Number of neighbors for KNNImputer (only used when
method = knn, default: 5)estimator: Estimator for IterativeImputer (only used when
method = iterative)bayesian_ridge(default) orrandom_forest
8. Configure Conformal Prediction (Optional)#
[Uncertainty]
alpha = 0.1
calibration_fraction = 0.25
alpha: Miscoverage rate for conformal prediction (0-0.5)
Default: 0.1 (90% nominal coverage per class)
Controls the width of prediction sets: lower alpha gives wider sets and higher coverage
calibration_fraction: Fraction of each training set held out to calibrate the prediction sets (default: 0.25; 0, or between 0.1 and 0.5)
0disables conformal prediction: no prediction sets, no coverage rows, and the hyperparameters are tuned on the whole training setWith a value above 0 the hyperparameters are tuned on the remaining (proper) part and the model is then refit on the whole training set, so the reported metrics depend on the fraction
A target whose calibration split is not feasible (for example too few resistant patients for the tuning folds of the proper part) runs without conformal prediction, with a warning
How it works: split-conformal prediction, Mondrian (class-conditional)
From every training set (each outer CV fold, the temporal training period, the full data in
train) a calibration split is held out, stratified by class and with whole patients whengroup_columnis setA conformal model is fitted on the rest only (one-hot encoding, scaling or imputation, hyperparameter search and probability calibration), and its scores on the calibration split give a
q_hatper classNonconformity score:
s(x, y) = 1 - p̂(y | x); a class is included in the prediction set if1 - p̂(class | x) <= q_hat[class]. Prediction sets:{S}(susceptible only),{R}(resistant only),{S, R}(uncertain) or emptyFinite-sample guarantee: for exchangeable data the expected coverage of each class, over calibration and test samples, is at least
1 - alpha(Vovk et al., 2005); the coverage observed in one fold or cohort fluctuates around it. Withgroup_columnthe guarantee is for exchangeable rows and holds approximately when patients contribute several rows. A temporal or external cohort need not be exchangeable with the training data, so there1 - alphais a target, not a guaranteePrediction sets come from the conformal model; point predictions and metrics come from the model fitted on the whole training set, so the two can occasionally disagree
Calibration size: a class needs at least
conformal_min_calibration_size(alpha)calibration samples (9 foralpha = 0.1, 19 foralpha = 0.05); with fewer, itsq_hatis 1, the class is always included and a warning says so. Increasecalibration_fractionoralphafor rare resistancesFeasibility check: before the folds of a model and target are fitted, the calibration splits and the folds that the hyperparameter search and the probability calibration will use on the proper part are checked; if any lacks a class, that model and target run without conformal prediction (hyperparameters tuned on the whole training set) and a warning names it
Output: the metrics CSV includes per-fold conformal diagnostics (empirical coverage overall and per class, average set size, fraction uncertain or empty,
conformal_n_calib_class_*,conformal_q_hat_class_*,conformal_trivial_class_*), anominal_coveragerow and aconformal_statusrow (0 off, 1 run, 2 not run because the split was infeasible, 3 not run because the folds come from a checkpoint of a previous procedure)
9. Configure Preprocessing (Optional)#
[Preprocessing]
ohe_min_frequency = 0.05
ohe_min_frequency: Minimum frequency for categorical values in OneHotEncoder
Categories appearing below this threshold are grouped into an “infrequent” category
Values in (0, 1): proportion of samples (e.g., 0.05 = at least 5% of samples)
Values >= 1: absolute count (e.g., 10 = at least 10 occurrences)
Set to 0, omit, or comment out to disable (keep all categories)
Useful for reducing noise from rare categorical values and preventing overfitting
10. Adjust Resources#
[Resources]
n_jobs = -1 # Use all cores
-1: Use all available CPU cores1: No parallelization (useful for debugging)N: Use N cores
11. Configure Model Saving#
[ModelSaving]
enable = true
compression = 3
enable: Set to
trueto save models every foldscompression: 1-9 (1=minimal compression, 3=balanced, 9=maximum)
12. Set Output Location#
[Output]
out_folder = ./results/
The folder will be created if it doesn’t exist.
13. Configure Validation Strategy (Optional)#
[Metadata]
# temporal_column = collection_date
[Validation]
validation_strategy = cv
# temporal_split_date = 2023-01-01
# temporal_split_ratio = 0.8
temporal_column (in
[Metadata]): Name of the date/time column for temporal splittingRequired when
validation_strategyistemporalorboth
validation_strategy: Validation approach (default:
cv)cv: Standard nested cross-validation onlytemporal: Temporal (prospective-style) validation onlyboth: Run both CV and temporal validation
temporal_split_date: Cutoff date in ISO format (e.g.,
2023-01-01)Train set: dates before cutoff; test set: dates on or after cutoff
Mutually exclusive with
temporal_split_ratio
temporal_split_ratio: Fraction of data for training by sorted date order
Must be between 0 and 1 (exclusive)
Mutually exclusive with
temporal_split_date
Note: When group_column is configured in [Metadata], temporal splitting assigns entire groups based on the group’s latest date to prevent data leakage.
See Also#
Run Command - Execute the nested CV pipeline
Train Command - Train models on entire dataset for cross-dataset validation
Validate Config Command - Validate configuration before running