Train Command#

The train command trains models on the entire dataset using GridSearchCV for hyperparameter tuning, then saves the best model for each model-target combination.

Usage#

respredai train --config <path_to_config.ini> [options]

Options#

Required#

  • --config, -c - Path to the configuration file (INI format)

Optional#

  • --quiet, -q - Suppress banner and progress output

  • --models, -m - Override models (comma-separated)

  • --targets, -t - Override targets (comma-separated)

  • --output, -o - Override output folder

  • --seed, -s - Override random seed

How It Differs from run#

Aspect

run Command

train Command

Purpose

Evaluate model performance

Train models for cross-dataset validation

CV Strategy

Nested CV (outer + inner)

Single CV (only for HP tuning)

Data Split

Multiple train/test splits

Uses entire dataset

Output

Metrics, confusion matrices

Trained model files, ready for evaluation on another dataset

Model Files

Per-fold models (optional)

Single model per target

Configuration Parameters#

The train command uses the same configuration file as run, but some parameters are ignored:

Parameter

Used

Notes

data_path

Yes

Path to training data

targets

Yes

Target columns to train

continuous_features

Yes

Features to scale

group_column (in [Metadata])

Yes

Used for grouped CV during HP tuning

models

Yes

Models to train

inner_folds

Yes

CV folds for hyperparameter tuning

outer_folds

No

Only used by run for nested CV

calibrate_threshold

Yes

Enables threshold optimization

threshold_method

Yes

Method for threshold optimization

threshold_objective

Yes

Objective function for threshold optimization

calibrate_probabilities

Yes

Enables probability calibration

probability_calibration_method

Yes

Calibration method (sigmoid or isotonic)

probability_calibration_cv

Yes

CV folds for probability calibration

confidence_level

Yes

Confidence level for bootstrap CIs

n_bootstrap

Yes

Number of bootstrap resamples for CIs

seed

Yes

Random seed for reproducibility

n_jobs

Yes

Parallel jobs

out_folder

Yes

Output directory

[ModelSaving] enable

No

Always true for train (models are always saved)

[ModelSaving] compression

Yes

Compression level for saved models

Output Structure#

output_folder/
├── trained_models/
│   ├── LR_Target1.joblib
│   ├── LR_Target2.joblib
│   ├── RF_Target1.joblib
│   ├── ...
│   └── training_metadata.json
└── reproducibility.json                  # Reproducibility manifest

Model Bundle Contents#

Each .joblib file contains:

{
    "model": <fitted_classifier>,
    "transformer": <fitted_scaler>,
    "threshold": 0.42,                  # Calibrated threshold
    "hyperparams": {"C": 0.1, ...},
    "feature_names": [...],             # Original feature names
    "feature_names_transformed": [...], # After one-hot encoding
    "target_name": "Target1",
    "model_name": "LR",
    "training_timestamp": "2025-12-11T...",
    "conformal_model": <fitted_classifier>,        # fitted on the proper part only
    "conformal_transformer": <fitted_scaler>,
    "conformal_ohe_transformer": <fitted_encoder>,
    "conformal_q_hat": {0: 0.61, 1: 0.48},         # per-class thresholds
    "conformal_n_calib": {0: 180, 1: 45},          # calibration samples per class
    "conformal_alpha": 0.1,
    "conformal_calibration_fraction": 0.25,
    "conformal_fallback": False,                   # True when the split was not feasible
}

model is the classifier fit on the whole data set (with the hyperparameters selected on the proper part when conformal prediction is on) and gives the point predictions; conformal_model gives the prediction sets. respredai train warns when a class has too few calibration samples for a non-trivial q_hat or when the calibration split is not feasible (see the [Uncertainty] section of Create Config Command).

training_metadata.json#

Contains information needed for evaluation on new data:

{
    "features": ["age", "sex", "category_col"],
    "continuous_features": ["age"],
    "categorical_features": ["sex", "category_col"],
    "targets": ["Target1", "Target2"],
    "feature_names_transformed": ["age", "sex_M", "category_col_A", "..."],
    "feature_dtypes": {"age": "float64", "sex": "object"},
    "training_data_path": "data.csv",
    "training_timestamp": "2025-12-11T...",
    "config": {
        "inner_folds": 3,
        "calibrate_threshold": true,
        "threshold_method": "oof",
        "seed": 42,
        "conformal_alpha": 0.1,
        "calibration_fraction": 0.25,
        "calibration_bins": 10,
        "confidence_level": 0.95,
        "n_bootstrap": 1000,
        "group_column": "PatientID"
    }
}

threshold_method is the resolved method (oof or cv), not the auto placeholder. calibration_bins, confidence_level and n_bootstrap are reused by respredai evaluate for the ECE and MCE metrics and their bootstrap intervals, and group_column (null when no grouping was used) lets evaluate resample whole patients of the new cohort.

Example Workflow#

# 1. Train models
respredai train --config config.ini --output ./trained_output/

# 2. Later, evaluate on new data
respredai evaluate \
    --models-dir ./trained_output/trained_models \
    --data new_patients.csv \
    --output ./evaluation_results/

Calibration#

Probability Calibration (if calibrate_probabilities = true):

  1. Wraps best estimator in CalibratedClassifierCV

  2. Fits on full dataset with internal CV

  3. Saves calibrated model

Threshold Optimization (if calibrate_threshold = true):

  1. Runs GridSearchCV to find best hyperparameters

  2. Selects method: OOF (< 1000 samples) or CV (>= 1000 samples)

  3. OOF: Gets out-of-fold predictions, optimizes threshold using configured objective

  4. CV: Searches the threshold with TunedThresholdClassifierCV on inner CV folds (group-aware splits when a group_column is set, including the probability calibrator inside the search) and keeps the calibrated model fit on the full dataset

  5. Saves optimized threshold with the model

Both calibrations are automatically applied during evaluation.

See Also#