Changelog#
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Unreleased#
1.10.0 - 2026-11-04#
Added#
[Uncertainty] calibration_fractionoption (default 0.25): share of each training set held out to calibrate the conformal prediction sets; 0 disables conformal predictionConformal diagnostics in the metrics CSVs (per-class calibration size and
q_hat,conformal_status), with a warning when a class has too few calibration samplesPooledcolumn in the metrics CSVs: the metric on the pooled out-of-fold predictions, to which the confidence interval refersBootstrap confidence intervals in the subgroup CSVs, with an
Unreliableflag for small subgroups, and inrespredai evaluate
Changed#
Split conformal (Mondrian) prediction replaces CV+: per-class coverage of at least
1 - alpha, with hyperparameters tuned without the calibration split (reported metrics change)Bootstrap confidence intervals stratified by class, and much faster to compute
Coverage row
guaranteed_coveragerenamed tonominal_coverageGroup-aware probability calibration inside the threshold search
HTML report shows each interval next to the pooled estimate and marks subgroup tables as descriptive
training_metadata.jsonrecordsconfidence_level,n_bootstrap,group_columnandcalibration_fractionModel bundles from earlier versions still work in
respredai evaluate, with a warning that their prediction sets carry no coverage guarantee
Removed#
is_cv_plusparameter ofconformal_coverage_report
Fixed#
With repeated CV, confidence intervals and subgroup tables used a threshold chosen on the test labels
Conformal coverage reported as a
1 - 2 alphaguarantee that CV+ did not provideBootstrap confidence intervals ignored the
group_column(too narrow with several samples per patient) and scored single-class resamples as AUROC 0.5 and MCC 0Subgroup ECE and MCE ignored
calibration_bins[nan, nan]for empty intervals in the HTML temporal tableInline comments (
key = value # comment) were read as part of the value, so the README example configuration and template lines uncommented as written failed validation
1.9.4 - 2026-09-30#
Added#
Per-fold feature-importance CSV (
{model}_feature_importance_per_fold.csv) with one row per outer CV fold and, for linear models, the intercept of each foldDocumentation on reading linear coefficients: they refer to standardized and one-hot encoded features and are taken before probability calibration; the
feature-importancepage now shows how they relate to the calibrated probabilities and how to extract them from the saved modelsConfiguration validation: model names are checked against the available models (a misspelt name used to be skipped at run time with a warning while the run reported success),
outer_foldsandinner_foldsmust be at least 2,verbositymust be 0, 1 or 2, and an emptytargetsis rejectedWarning when a categorical feature has high cardinality (at least 20 distinct values and more than half as many as rows), since it is one-hot encoded into one column per level
training_metadata.jsonrecordscalibration_binsand the resolvedthreshold_method(ooforcvinstead ofauto);respredai evaluateuses the recorded bin count for ECE and MCETemporal validation reports progress in the CLI
Checkpoints store the per-fold reliability-curve data, so a resumed or re-loaded run still produces reliability curves
Changed#
extract_feature_importance_from_models()also returns the per-fold interceptsLogistic-regression hyperparameter grid expressed with
l1_ratioon scikit-learn 1.8 or newer (same 71 combinations), avoiding the deprecation ofpenaltyand its removal planned for 1.10Model names are case-insensitive and normalised to their canonical spelling (
lrselectsLR), duplicates are dropped; applies to[Pipeline] models,run --models,train --modelsandfeature-importance --modelcontinuous_featuresmay be empty or omitted when every feature is categoricalWith imputation disabled, the missing-value check covers the feature columns only and names the offending columns; subgroup columns may contain missing values, which are reported as
UnknownFeature-name sanitisation also replaces
[and], which XGBoost rejects in feature names, in addition to<and>With
threshold_method = cvthe saved fold model is the (calibrated) classifier itself with the tuned threshold stored beside it, as foroof;TunedThresholdClassifierCVis used only to search the threshold. With agroup_columnthat search runs on group-aware inner splits, while the calibrator refit inside the search uses plain stratified folds
Removed#
tqdmdependency (unused);torchis no longer required unless thetabpfnextra is installed
Fixed#
import respredaifailed withouttorch, which has not been a dependency of the base package sincetabpfnbecame an optional extra;torchis now imported only when theTabPFNmodel is builtResuming a run from a partially completed checkpoint raised
KeyErrorand wrote a corrupted checkpoint (duplicated metric rows, missing fold models,completed_foldsequal to the total); the per-fold bookkeeping is now recorded in one step after every fallible computationthreshold_method = cv(also selected byautofrom 1000 training rows) with agroup_columnfailed every fold, because sklearn rejectsgroups=onTunedThresholdClassifierCVwithout metadata routing; the group-aware splits are now passed as a precomputed listrespredai trainwith agroup_columnandcalibrate_probabilities = truecrashed with “indices are out-of-bounds” in the conformal stepTemporal validation swallowed per-target failures and the CLI reported success; a failed model/target now raises
RuntimeErrorlike the CV pipelineThe HTML report assumed
CI95_*columns, so confidence intervals disappeared whenconfidence_levelwas not 0.95 while the headers still said 95%Progress bars used
outer_foldsinstead of folds times repeats withouter_cv_repeats > 1Loading several configurations in one process added a log file handler each time, so later runs also logged to the earlier files
feature-importancedropped the sign ofLRandLinear_SVCcoefficients, so protective (negative) coefficients were reported as positiveWith
calibrate_probabilities = true, feature importance came from a single calibration sub-model instead of the whole fold model; it is now averaged over all sub-models
1.9.3 - 2026-06-16#
Added#
Input validation: targets must be binary 0/1 with both classes present, declared
continuous_featuresmust exist in the data, and group/temporal columns must not contain missing values;respredai evaluateuses the same reader and validator as training[Pipeline] calibration_binsoption (default 10) to set the number of bins for the ECE/MCE metrics (point estimates and bootstrap confidence intervals)Importance provenance: feature-importance CSVs include
Importance_Type(gain, impurity, coefficient, mean_abs_shap - not comparable across model families) andN_folds_present; aNeutral (~0)direction label is reported for features with no clear directionReproducibility manifest records
n_jobs, the git commit and the full installed-package versionsCI job running the slow (integration) test suite
Changed#
Cross-fold feature-importance aggregation skips folds where a feature is absent instead of filling zeros, so rare one-hot categories are not diluted toward zero
Fixed#
Empty results with the default config:
respredai runwithcalibrate_threshold = false(thecreate-configdefault) raised anUnboundLocalErrorper fold that was swallowed, producing empty/NaN metrics while reporting successSilent fold/target failures: per-fold errors are warned and counted, and a model/target whose folds all fail raises a clear
RuntimeErrorinstead of writing NaN metrics as successfeature-importance --directionsilently returned nothing for RandomForest because the 3-D SHAP TreeExplainer output was not handled; SHAP output of any shape is now reduced to the positive classCalibration bin collapse: quantile bin edges are deduplicated so tied probabilities no longer collapse ECE/MCE/reliability bins
calibrate_probabilities = truewith agroup_columnmade every fold fail (“indices are out-of-bounds”) in the conformal step
1.9.2 - 2026-05-22#
Added#
TABPFN_TOKENenvironment variable (PriorLabs API token) required for TabPFN v3; the pipeline validates it before starting whenTabPFNis among the requested modelsensure_tabpfn_available()helper inrespredai.core.model_builderthat checks both the installed package and the token
Changed#
TabPFN is an optional dependency: install with
pip install respredai[tabpfn]to enable the TabPFN modelTabPFN bumped to
>=8.0.0,<9.0.0and switched to the v3 model by default
1.9.1 - 2026-04-13#
Fixed#
Group leakage in
perform_trainingconformal CV:StratifiedGroupKFoldis used when groups are available, matchingperform_pipelineandperform_temporal_validation
1.9.0 - 2026-04-10#
Added#
FOR (False Omission Rate) metric in all metrics CSVs and HTML reports
CV+ Mondrian conformal prediction: per-class prediction sets
{S},{R}or{S, R}with distribution-free coverage guarantee1 - 2α, computed per fold inside nested CV;q_hatper class saved in model bundles; dedicated HTML report section with per-model coverage tables[Uncertainty] alphaconfig option (miscoverage rate)Model category constants
SHAP_FALLBACK_MODELSandNO_CLASS_WEIGHT_MODELSinconstants.pyDocumentation on configuring the
[Metadata]section for subgroup analysis
Changed#
Feature importance functions (
has_feature_importance,get_feature_importance,compute_shap_importance) require amodel_nameparameter and dispatch on category constants instead of duck-typing
Removed#
Legacy config fallback for
group_columnin[Data]andtemporal_split_columnin[Validation](deprecated since 1.8.0); use the[Metadata]sectioncalculate_uncertainty()function, replaced bycompute_conformal_qhat()andconformal_prediction_sets()[Uncertainty] marginconfig key, replaced by[Uncertainty] alphauncertainty_marginfield in model bundles, replaced byconformal_q_hatandconformal_alpha
1.8.0 - 2026-03-25#
Added#
Unified
[Metadata]config section:group_column(moved from[Data]),temporal_column(moved from[Validation]astemporal_split_column) and the new comma-separatedsubgroup_columnsSubgroup performance evaluation: full metric set (AUROC, F1, MCC, Precision, Recall, ECE, MCE, Brier Score) per subgroup value, with sample size and class prevalence, for both CV and temporal validation; results saved as CSV in
subgroup_analysis/and shown in the HTML report; warns when a subgroup has fewer than 10 samplesSigned feature importance direction (
compute_feature_directionconfig flag,respredai feature-importance --direction): linear models use the sign of the coefficients, tree-based models useshap.TreeExplainer, other models fall back toshap.KernelExplainer; adds aDirectioncolumn (Risk (+)/Protective (-)) to the CSV and colors the plot by direction
Deprecated#
group_columnin[Data]andtemporal_split_columnin[Validation]; use the[Metadata]section
1.7.1 - 2026-03-23#
Added#
BCa (bias-corrected and accelerated) bootstrap confidence intervals via
scipy.stats.bootstrap, replacing percentile bootstrap, for better coverage on small samples and bounded or skewed metricsNadeau-Bengio corrected standard error: new
SEcolumn in the metrics CSV using(1/k + n_test/n_train) * s**2, accounting for training-set overlap in k-fold CV; the summary report±notation uses SE instead of raw Stdconfidence_levelandn_bootstrapoptions in the[Pipeline]config sectionrespredai/core/constants.pywith centralized validation lists, directory names and defaults
Changed#
ConfigHandlersplit into seven domain-specific dataclassesConfig validation lists reference the centralized constants
README quick-start config example shows the
[Validation]sectionHTML report confidence-interval row reflects the configured
confidence_levelandn_bootstrap
Fixed#
Name sanitization unified through
sanitize_name()/sanitize_metric_name()assertin the temporal split replaced with a properValueErrorEmpty CV folds are validated, with a warning when train or test sets are empty after splitting
Explicit warning when all bootstrap samples fail (previously returned NaN silently)
1.7.0 - 2026-03-18#
Added#
Temporal (prospective-style) validation:
validation_strategyconfig option (cv,temporalorboth),temporal_split_column,temporal_split_dateandtemporal_split_ratiooptions, group-aware temporal splitting,--validation-strategyCLI override forrun, and a temporal validation section in the HTML reportPer-fold one-hot encoding fitted on the training part of each CV fold instead of the full dataset, preventing category leakage
NaN-safe scaling for KNN imputation: features are pre-scaled with NaN-tolerant statistics before distance-based imputation
scale_pos_weightin the XGBoost hyperparameter grid
Changed#
validate-configsummary table shows the validation strategy and temporal split parameterscreate-configtemplate includes a commented[Validation]sectionuncertainty_marginstored in training metadata and model bundles
Fixed#
SVC models set
probability=Falsewhen external calibration is enabled, avoiding double Platt scalingTunedThresholdClassifierCV(CV threshold method) usesStratifiedGroupKFoldand passes group labels when grouped CV is configuredRepeated CV deduplication uses the threshold-aware decision boundary instead of a fixed 0.5 cutoff
All
nanstdcalls useddof=1for the unbiased sample standard deviationReliability curve binning replaced with a self-contained implementation to keep
bin_countsalignedLogger initialization deferred until after CLI overrides so the log file uses the correct output folder
Feature importance name resolution improved with a multi-source fallback
evaluatereuses the saved one-hot encoder from training instead of ad-hocpd.get_dummies
1.6.2 - 2026-03-05#
Added#
Makefile for development workflows
Fixed#
Bootstrap confidence intervals deduplicate samples when using repeated outer CV
Threshold optimization (CV method) uses the calibrated estimator when probability calibration is enabled
Metrics aggregation respects the repeat structure
Reliability curve fold labels indicate the repeat number when using repeated CV
Reliability curves use quantile binning for smoother calibration plots on imbalanced data
1.6.1 - 2026-02-06#
Changed#
create-configtemplate includesthreshold_objective,vme_costandme_costvalidate-configsummary table shows probability calibration and threshold objective details
Fixed#
trainapplies probability calibration (CalibratedClassifierCV) whencalibrate_probabilities = truetrainsupports the CV threshold method (TunedThresholdClassifierCV) in addition to OOFReproducibility manifest includes the probability calibration parameters
1.6.0 - 2026-02-05#
Added#
Optional post-hoc probability calibration (
sigmoidPlatt scaling orisotonic) on the best estimator of each outer CV fold, applied after hyperparameter tuning and before threshold tuningCalibration diagnostics: Brier Score, ECE (Expected Calibration Error), MCE (Maximum Calibration Error) and reliability curves per outer CV fold and aggregate
Repeated stratified cross-validation via the
outer_cv_repeatsconfig option (default 1)
Changed#
metric_dict()includes Brier Score, ECE and MCE by defaultHTML report includes a calibration diagnostics section
Output folder includes a
calibration/directory with reliability curve images
1.5.1 - 2026-01-29#
Added#
OneHotEncoder
min_frequencyparameter to reduce noise from rare categorical values
Changed#
requirements.txtpins explicit version constraints for all dependencies (scikit-learn>=1.5.0is required forTunedThresholdClassifierCV)
1.5.0 - 2026-01-20#
Added#
VME (Very Major Error: predicted susceptible when resistant) and ME (Major Error: predicted resistant when susceptible) in reports
Flexible threshold objectives:
threshold_objectiveconfig option (youden,f1,f2,cost_sensitive) with configurablevme_costandme_costweightsPer-prediction uncertainty quantification:
uncertainty_marginconfig option (default 0.1) flags predictions near the decision threshold, with 0-1 uncertainty scores for each predictionReproducibility manifest (
reproducibility.json) generated byrunandtrainwith environment info, data fingerprint and full configuration
Changed#
HTML report framework summary shows the threshold objective and cost weights
Evaluation output includes
uncertaintyandis_uncertaincolumns
1.4.1 - 2026-01-15#
Changed#
Documentation migrated from MkDocs to Sphinx
Documentation and development dependencies loaded from
docs-requirements.txtanddev-requirements.txt
1.4.0 - 2026-01-14#
Added#
K-Nearest Neighbors (KNN) classifier
Missing data imputation:
SimpleImputer(mean,median,most_frequent),KNNImputerandIterativeImputerwithBayesianRidgeorRandomForestestimatorHTML report with run metadata, framework summary, results tables with 95% confidence intervals and confusion matrices
Ruff linter in the CI workflow
Changed#
Bootstrap confidence intervals use sample-level predictions instead of fold-level metrics
CI workflow runs lint checks before tests and includes Python 3.13
1.3.1 - 2026-01-08#
Added#
docs/structure built with MkDocs
Changed#
Package reorganized into sub-packages:
respredai/core/(pipeline, metrics, models),respredai/io/(configuration and data handling) andrespredai/visualization/(plots)
1.3.0 - 2025-12-12#
Added#
traincommand for training on the entire dataset (cross-dataset validation): GridSearchCV tuning with inner CV only, one model file per model-target combination,training_metadata.jsonfor evaluation compatibilityevaluatecommand to apply trained models to new data: validates columns against the training metadata, outputs per-sample predictions with probabilities and computes metrics against ground truthAutomatic summary report after
run:summary.csvper target andsummary_all.csvglobally with Mean±Std for all metrics across modelsSHAP-based feature importance as fallback for models without native importance (MLP, RBF_SVC, TabPFN via KernelExplainer): mean absolute SHAP values across CV test folds,
_shapsuffix on output files,--seedflag for reproducibilityDocumentation pages for the
trainandevaluatecommands
Changed#
feature-importancedocumentation describes the SHAP fallback
1.2.0 - 2025-12-10#
Added#
validate-configcommand to validate configuration files without running the pipeline, with an optional--check-dataflag to verify the data file and its columnsCLI override options for
run:--models,--targets,--output,--seedCONTRIBUTING.mdwith development setup and contribution workflowDocumentation page for the
validate-configcommand
Changed#
Bootstrap confidence intervals (10,000 resamples) replace the t-distribution CI in the metrics output
User-friendly error messages for missing config files or data paths
rundocumentation describes the CLI overrides
1.1.0 - 2025-12-04#
Added#
Threshold optimization with Youden’s J statistic using either the OOF method (global optimization on concatenated out-of-fold predictions) or the CV method (per-fold optimization with threshold averaging), selected automatically by dataset size (OOF below 1000 samples, CV otherwise)
Grouped cross-validation (
StratifiedGroupKFold) to prevent data leakage in clinical datasetsCommand documentation for
run,create-configandfeature-importanceThis changelog
Changed#
Expanded hyperparameter grids for XGBoost, Random Forest, CatBoost and MLP
Enhanced CLI information display
README with logo, quick start guide and output structure
Fixed#
XGBoost feature naming with special characters
Color scheme in feature importance plots
1.0.0 - 2025-12-01#
Added#
Nested cross-validation framework (outer loop for evaluation, inner loop for hyperparameter tuning)
Eight machine learning models: LR, Linear SVC, RBF SVC, MLP, RF, XGBoost, CatBoost, TabPFN
Metrics: Precision, Recall, F1, MCC, Balanced Accuracy, AUROC
Data preprocessing: StandardScaler, one-hot encoding, multi-target support
INI-based configuration system
Structured output: CSV metrics, confusion matrix plots, logs
feature-importancecommand with plot and CSV export