mlchem.ml.modelling package

Model evaluation utilities keep default execution quiet, and expose optional logging diagnostics where relevant (for example in y_scrambling). SHAP plotting paths also perform explicit shape-layout validation for safer error handling.

Submodules

mlchem.ml.modelling.model_evaluation module

class ApplicabilityDomain

Bases: object

A class to calculate the leverage of data points in a dataset for applicability domain analysis.

The leverage is a measure of the influence of a data point in a regression model. It helps identify data points that have a significant impact on the model’s predictions. This class provides a method to calculate the leverage values for a given dataset and determine whether each data point is within the applicability domain based on a threshold.

Methods:

leverage(X: np.ndarray):

Calculates the leverage values for the given dataset and determines whether each data point is within the applicability domain based on a threshold.

static leverage(X: ndarray) → dict[str, list[float] | list[bool] | float]

Calculate leverage values for a dataset and determine applicability domain.

Parameters:

X (numpy.ndarray) – Feature matrix of shape (n_samples, n_features).

Returns:

Dictionary containing:

  • 'leverages' (list of float): leverage values for each data point.

  • 'results' (list of bool): boolean flags indicating whether each point is within the domain.

  • 'threshold' (float): threshold used to determine domain inclusion.

Return type:

dict of str to list or float

class MajorityVote

Bases: object

MajorityVote ensemble evaluation utility.

This class supports two workflows:

  1. evaluate_cv() for fold-based evaluation and estimator ranking.

  2. fit() for final consensus fitting on full train/test data.

Parameters:

  • train_set (pandas.DataFrame) – Training feature matrix.

  • y_train (iterable) – Training labels/targets.

  • task_type ({'classification', 'regression'}) – Task type used to select voting strategy.

  • estimator_list (list) – List of scikit-learn compatible estimators.

  • column_list (list[list[str]]) – Per-estimator feature subsets aligned with estimator_list.

  • test_set (pandas.DataFrame or None, optional) – Hold-out feature matrix used by fit().

  • y_test (iterable or None, optional) – Hold-out labels/targets used by fit().

  • estimator_names (list[str] or None, optional) – Optional user-defined estimator names.

  • n_jobs (int, optional) – Number of workers used by parallel sections.

  • log_level (int or str, optional) – Logging threshold.

__init__(train_set: DataFrame, y_train: Iterable, task_type: Literal['classification', 'regression'], estimator_list: list, column_list: list[list[str]], test_set: DataFrame | None = None, y_test: Iterable | None = None, estimator_names: list[str] | None = None, n_jobs: int = 1, log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20) → None
evaluate_cv(metric: Callable, metric_name: str, cv_iter: int = 5, cv_splitter=None, cv_indices: Iterable | None = None, groups: ndarray | Series | None = None, n_estimators_max: int = 5, logic: Literal['lower', 'greater'] = 'greater', desired_performance_score: Literal['train', 'cv', 'train_cv_average'] = 'train_cv_average', shuffle: bool = False, random_state: int | None = None) → DataFrame

Evaluate ensemble combinations using fold-level cross-validation.

The same explicit fold definitions are reused for every estimator and combination, ensuring fair fold-wise comparison.

Parameters:

  • metric (callable) – Scoring function accepting (y_true, y_pred).

  • metric_name (str) – Label appended to generated combination identifiers.

  • cv_iter (int, optional (default=5)) – Number of folds used when cv_indices and cv_splitter are not supplied.

  • cv_splitter (object, optional) – Scikit-learn compatible splitter exposing split(X, y, groups).

  • cv_indices (iterable of (train_idx, valid_idx), optional) – Explicit precomputed fold manifest. When supplied, all combinations are evaluated on exactly these folds.

  • groups (array-like, optional) – Group labels forwarded to group-aware cv_splitter objects.

  • n_estimators_max (int, optional (default=5)) – Maximum combination size passed to generate_combination_cascade.

  • logic ({'lower', 'greater'}, optional (default='greater')) – Direction of optimization for reliability computation.

  • desired_performance_score ({'train', 'cv', 'train_cv_average'}, optional) – Performance component used inside get_reliability_score_components.

  • shuffle (bool, optional (default=False)) – Whether generated CV folds are shuffled when using cv_iter.

  • random_state (int or None, optional) – Random seed used with generated shuffled folds.

Returns:

One row per evaluated ensemble variant. Columns include: combination, train_mean, train_se, validation_mean, validation_se, reliability_mean, reliability_se, train_folds, validation_folds, reliability_folds. Results are ranked by reliability_mean descending.

Return type:

pandas.DataFrame

fit(selected_estimators: Iterable[int | str] | None = None, update_active_estimators: bool = False, build_consensus: bool = True, return_prediction_dataframes: bool = False) → dict[str, list[str] | int] | dict

Hold-out fitting using train and test sets.

Parameters:

  • selected_estimators (iterable of int or str, optional) – Optional subset of estimators to fit, provided as indices and/or resolved estimator names. If None, all estimators are fitted.

  • update_active_estimators (bool, optional (default=False)) – If True, permanently replace estimator_list / column_list / estimator_names with the selected subset after resolution. This is useful for intentionally narrowing the final consensus fit stage to shortlisted estimators.

  • build_consensus (bool, optional (default=True)) – For classification tasks, build transparent hold-out prediction tables with per-estimator probability/class columns plus CONSENSUS_hard and CONSENSUS_soft.

  • return_prediction_dataframes (bool, optional (default=False)) – If True, return the generated prediction DataFrames together with the fit report.

Returns:

Fit report with keys: requested_count, successful_count, failed_count, successful_estimators, failed_estimators. When return_prediction_dataframes=True, returns a dictionary containing the fit report and generated prediction DataFrames.

Return type:

dict

crossval(estimator, X: ndarray | DataFrame, y: ndarray | DataFrame, metric: Callable, n_fold: int = 5, task_type: Literal['classification', 'regression'] = 'classification', random_state: int | None = None, shuffle: bool = False, cv_splitter=None, cv_indices: Iterable | None = None, groups: ndarray | Series | None = None) → dict[str, ndarray | float]

Evaluate an estimator using cross-validation.

This function performs K-fold cross-validation on the given dataset using the specified estimator and metric function. It supports both classification and regression tasks.

By default (cv_splitter=None and cv_indices=None), behaviour is unchanged from previous releases: n_fold folds are generated internally using StratifiedKFold/KFold. For fully index-driven, realistic validation strategies (GroupKFold, StratifiedGroupKFold, PredefinedSplit, scaffold-based folds, UMAP-cluster folds, or any externally generated fold manifest), pass cv_splitter and/or cv_indices instead. Internally, all cross-validation execution operates on explicit, validated train/validation index pairs regardless of which input mode is used (see generate_cv_indices()).

Parameters:

  • estimator (object) – A scikit-learn compatible estimator.

  • X (numpy.ndarray or pandas.DataFrame) – Feature matrix of shape (n_samples, n_features).

  • y (numpy.ndarray or pandas.DataFrame) – Target vector of shape (n_samples,) or (n_samples, 1).

  • metric (callable) – A scoring function that accepts (y_true, y_pred) as arguments.

  • n_fold (int, optional (default=5)) – Number of folds for cross-validation. If equal to n_samples, performs leave-one-out cross-validation. Ignored when cv_splitter or cv_indices is provided.

  • task_type ({'classification', 'regression'}, optional (default='classification')) – Type of task to determine the cross-validation strategy.

  • random_state (int or None, optional (default=None)) – Random seed for reproducibility.

  • shuffle (bool, optional (default=False)) – Whether to shuffle samples before splitting into batches.

  • cv_splitter (object, optional) – A scikit-learn compatible cross-validation splitter (e.g. GroupKFold(n_splits=5), StratifiedGroupKFold(...), or PredefinedSplit(...)). When provided, it takes precedence over n_fold/task_type-based fold generation. Combine with groups for group-aware splitters.

  • cv_indices (iterable of (array-like, array-like), optional) – Explicit, pre-computed (train_idx, valid_idx) pairs (e.g. a scaffold-based or UMAP-cluster-based fold manifest generated outside mlchem). Takes precedence over both cv_splitter and n_fold when provided. This is the most generic API.

  • groups (array-like, optional) – Group labels propagated to cv_splitter.split (e.g. scaffold IDs for GroupKFold). Ignored when cv_indices is provided.

Returns:

Dictionary with fold-level arrays and summary statistics: train_scores, train_mean, train_se, cv_scores, cv_mean, cv_se.

Return type:

dict

generate_cv_indices(X: ndarray | DataFrame, y: ndarray | DataFrame | None = None, cv_iter: int = 5, cv_splitter=None, cv_indices: Iterable | None = None, groups: ndarray | Series | None = None, task_type: Literal['classification', 'regression'] = 'classification', shuffle: bool = False, random_state: int | None = None, strict_coverage: bool = False) → list[tuple[ndarray, ndarray]]

Resolve any supported cross-validation input into explicit fold indices.

This is the single entry point that converts cv_iter (existing behaviour), a scikit-learn cv_splitter (optionally combined with groups), or a user-supplied cv_indices manifest into the same canonical representation: a validated list of (train_idx, valid_idx) index pairs. Fold-generation logic external to mlchem (scaffold splits, UMAP-cluster splits, etc.) only needs to produce cv_indices in this format.

Parameters:

  • X (numpy.ndarray or pandas.DataFrame) – Feature matrix of shape (n_samples, n_features). Only its length is used unless a cv_splitter requires X for splitting.

  • y (numpy.ndarray, pandas.DataFrame, or None, optional) – Target vector, forwarded to cv_splitter.split when needed (e.g. for stratified splitters).

  • cv_iter (int, optional (default=5)) – Number of folds to generate when neither cv_splitter nor cv_indices is supplied. Mirrors the legacy behaviour.

  • cv_splitter (object, optional) – A scikit-learn compatible cross-validation splitter (e.g. GroupKFold, StratifiedGroupKFold, PredefinedSplit) exposing a split(X, y, groups) method.

  • cv_indices (iterable of (array-like, array-like), optional) – Explicit, pre-computed (train_idx, valid_idx) pairs. Takes precedence over cv_splitter and cv_iter when provided. This is the most generic API and is suitable for externally generated fold manifests (scaffold-based folds, UMAP-cluster folds, etc.).

  • groups (array-like, optional) – Group labels used by group-aware splitters such as GroupKFold or StratifiedGroupKFold. Ignored unless cv_splitter is provided and supports grouping.

  • task_type ({'classification', 'regression'}, optional (default='classification')) – Used only to pick the default splitter (StratifiedKFold or KFold) when neither cv_splitter nor cv_indices is given.

  • shuffle (bool, optional (default=False)) – Whether to shuffle samples before splitting, for the default cv_iter-based splitter.

  • random_state (int or None, optional (default=None)) – Random seed for the default cv_iter-based splitter when shuffle=True.

  • strict_coverage (bool, optional (default=False)) – Forwarded to validate_cv_indices().

Returns:

Validated (train_idx, valid_idx) pairs.

Return type:

list of (numpy.ndarray, numpy.ndarray)

summarise_fold_scores(scores: Iterable[float]) → dict

Compute mean and standard error from fold scores.

Parameters:

scores (Iterable[float]) – Iterable of fold scores.

Returns:

Dictionary containing the fold scores, the mean and standard error of the fold scores.

Return type:

dict

validate_cv_indices(cv_indices: Iterable, n_samples: int, strict_coverage: bool = False) → list[tuple[ndarray, ndarray]]

Validate an explicit list of (train_idx, valid_idx) cross-validation folds.

Parameters:

  • cv_indices (iterable of (array-like, array-like)) – Explicit train/validation index pairs, one per fold.

  • n_samples (int) – Number of samples the indices are expected to reference.

  • strict_coverage (bool, optional (default=False)) – If True, raise a ValueError when some samples are never used as validation data in any fold. If False (default), only emit a RuntimeWarning in that case, since some valid fold manifests (e.g. a PredefinedSplit excluding held-out samples) may legitimately not cover every sample.

Returns:

Normalised, validated (train_idx, valid_idx) pairs.

Return type:

list of (numpy.ndarray, numpy.ndarray)

Raises:

ValueError – If folds overlap, contain out-of-range/duplicate indices, are empty, or are otherwise inconsistently defined.

y_scrambling(estimator, train_set: ndarray | DataFrame, y_train: Iterable, metric: Callable, n_scrambles: int = 100, n_fold: int = 5, cv_splitter=None, groups=None, cv_indices: Iterable | None = None, logic: Literal['lower', 'greater'] = 'greater', task_type: Literal['classification', 'regression'] = 'classification', plot: bool = True, n_jobs: int = 1, safety_multiplier: float = 2.3, log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20) → dict[str, float | ndarray]

Perform y-scrambling to estimate chance-level model performance.

The target vector is shuffled n_scrambles times, each shuffled target is cross-validated, and the null-score distribution is compared with the score obtained from the unshuffled labels.

Parameters:

  • estimator (object) – Scikit-learn compatible estimator.

  • train_set (numpy.ndarray or pandas.DataFrame) – Training feature matrix.

  • y_train (iterable) – Target values.

  • metric (callable) – Scoring function metric(y_true, y_pred).

  • n_scrambles (int, optional) – Number of shuffled-label iterations. Default is 100.

  • n_fold (int, optional) – Number of CV folds when explicit folds are not provided.

  • cv_splitter (object, optional) – CV splitter overriding n_fold.

  • groups (array-like, optional) – Group labels passed to group-aware splitters.

  • cv_indices (iterable, optional) – Explicit fold definitions overriding cv_splitter and n_fold.

  • logic ({'lower', 'greater'}, optional) – Score direction used for comparisons.

  • task_type ({'classification', 'regression'}, optional) – Task type used by default CV strategy.

  • plot (bool, optional) – If True, display a histogram of scrambled scores.

  • n_jobs (int, optional) – Number of parallel workers. Use -1 for all CPUs.

  • safety_multiplier (float, optional) – Multiplier defining safety_margin = safety_multiplier * scrambled_std.

  • log_level (int or str, optional) – Logging threshold.

Returns:

Mapping with keys:

  • reference_score

  • scrambled_scores

  • scrambled_std

  • best_random_score

  • probability_better

  • obtained_margin

  • safety_margin

  • safety_margin_ratio

  • safety_multiplier

Return type:

dict

mlchem.ml.modelling.model_interpretation module

class DescriptorExplainer

Bases: object

A class to perform feature selection and model evaluation using combinatorial selection methods.

This class provides methods to fit and evaluate machine learning models using combinatorial feature selection. It supports both classification and regression tasks, offering tools to identify the best feature subsets and visualise model performance.

df_train

The training dataset.

Type:

pandas.DataFrame

df_test

The testing dataset.

Type:

pandas.DataFrame

target_train

The target values for the training dataset.

Type:

pandas.DataFrame

target_test

The target values for the testing dataset.

Type:

pandas.DataFrame

target_name

The name of the target variable.

Type:

str

estimator

The machine learning model to be used for feature selection and evaluation.

Type:

object

metric

The metric function to evaluate the model performance.

Type:

callable

logic

The logic to determine whether to minimise or maximise the metric.

Type:

{‘lower’, ‘greater’}

task_type

The type of task to perform.

Type:

{‘classification’, ‘regression’}

__init__(df_train: DataFrame, df_test: DataFrame, target_train: DataFrame, target_test: DataFrame, estimator, metric, logic: Literal['lower', 'greater'] = 'greater', task_type: Literal['classification', 'regression'] = 'regression', n_jobs: int = 1, log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20, cv_splitter=None, cv_indices=None, groups=None) → None

Initialise the DescriptorExplainer with training/testing data, model, and evaluation settings.

Parameters:

  • df_train (pandas.DataFrame) – The training dataset.

  • df_test (pandas.DataFrame) – The testing dataset.

  • target_train (pandas.DataFrame) – The target values for the training dataset.

  • target_test (pandas.DataFrame) – The target values for the testing dataset.

  • estimator (object) – The machine learning model to be used.

  • metric (callable) – The evaluation metric function.

  • logic ({'lower', 'greater'}, optional (default='greater')) – Whether to minimise or maximise the metric.

  • task_type ({'classification', 'regression'}, optional (default='regression')) – The type of task to perform.

  • n_jobs (int, optional (default=1)) – Number of parallel workers for feature selection stages. -1 uses all available CPUs.

  • log_level (int or str or Literal, optional (default=logging.INFO)) – Logging level for diagnostics. Can be integer (10, 20, 30, etc.) or string (‘DEBUG’, ‘INFO’, etc.).

  • cv_splitter (object, optional) – A scikit-learn compatible cross-validation splitter (e.g. GroupKFold(5), StratifiedGroupKFold(...), PredefinedSplit(...)). Combine with groups for group-aware splitters. Takes precedence over the cv_iter passed to fit_stage_1/fit_stage_2.

  • cv_indices (iterable of (array-like, array-like), optional) – Explicit, pre-computed (train_idx, valid_idx) pairs, e.g. a scaffold-based or UMAP-cluster-based fold manifest generated outside mlchem. Takes precedence over both cv_splitter and cv_iter.

  • groups (array-like, optional) – Group labels (e.g. scaffold IDs) propagated to cv_splitter. Ignored when cv_indices is provided.

display(subset_index: int) → None

Display the performance of the model using a selected feature subset.

This method visualises model performance using the selected feature subset. For regression tasks, it plots true vs. predicted values and prints model coefficients and adjusted R². For classification, it shows a confusion matrix and prints performance metrics.

Parameters:

subset_index (int) – The index of the feature subset to use for evaluation.

Return type:

None

fit_stage_1(k: int = 2, training_threshold: float = 1.5, cv_train_ratio: float = 0.8, cv_iter: int = 5) → None

Perform the first stage of combinatorial feature selection.

This method evaluates combinations of features using cross-validation and filters subsets based on a training score threshold. Parallelization is controlled by the n_jobs parameter specified during initialization.

Parameters:

  • k (int, optional (default=2)) – The number of features to combine.

  • training_threshold (float, optional (default=1.5)) – The threshold for the training score to consider a subset.

  • cv_train_ratio (float, optional (default=0.8)) – The minimum accepted cv score/train score ratio.

  • cv_iter (int, optional (default=5)) – The number of cross-validation iterations.

Return type:

None

fit_stage_2(top_n_subsets: int = 10, cv_iter: int = 5) → None

Perform the second stage of combinatorial feature selection.

This method evaluates the top feature subsets from stage 1 using cross-validation to identify the best-performing combinations. Parallelization is controlled by the n_jobs parameter specified during initialization.

Parameters:

  • top_n_subsets (int, optional (default=10)) – The number of top feature subsets to consider.

  • cv_iter (int, optional (default=5)) – The number of cross-validation iterations.

Return type:

None

class ShapExplainer

Bases: object

A class to generate SHAP (SHapley Additive exPlanations) values for model interpretability.

This class provides methods to explain the output of machine learning models using SHAP values. It supports both tree-based models and general estimators, and offers a variety of visualisation tools to understand the impact of each feature on model predictions.

model_type

The type of model (‘tree’ or other).

Type:

str

data

The dataset used for generating SHAP values.

Type:

pandas.DataFrame

y

The target values corresponding to the dataset.

Type:

array-like

estimator

The machine learning model to be explained.

Type:

object

Methods:

  • explain() – Generates SHAP values for the model.

  • load(base_values, shap_values) – Loads precomputed SHAP values.

  • save(path, filename) – Saves the SHAP values to the specified path.

  • force_plot() – Generates a force plot to visualise SHAP values.

  • force_plot_single(i) – Generates a force plot for a single instance.

  • dependence_plot(column) – Generates a dependence plot for a specified feature.

  • dependence_plot_all() – Generates dependence plots for all features.

  • summary_plot(plot_type=’dot’) – Generates a summary plot of SHAP values.

  • waterfall_plot(i) – Generates a waterfall plot for a single instance.

  • bar_plot() – Generates a bar plot of SHAP values.

  • feature_importance() – Calculates and returns feature importance based on SHAP values.

  • decision_plot(interval_lower, interval_upper) – Generates a decision plot for a specified probability interval.

__init__(estimator, data: DataFrame, y: Iterable, is_tree: bool = False, max_samples: int | None = None) → None

Initialise the ShapExplainer with a model, dataset, and target values.

Parameters:

  • estimator (object) – The machine learning model to be explained.

  • data (pandas.DataFrame) – The dataset used for generating SHAP values.

  • y (Iterable) – The target values corresponding to the dataset.

  • is_tree (bool, optional (default=False)) – Whether the model is a tree-based model (e.g., XGBoost, LightGBM).

  • max_samples (int or None, optional (default=None)) – Maximum number of samples to use for background dataset. If None, uses all samples. Set to the dataset size to avoid subsampling warnings.

bar_plot() → None

Generate a bar plot of SHAP values.

Return type:

None

decision_plot(interval_lower: float, interval_upper: float) → None

Generate a decision plot for a specified probability interval.

This method creates a SHAP decision plot to visualise how features contribute to model predictions for samples within a given probability interval. It also prints classification statistics for the selected interval.

Note

Supported for binary classifiers with predict_proba output. The method handles both tree and non-tree SHAP layouts and selects the positive class contributions when class-wise SHAP values are provided.

Parameters:

  • interval_lower (float) – The lower bound of the probability interval.

  • interval_upper (float) – The upper bound of the probability interval.

Return type:

None

dependence_plot(column: int) → None

Generate a dependence plot for a specified feature.

Parameters:

column (int) – The index of the feature to visualise.

Return type:

None

dependence_plot_all() → None

Generate dependence plots for all features.

Return type:

None

explain() → None

Generate SHAP values for the model.

This method creates a SHAP explainer based on the model type and computes the SHAP values for the provided dataset.

Return type:

None

feature_importance() → DataFrame

Calculate and visualise feature importance based on SHAP values.

This method computes feature importance by summing the absolute SHAP values for each feature. It also calculates the Spearman correlation between each feature and its SHAP values, and derives an overall impact score. The results are visualised using a color-coded bar plot and returned as a DataFrame.

Returns:

A DataFrame containing the feature importance weights, correlations, and overall impact scores for each feature.

Return type:

pandas.DataFrame

force_plot() → HTML

Generate a force plot to visualise SHAP values.

Returns:

An HTML object containing the SHAP force plot.

Return type:

IPython.core.display.HTML

force_plot_single(i: int) → HTML

Generate a force plot for a single instance.

Parameters:

i (int) – The index of the instance to visualise.

Returns:

An HTML object containing the SHAP force plot for the specified instance.

Return type:

IPython.core.display.HTML

heatmap() → None

Generate a heatmap of SHAP values.

This method creates a heatmap to visualise the SHAP values for the model’s predictions. It initialises the SHAP JavaScript visualisation and displays the plot.

Note

Heatmaps are not available for tree-based models.

Return type:

None

load(base_values: ndarray, shap_values: ndarray) → None

Load precomputed SHAP values.

Parameters:

  • base_values (numpy.ndarray) – The precomputed base values.

  • shap_values (numpy.ndarray) – The precomputed SHAP values.

Return type:

None

save(path: str, filename: str) → None

Save SHAP values to the specified path.

Parameters:

  • path (str) – The directory path where the SHAP values will be saved.

  • filename (str) – The base filename for the saved SHAP values.

Return type:

None

summary_plot(plot_type: Literal['dot', 'bar', 'violin', 'layered_violin', 'compact_dot'] = 'dot') → None

Generate a summary plot of SHAP values.

Parameters:

  • plot_type ({'dot', 'bar', 'violin', 'layered_violin', 'compact_dot'},)

  • (default='dot') (optional) – The type of plot to generate.

Return type:

None

waterfall_plot(i: int) → None

Generate a waterfall plot for a single instance.

Note: Not available for tree-based models.

Parameters:

i (int) – The index of the instance to visualise.

Return type:

None