mlchem.ml.feature_selection package

The wrapper selectors support optional runtime diagnostics through the log_level constructor argument and the set_log_level(…) runtime toggle.

Submodules

mlchem.ml.feature_selection.filters module

collinearity_filter(df: DataFrame, threshold: float, target_variable: str = None, method: Literal['pearson', 'kendall', 'spearman'] = 'pearson', numeric_only: bool = False) → DataFrame

Filter features based on collinearity threshold.

Returns a subset of DataFrame columns whose squared correlation (R²) values are below the specified threshold. If a target variable is provided, the function retains the feature with the higher correlation to the target when multiple features are collinear.

Parameters:

  • df (pandas.DataFrame) – The input dataset.

  • threshold (float) – The maximum allowed squared correlation between features.

  • target_variable (str, optional) – The name of the target variable. If provided, it is used to resolve collinearity conflicts.

  • method ({'pearson', 'kendall', 'spearman'}, optional) – The correlation method to use. Default is ‘pearson’.

  • numeric_only (bool, optional) – Whether to include only numeric columns. Default is False.

Returns:

A DataFrame containing the filtered columns.

Return type:

pandas.DataFrame

diversity_filter(df: DataFrame, threshold: float, target_variable: str = None) → DataFrame

Filter features based on diversity ratio using Shannon entropy.

Calculates the diversity ratio of each feature by comparing its Shannon entropy to that of an ideal uniform distribution. Retains features with diversity ratios above the specified threshold.

Parameters:

  • df (pandas.DataFrame) – The input dataset.

  • threshold (float) – The minimum diversity ratio required to retain a feature.

  • target_variable (str, optional) – The name of the target variable to retain regardless of its diversity score.

Returns:

A DataFrame containing the filtered columns with diversity higher than the threshold.

Return type:

pandas.DataFrame

mlchem.ml.feature_selection.wrappers module

class CombinatorialSelection

Bases: object

Combinatorial feature selection wrapper.

This class performs a two-stage combinatorial search over feature subsets and ranks the surviving subsets with the same reliability score logic used by SequentialForwardSelection: the performance term is controlled by desired_performance_score, and can correspond to cv_score, train_score or their average. Lower-is-better metrics are sign-flipped before the reliability score is computed. Results are then ranked by reliability_score = performance_score / (1 + instability_score), where instability_score is the sum of pairwise gaps between the available scores.

Public Methods

__init__(): Initialise the combinatorial selector.

set_log_level(): Set the logging level for diagnostics.

rank_features_by_relevance_redundancy(): Optional pre-ranking of features before subset generation.

fit_stage_1(): Evaluate and filter candidate subsets in stage 1.

fit_stage_2(): Refine the stage 1 winners in a second combinatorial pass.

display_best(): DEPRECATED Fit and report the best subset from stage 2.

estimator

The machine learning estimator used to fit the data.

Type:

object

metric

A metric function to evaluate estimator performance. Must accept (y_true, y_pred).

Type:

callable

logic

Determines whether a higher or lower score is considered better.

Type:

{‘greater’, ‘lower’}

task_type

Specifies the type of task.

Type:

{‘classification’, ‘regression’}

cv_splitter

A scikit-learn compatible cross-validation splitter.

Type:

object, optional

cv_indices

Explicit, pre-computed (train_idx, valid_idx) pairs.

Type:

iterable of (array-like, array-like), optional

groups

Group labels propagated to cv_splitter.

Type:

array-like, optional

log_level

Logging threshold used by the wrapper logger.

Type:

int or str

Examples

>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.datasets import make_classification
>>> import pandas as pd
>>> import numpy as np
>>> from mlchem.metrics import get_geometric_S

Standard cross-validation (existing behaviour):

>>> cs = CombinatorialSelection(estimator=LogisticRegression(),
...                              metric=get_geometric_S,
...                              logic='greater')
>>> X, y = make_classification(200, 12, n_informative=6, random_state=1)
>>> train_set = pd.DataFrame(X, columns=np.arange(X.shape[1]))
>>> features = train_set.columns.tolist()
>>> stage1 = cs.fit_stage_1(train_set, y,
...                         features=features,
...                         k=2,
...                         cv_iter=3,
...                         n_jobs=1)
>>> stage2 = cs.fit_stage_2(top_n_subsets=5, cv_iter=3, n_jobs=1)

GroupKFold with scaffold groups:

>>> from sklearn.model_selection import GroupKFold
>>> cs = CombinatorialSelection(estimator=LogisticRegression(),
...                              metric=get_geometric_S,
...                              cv_splitter=GroupKFold(5),
...                              groups=scaffold_ids)

Precomputed scaffold or UMAP-cluster folds:

>>> cs = CombinatorialSelection(estimator=LogisticRegression(),
...                              metric=get_geometric_S,
...                              cv_indices=scaffold_fold_manifest)
__init__(estimator, metric, logic: Literal['lower', 'greater'] = 'greater', task_type: Literal['classification', 'regression'] = 'classification', cv_splitter=None, cv_indices: Iterable | None = None, groups=None, desired_performance_score: Literal['train', 'cv', 'train_cv_average'] = 'train_cv_average', log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20) → None

Initialise the CombinatorialSelection object.

Parameters:

  • estimator (object) – The machine learning estimator used to fit the data.

  • metric (callable) – A metric function to evaluate estimator performance.

  • logic ({'greater', 'lower'}, optional) – Determines whether a higher or lower score is considered better. Default is ‘greater’.

  • task_type ({'classification', 'regression'}, optional) – Specifies the type of task. Default is ‘classification’.

  • cv_splitter (object, optional) – A scikit-learn compatible cross-validation splitter. Takes precedence over the cv_iter passed to fit_stage_1/fit_stage_2.

  • cv_indices (iterable of (array-like, array-like), optional) – Explicit (train_idx, valid_idx) fold pairs. Takes precedence over both cv_splitter and cv_iter.

  • groups (array-like, optional) – Group labels forwarded to cv_splitter. Ignored when cv_indices is provided.

  • desired_performance_score ({'train','cv','train_cv_average'}, optional) – Which score to use as the performance score when calculating reliability. Options are: ‘train’ for the training score, ‘cv’ for the cross-validation score, and ‘train_cv_average’ for the average of the training and cross-validation scores. Default is ‘train_cv_average’.

  • log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int, optional) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output. Default is logging.INFO.

display_best(row: int = 1) → None

Display the best feature subset based on the specified row.

Parameters:

row (int, optional) – Row index of the best feature subset to display. Default is 1.

Return type:

None

Notes

  • Fits the estimator on the selected subset.

  • Displays training, cross-validation, and test scores.

fit_stage_1(train_set: DataFrame, y_train: Iterable, features: list[str] | None = None, k: int = 2, training_threshold: float = 0.25, cv_train_ratio: float = 0.7, cv_iter: int = 5, max_subsets: int | None = None, n_jobs: int = 1, ranking_target: Iterable | None = None, alpha: float = 1.0, beta: float = 0.2, top_ranked_features: int | None = None, relevance_metric: Literal['mutual_info', 'pearson', 'spearman'] = 'mutual_info', redundancy_metric: Literal['pearson', 'spearman'] = 'pearson', ranking_random_state: int = 1) → DataFrame

Perform the first stage of combinatorial feature selection.

Parameters:

  • train_set (pandas.DataFrame) – The training dataset.

  • y_train (iterable) – Target values for the training dataset.

  • features (list of str, optional) – List of features to consider. Default is an empty list.

  • k (int, optional) – Number of features to combine. Default is 2.

  • training_threshold (float, optional) – Minimum training score required to consider a subset. Default is 0.25.

  • cv_train_ratio (float, optional) – Minimum ratio of cross-validation to training score. Default is 0.7.

  • cv_iter (int, optional) – Number of cross-validation iterations. Default is 5.

  • max_subsets (int or None, optional) – Hard cap on the number of generated feature subsets. If None, no hard cap is applied. Default is None.

  • n_jobs (int, optional) – Number of parallel workers used to evaluate candidate feature subsets. Use -1 to use all available CPUs. Default is 1.

  • ranking_target (iterable or None, optional) – Target variable used by relevance-redundancy feature ranking. If None, no pre-ranking is applied unless top_ranked_features is provided (which then raises an error).

  • alpha (float, optional) – Relevance coefficient in ranking score.

  • beta (float, optional) – Redundancy penalty coefficient in ranking score.

  • top_ranked_features (int or None, optional) – Number of top ranked features to keep before combinatorial subset generation. If None and ranking is enabled, all ranked features are kept.

  • relevance_metric ({'mutual_info', 'pearson', 'spearman'}, optional) – Relevance metric used in ranking.

  • redundancy_metric ({'pearson', 'spearman'}, optional) – Redundancy metric used in ranking.

  • ranking_random_state (int, optional) – Random seed used by mutual information estimators.

Returns:

A DataFrame containing the results of the first stage of feature selection.

Return type:

pandas.DataFrame

Notes

Generates all possible feature subsets of size k and evaluates each subset using training and cross-validation scores. Results are filtered using training/CV thresholds and ranked by reliability_score. The score uses the same arithmetic-mean performance term and lower-is-better orientation as SFS. The legacy geometric_mean formulation is no longer used.

fit_stage_2(top_n_subsets: int = 10, cv_iter: int = 5, max_subsets: int | None = None, n_jobs: int = 1) → DataFrame

Perform the second stage of combinatorial feature selection.

Parameters:

  • top_n_subsets (int, optional) – Number of top feature subsets from stage 1 to consider. Default is 10.

  • cv_iter (int, optional) – Number of cross-validation iterations. Default is 5.

  • max_subsets (int or None, optional) – Hard cap on the number of generated feature subsets. If None, no hard cap is applied. Default is None.

  • n_jobs (int, optional) – Number of parallel workers used to evaluate candidate feature subsets. Use -1 to use all available CPUs. Default is 1.

Returns:

A DataFrame containing the results of the second stage of feature selection.

Return type:

pandas.DataFrame

rank_features_by_relevance_redundancy(dataframe: DataFrame, target: Iterable, features: list[str] | None = None, alpha: float = 1.0, beta: float = 0.2, top_features: int | None = None, relevance_metric: Literal['mutual_info', 'pearson', 'spearman'] = 'mutual_info', redundancy_metric: Literal['pearson', 'spearman'] = 'pearson', random_state: int = 42) → DataFrame

Rank features with a global relevance-redundancy criterion.

Higher alpha increases target relevance importance. Higher beta increases redundancy penalty importance.

set_log_level(log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) → None

Set the logging level for wrapper diagnostics.

Parameters:

log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output.

class SequentialForwardSelection

Bases: object

Sequential Forward Feature Selection wrapper.

This class iteratively builds a feature subset by selecting the next feature that maximizes cross-validation performance. It tracks train/CV fold metrics and a reliability score, and can apply a parsimony rule in find_best().

Parameters:

  • estimator (object) – Scikit-learn compatible estimator used for feature selection.

  • estimator_string (str, optional) – Display name for the estimator. If None, inferred from the estimator.

  • metric (callable) – Scoring function with signature metric(y_true, y_pred).

  • max_features (int, optional) – Maximum number of features to select. Default is 25.

  • cv_iter (int, optional) – Number of CV folds when cv_splitter/cv_indices are not provided.

  • cv_splitter (object, optional) – Scikit-learn compatible CV splitter. Takes precedence over cv_iter.

  • groups (array-like, optional) – Group labels passed to group-aware splitters.

  • cv_indices (iterable of (array-like, array-like), optional) – Precomputed (train_idx, valid_idx) fold manifest. Takes precedence over cv_splitter and cv_iter.

  • logic ({'lower', 'greater'}, optional) – Metric optimization direction. Default is 'greater'.

  • task_type ({'classification', 'regression'}, optional) – Task type. Default is 'classification'.

  • desired_performance_score ({'train', 'cv', 'train_cv_average'}, optional) – Performance component used by the reliability score.

  • log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int, optional) – Logging threshold. Default is logging.INFO.

Notes

Reliability combines performance and instability. For each subset:

  • instability_score = |train - cv|

  • For desired_performance_score='train', performance_score = train

  • For desired_performance_score='cv', performance_score = cv

  • For desired_performance_score='train_cv_average', performance_score = (train + cv) / 2

  • For higher-is-better metrics, reliability_score = performance_score / (1 + instability_score)

For lower-is-better metrics, performance is sign-oriented internally so the same maximization rule can be used.

Scientific Rationale

The optional parsimony selector prefers the smallest subset whose paired fold-level degradation relative to the reference subset k* remains within a configurable empirical uncertainty margin.

References

[1] Dietterich (1998), doi:10.1162/089976698300017197 [2] Nadeau & Bengio (2003), doi:10.1023/A:1024068626366 [3] Bengio & Grandvalet (2004), JMLR 5:1089-1105

__init__(estimator, estimator_string: str | None, metric: Callable, max_features: int = 25, cv_iter: int = 5, cv_splitter=None, groups=None, cv_indices: Iterable | None = None, logic: Literal['lower', 'greater'] = 'greater', task_type: Literal['classification', 'regression'] = 'classification', desired_performance_score: Literal['train', 'cv', 'train_cv_average'] = 'train_cv_average', log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20) → None

Initialise a SequentialForwardSelection wrapper.

Parameters:

  • estimator (object) – Scikit-learn compatible estimator.

  • estimator_string (str, optional) – Display name for the estimator.

  • metric (callable) – Scoring function metric(y_true, y_pred).

  • max_features (int, optional) – Maximum number of selected features.

  • cv_iter (int, optional) – Number of CV folds when explicit folds are not provided.

  • cv_splitter (object, optional) – CV splitter used instead of cv_iter.

  • groups (array-like, optional) – Group labels passed to cv_splitter.

  • cv_indices (iterable of (array-like, array-like), optional) – Explicit fold definitions.

  • logic ({'lower', 'greater'}, optional) – Metric optimization direction.

  • task_type ({'classification', 'regression'}, optional) – Task type.

  • desired_performance_score ({'train', 'cv', 'train_cv_average'}, optional) – Performance component used for reliability computation.

  • log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int, optional) – Logging threshold.

find_best(which: int | None = None, parsimony_mode: Literal['none', 'best', 'tolerance', 'standard_error', 'uncertainty'] | None = None, parsimony_tolerance: float = 0.0, parsimony_se_multiplier: float = 1.0) → dict

Return the selected feature subset.

Parameters:

  • which (int, optional) – If provided, return the subset at this 1-based prefix length. If None, automatic selection is used.

  • parsimony_mode ({None, 'none', 'best', 'tolerance', 'standard_error', 'uncertainty'}, optional) – Parsimony strategy used only when which is None. None (or 'none') disables parsimony and returns the reliability-optimal subset k*.

  • parsimony_tolerance (float, optional) – Absolute tolerance used by parsimony modes that require it.

  • parsimony_se_multiplier (float, optional) – Standard-error multiplier used by parsimony modes that require it.

Returns:

Selection payload including best_index and features. Automatic non-parsimony mode also returns reliability_score and best_score. Parsimony modes also return parsimony_mode and parsimony_diagnostics.

Return type:

dict

fit(train_set: DataFrame, y_train: Iterable, test_set: DataFrame | None, y_test: Iterable, n_jobs: int = 1) → None

Fit the Sequential Forward Selection model.

Parameters:

  • train_set (pandas.DataFrame) – Training dataset.

  • y_train (iterable) – Target values for the training set.

  • test_set (pandas.DataFrame, optional) – Optional test dataset. If provided, unseen-set scores are stored in self.unseen_scores.

  • y_test (iterable) – Target values for the test set.

  • n_jobs (int, optional) – Number of parallel workers used to evaluate candidate features at each SFS cycle. Use -1 to use all available CPUs. Default is 1.

Return type:

None

plot(best_feature: int | Literal['auto'] | None = 'auto', figsize: tuple[int, int] = (10, 6), colours: list[str] = ['steelblue', 'orange', 'green'], alphas: list[float] = [0.2, 0.2, 0.2], title: str | None = None, title_size: int = 20, xlabel: str = '# of features', ylabel: str = 'Score', fontsize: int = 14, legendsize: int = 13, save: bool = False) → None

Plot the performance of the Sequential Forward Selection process. The interval of 1 standard error is shown as a shaded region around the cv_score curve.

Parameters:

  • best_feature (int, 'auto', or None, optional) – Index of the best feature subset to highlight. If ‘auto’ or None, it is determined automatically using reliability-score selection. Default is ‘auto’.

  • figsize (tuple of int, optional) – Size of the plot. Default is (10, 6).

  • colours (list of str, optional) – Colours for training, validation, and reliability scores. Default is [‘steelblue’, ‘orange’, ‘green’].

  • alphas (list of float, optional) – Alpha values, with shape (3,), for the shaded regions representing the ±1 standard error. Setting any of these values to zero will make the corresponding shaded region fully transparent.

  • title (str, optional) – Title of the plot.

  • title_size (int, optional) – Font size of the title. Default is 20.

  • xlabel (str, optional) – Label for the x-axis. Default is ‘# of features’.

  • ylabel (str, optional) – Label for the y-axis. Default is ‘Score’.

  • fontsize (int, optional) – Font size for axis labels. Default is 14.

  • legendsize (int, optional) – Font size for the legend. Default is 13.

  • save (bool, optional) – Whether to save the plot. Default is False.

Return type:

None

Notes

The automatic algorithm for determining the best feature subset is the same as described in find_best() and uses the reliability scores already computed during fit(). Reliability is derived from train/CV scores (according to desired_performance_score) and train-vs-CV instability; test scores are not used in this selector. The highlighted subset is the one returned by find_best().

set_log_level(log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) → None

Set the logging level for wrapper diagnostics.

Parameters:

log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output.