mlchem.ml.feature_selection package

The wrapper selectors support optional runtime diagnostics through the log_level constructor argument and the set_log_level(…) runtime toggle.

Submodules

mlchem.ml.feature_selection.filters module

collinearity_filter(df: DataFrame, threshold: float, target_variable: str = None, method: Literal['pearson', 'kendall', 'spearman'] = 'pearson', numeric_only: bool = False) → DataFrame

Filter features based on collinearity threshold.

Returns a subset of DataFrame columns whose squared correlation (R²) values are below the specified threshold. If a target variable is provided, the function retains the feature with the higher correlation to the target when multiple features are collinear.

Parameters:

  • df (pandas.DataFrame) – The input dataset.

  • threshold (float) – The maximum allowed squared correlation between features.

  • target_variable (str, optional) – The name of the target variable. If provided, it is used to resolve collinearity conflicts.

  • method ({'pearson', 'kendall', 'spearman'}, optional) – The correlation method to use. Default is ‘pearson’.

  • numeric_only (bool, optional) – Whether to include only numeric columns. Default is False.

Returns:

A DataFrame containing the filtered columns.

Return type:

pandas.DataFrame

diversity_filter(df: DataFrame, threshold: float, target_variable: str = None) → DataFrame

Filter features based on diversity ratio using Shannon entropy.

Calculates the diversity ratio of each feature by comparing its Shannon entropy to that of an ideal uniform distribution. Retains features with diversity ratios above the specified threshold.

Parameters:

  • df (pandas.DataFrame) – The input dataset.

  • threshold (float) – The minimum diversity ratio required to retain a feature.

  • target_variable (str, optional) – The name of the target variable to retain regardless of its diversity score.

Returns:

A DataFrame containing the filtered columns with diversity higher than the threshold.

Return type:

pandas.DataFrame

mlchem.ml.feature_selection.wrappers module

class CombinatorialSelection

Bases: object

Combinatorial feature selection using a given estimator and metric.

This class performs a two-stage combinatorial feature selection process to identify optimal feature subsets based on reliability score. For each retained subset, performance_score is the geometric mean of training, cross-validation, and test scores for higher-is-better metrics. For lower-is-better metrics, the geometric mean is inverted. The final reliability_score is performance_score / (1 + instability_score), where instability_score = |train-cv| + |train-test| + |cv-test|.

estimator

The machine learning estimator used to fit the data.

Type:

object

metric

A metric function to evaluate estimator performance. Must accept (y_true, y_pred).

Type:

callable

logic

Determines whether a higher or lower score is considered better.

Type:

{‘greater’, ‘lower’}

task_type

Specifies the type of task.

Type:

{‘classification’, ‘regression’}

Examples

>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.datasets import make_classification
>>> from mlchem.metrics import get_geometric_S

>>> cs = CombinatorialSelection(estimator=LogisticRegression(),
...                              metric=get_geometric_S,
...                              logic='greater')

>>> X, y = make_classification(500, 10, n_informative=4)
>>> X_train, y_train = X[:350], y[:350]
>>> X_test, y_test = X[350:], y[350:]

>>> train_set = pd.DataFrame(X_train, columns=np.arange(X_train.shape[1]))
>>> test_set = pd.DataFrame(X_test, columns=np.arange(X_test.shape[1]))

>>> results_stage_1 = cs.fit_stage_1(train_set, y_train, test_set, y_test,
...                                  train_set.columns, training_threshold=0.7)
>>> results_stage_2 = cs.fit_stage_2(top_n_subsets=10, cv_iter=5)
__init__(estimator, metric, logic: Literal['lower', 'greater'] = 'greater', task_type: Literal['classification', 'regression'] = 'classification', log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20) → None

Initialise the CombinatorialSelection object.

Parameters:

  • estimator (object) – The machine learning estimator used to fit the data.

  • metric (callable) – A metric function to evaluate estimator performance.

  • logic ({'greater', 'lower'}, optional) – Determines whether a higher or lower score is considered better. Default is ‘greater’.

  • task_type ({'classification', 'regression'}, optional) – Specifies the type of task. Default is ‘classification’.

  • log_level ({{'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'}} or int, optional) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output. Default is logging.INFO.

display_best(row: int = 1) → None

Display the best feature subset based on the specified row.

Parameters:

row (int, optional) – Row index of the best feature subset to display. Default is 1.

Return type:

None

Notes

  • Fits the estimator on the selected subset.

  • Displays training, cross-validation, and test scores.

fit_stage_1(train_set: DataFrame, y_train: Iterable, test_set: DataFrame, y_test: Iterable, features: list[str] | None = None, k: int = 2, training_threshold: float = 0.25, cv_train_ratio: float = 0.7, cv_iter: int = 5, max_subsets: int | None = None, n_jobs: int = 1, ranking_target: Iterable | None = None, alpha: float = 1.0, beta: float = 0.2, top_ranked_features: int | None = None, relevance_metric: Literal['mutual_info', 'pearson', 'spearman'] = 'mutual_info', redundancy_metric: Literal['pearson', 'spearman'] = 'pearson', ranking_random_state: int = 1) → DataFrame

Perform the first stage of combinatorial feature selection.

Parameters:

  • train_set (pandas.DataFrame) – The training dataset.

  • y_train (iterable) – Target values for the training dataset.

  • test_set (pandas.DataFrame) – The testing dataset.

  • y_test (iterable) – Target values for the testing dataset.

  • features (list of str, optional) – List of features to consider. Default is an empty list.

  • k (int, optional) – Number of features to combine. Default is 2.

  • training_threshold (float, optional) – Minimum training score required to consider a subset. Default is 0.25.

  • cv_train_ratio (float, optional) – Minimum ratio of cross-validation to training score. Default is 0.7.

  • cv_iter (int, optional) – Number of cross-validation iterations. Default is 5.

  • max_subsets (int or None, optional) – Hard cap on the number of generated feature subsets. If None, no hard cap is applied. Default is None.

  • n_jobs (int, optional) – Number of parallel workers used to evaluate candidate feature subsets. Use -1 to use all available CPUs. Default is 1.

  • ranking_target (iterable or None, optional) – Target variable used by relevance-redundancy feature ranking. If None, no pre-ranking is applied unless top_ranked_features is provided (which then raises an error).

  • alpha (float, optional) – Relevance coefficient in ranking score.

  • beta (float, optional) – Redundancy penalty coefficient in ranking score.

  • top_ranked_features (int or None, optional) – Number of top ranked features to keep before combinatorial subset generation. If None and ranking is enabled, all ranked features are kept.

  • relevance_metric ({'mutual_info', 'pearson', 'spearman'}, optional) – Relevance metric used in ranking.

  • redundancy_metric ({'pearson', 'spearman'}, optional) – Redundancy metric used in ranking.

  • ranking_random_state (int, optional) – Random seed used by mutual information estimators.

Returns:

A DataFrame containing the results of the first stage of feature selection.

Return type:

pandas.DataFrame

Notes

Generates all possible feature subsets of size k and evaluates each subset using training, cross-validation, and test scores. Results are filtered using training/CV thresholds and ranked by reliability_score. The legacy geometric_mean column is retained for compatibility, but will be removed in a future version.

fit_stage_2(top_n_subsets: int = 10, cv_iter: int = 5, max_subsets: int | None = None, n_jobs: int = 1) → DataFrame

Perform the second stage of combinatorial feature selection.

Parameters:

  • top_n_subsets (int, optional) – Number of top feature subsets from stage 1 to consider. Default is 10.

  • cv_iter (int, optional) – Number of cross-validation iterations. Default is 5.

  • max_subsets (int or None, optional) – Hard cap on the number of generated feature subsets. If None, no hard cap is applied. Default is None.

  • n_jobs (int, optional) – Number of parallel workers used to evaluate candidate feature subsets. Use -1 to use all available CPUs. Default is 1.

Returns:

A DataFrame containing the results of the second stage of feature selection.

Return type:

pandas.DataFrame

Notes

Identifies the most recurrent features from the top stage-1 subsets, generates new combinations, and evaluates them. Results are filtered using training/CV thresholds and ranked by reliability_score. The legacy geometric_mean column is retained for compatibility, but will be removed in a future version.

rank_features_by_relevance_redundancy(dataframe: DataFrame, target: Iterable, features: list[str] | None = None, alpha: float = 1.0, beta: float = 0.2, top_features: int | None = None, relevance_metric: Literal['mutual_info', 'pearson', 'spearman'] = 'mutual_info', redundancy_metric: Literal['pearson', 'spearman'] = 'pearson', random_state: int = 42) → DataFrame

Rank features with a global relevance-redundancy criterion.

Higher alpha increases target relevance importance. Higher beta increases redundancy penalty importance.

set_log_level(log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) → None

Set the logging level for wrapper diagnostics.

Parameters:

log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output.

class SequentialForwardSelection

Bases: object

Sequential Forward Feature Selection wrapper.

This class performs Sequential Forward Feature Selection by iteratively adding features that yield the highest gain in cross-validation score.

estimator

The scikit-learn estimator used for feature selection.

Type:

object

estimator_string

A string representation of the estimator. If None, it is inferred from the estimator.

Type:

str, optional

metric

A function to evaluate model performance.

Type:

callable

max_features

Maximum number of features to select. Default is 25.

Type:

int, optional

cv_iter

Number of cross-validation iterations. Default is 5.

Type:

int, optional

logic

Whether to minimize or maximize the cross-validation score. Default is ‘greater’.

Type:

{‘lower’, ‘greater’}, optional

task_type

Type of task. Default is ‘classification’.

Type:

{‘classification’, ‘regression’}, optional

log_level

Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output. Default is logging.INFO.

Notes

Automatic best-subset selection uses a reliability score. For each selected prefix, performance_score = (train * cv * test) ** (1/3), instability_score = |train-cv| + |train-test| + |cv-test|, and for higher-is-better metrics reliability_score = performance_score / (1 + instability_score). For lower-is-better metrics, the geometric mean is inverted first so the same reliability score can be maximised.

Type:

{‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’} or int, optional

Examples

>>> import pandas as pd
>>> import numpy as np
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.datasets import make_classification
>>> from mlchem.metrics import get_geometric_S

>>> sfs = SequentialForwardSelection(estimator=LogisticRegression(),
...                                  metric=get_geometric_S,
...                                  max_features=5,
...                                  cv_iter=3,
...                                  logic='greater')

>>> X, y = make_classification(300, 10, n_informative=5)
>>> train_size = 0.8
>>> train_samples = int(train_size * len(X))

>>> X_train, y_train = X[:train_samples], y[:train_samples]
>>> X_test, y_test = X[train_samples:], y[train_samples:]

>>> train_set = pd.DataFrame(X_train, columns=np.arange(X_train.shape[1]))
>>> test_set = pd.DataFrame(X_test, columns=np.arange(X_test.shape[1]))

>>> sfs.fit(train_set, y_train, test_set, y_test)
>>> sfs.plot(best_feature='None')
__init__(estimator, estimator_string: str | None, metric: Callable, max_features: int = 25, cv_iter: int = 5, logic: Literal['lower', 'greater'] = 'greater', task_type: Literal['classification', 'regression'] = 'classification', log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'] = 20) → None

Initialise the SequentialForwardSelection object.

Parameters:

  • estimator (object) – The scikit-learn estimator used for feature selection.

  • estimator_string (str, optional) – A string representation of the estimator. If None, it is inferred from the estimator.

  • metric (callable) – A function to evaluate model performance.

  • max_features (int, optional) – Maximum number of features to select. Default is 25.

  • cv_iter (int, optional) – Number of cross-validation iterations. Default is 5.

  • logic ({'lower', 'greater'}, optional) – Whether to minimise or maximise the cross-validation score. Default is ‘greater’.

  • task_type ({'classification', 'regression'}, optional) – Type of task. Default is ‘classification’.

  • log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int, optional) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output. Default is logging.INFO.

find_best(which: int | None = None) → dict

Find the best feature subset based on reliability.

Parameters:

which (int, optional) – If specified, returns the feature subset at the given index. If None, the best subset is determined automatically using the reliability score.

Returns:

Dictionary containing the selected subset and reliability scores.

best_indexint

Number of selected features in the winning prefix.

featureslist

Selected feature names.

performance_scorefloat

Geometric performance contribution for the winning prefix.

instability_scorefloat

Sum of train/CV/test score gaps for the winning prefix.

reliability_scorefloat

Reliability score used for automatic selection.

best_scorefloat

Alias of reliability_score retained for backwards compatibility.

Return type:

dict

Notes

For each feature subset, the automatic algorithm computes:

reliability_score = performance_score / (1 + instability_score)

where:

instability_score = |train-cv| + |train-test| + |cv-test|

and, for higher-is-better metrics:

performance_score = (train_score * cv_score * test_score) ** (1/3)

For lower-is-better metrics, such as RMSE, lower performance scores are better, so the geometric mean is inverted before the same reliability calculation is applied:

performance_score = 1 / ((train_score * cv_score * test_score) ** (1/3))

reliability_score = performance_score / (1 + instability_score)

The subset with the highest reliability score is selected. The test score is intentionally included in the calculation.

fit(train_set: DataFrame, y_train: Iterable, test_set: DataFrame, y_test: Iterable, n_jobs: int = 1) → None

Fit the Sequential Forward Selection model.

Parameters:

  • train_set (pandas.DataFrame) – Training dataset.

  • y_train (iterable) – Target values for the training set.

  • test_set (pandas.DataFrame) – Test dataset.

  • y_test (iterable) – Target values for the test set.

  • n_jobs (int, optional) – Number of parallel workers used to evaluate candidate features at each SFS cycle. Use -1 to use all available CPUs. Default is 1.

Return type:

None

plot(best_feature: int | Literal['auto'] | None = 'auto', figsize: tuple[int, int] = (10, 6), colours: list[str] = ['steelblue', 'orange', 'green'], title: str | None = None, title_size: int = 20, xlabel: str = '# of features', ylabel: str = 'Score', fontsize: int = 14, legendsize: int = 13, save: bool = False) → None

Plot the performance of the Sequential Forward Selection process.

Parameters:

  • best_feature (int, 'auto', or None, optional) – Index of the best feature subset to highlight. If ‘auto’ or None, it is determined automatically using reliability-score selection. Default is ‘auto’.

  • figsize (tuple of int, optional) – Size of the plot. Default is (10, 6).

  • colours (list of str, optional) – Colours for training, validation, and test scores. Default is [‘steelblue’, ‘orange’, ‘green’].

  • title (str, optional) – Title of the plot.

  • title_size (int, optional) – Font size of the title. Default is 20.

  • xlabel (str, optional) – Label for the x-axis. Default is ‘# of features’.

  • ylabel (str, optional) – Label for the y-axis. Default is ‘Score’.

  • fontsize (int, optional) – Font size for axis labels. Default is 14.

  • legendsize (int, optional) – Font size for the legend. Default is 13.

  • save (bool, optional) – Whether to save the plot. Default is False.

Return type:

None

Notes

The automatic algorithm for determining the best feature subset is the same as described in find_best: performance_score = (train * cv * test) ** (1/3), instability_score = |train-cv| + |train-test| + |cv-test|, and for higher-is-better metrics reliability_score = performance_score / (1 + instability_score). For lower-is-better metrics, the geometric mean is inverted before applying the same reliability formula. The subset with the highest reliability score is highlighted.

set_log_level(log_level: int | str | Literal['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) → None

Set the logging level for wrapper diagnostics.

Parameters:

log_level ({'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'} or int) – Logging level threshold. Use ‘DEBUG’ for detailed diagnostics, ‘INFO’ for standard output, ‘WARNING’ to suppress most output.