Breast Cancer Preprocessing Pipeline

What this project is

Breast Cancer Preprocessing Pipeline is a scikit-learn preprocessing system designed to prepare medical tabular data for machine learning experiments. The project combines custom transformers, ColumnTransformer, FeatureUnion, feature selection, and PCA into a reusable workflow.

Technical context

A breast cancer research dataset contains 569 observations and 30 input variables, but it cannot be passed directly to a machine learning model. Every feature contains missing values, numerical distributions show different levels of skewness, and the area error column is categorical.

The design challenge was to transform heterogeneous data into one consistent matrix while combining different strategies in the same workflow: conditional subset processing, supervised feature selection, and dimensionality reduction.

Objective

The objective was to build a reusable preprocessing system with the scikit-learn API:

• impute missing values with strategies appropriate to each variable type;
• separate symmetric and skewed numerical features;
• correct distributions whose skewness exceeds 0.75;
• encode the categorical variable;
• select the five most informative features;
• reduce dimensionality while retaining 80% of variance;
• combine all three results into one output.

The project focuses on data preparation and does not train a diagnostic classifier. Its output is intended as structured input for downstream experiments.

Architecture

The source DataFrame contains 31 columns: 30 features and one target variable. After separating X and y, three pipelines run in parallel and are combined through FeatureUnion.

1. Pipeline 1 applies differentiated transformations to the target=1 subset;
2. Pipeline 2 prepares the full dataset and selects five features with SelectKBest;
3. Pipeline 3 prepares numerical variables and applies PCA with an 80% variance threshold;
4. FeatureUnion concatenates the three outputs horizontally;
5. the final matrix has shape (569, 43).

Pipeline 1: conditional transformation

The first branch studies the subset identified by target=1. Skewness is calculated on numerical variables, which are divided into two groups:

• symmetric features: mean imputation and StandardScaler;
• skewed features: median imputation, logarithmic correction, and standardization;
• categorical feature: most-frequent imputation and OneHotEncoder.

The custom PipelineWithRowFilter transformer runs this pipeline only on rows that satisfy the condition. It fills all other rows with zeros, preserving the total number of observations. This branch produces 30 features.

Custom skewness transformer

SkewnessCorrector implements BaseEstimator and TransformerMixin. During fit, it identifies columns whose absolute skewness exceeds 0.75; during transform, it applies log1p only to those columns.

Compatibility with the scikit-learn API allows the transformation to be embedded in both Pipeline and ColumnTransformer, avoiding manual steps outside the workflow.

Pipeline 2: feature selection

The second branch processes all 569 observations:

• numerical variables are mean-imputed and discretized into three quantile bins;
• area error is imputed and ordinally encoded according to categories A, B, and C;
• SelectKBest with f_classif retains five features.

The selected features are mean perimeter, worst perimeter, worst area, worst concavity, and worst concave points.

Pipeline 3: dimensionality reduction

The third branch uses numerical features only. The sequence applies mean imputation, skewness correction, standardization, PCA, and final normalization through MinMaxScaler.

PCA(n_components=0.80) automatically determines the number of components required to retain at least 80% of variance. On this dataset, the resulting representation contains eight components.

Composition through FeatureUnion

After independent testing, all three pipelines are registered in one FeatureUnion. The final transformation concatenates:

• 30 features from Pipeline 1;
• 5 features from Pipeline 2;
• 8 components from Pipeline 3.

The output is a NumPy array with shape (569, 43). The architecture demonstrates how different preprocessing strategies can be isolated, tested separately, and composed into a single scikit-learn object.

Results

• 569 observations preserved throughout the workflow;
• missing values handled according to feature type and distribution;
• two custom transformers compatible with scikit-learn;
• five features retained through supervised selection;
• eight PCA components representing at least 80% of variance;
• 43 total features produced by three independent branches.

Future evolution

The pipeline fulfills its educational objective: applying different preprocessing strategies and combining them into a structured matrix for downstream analysis. Training a predictive model or producing clinical metrics was not part of the required scope.

To turn the notebook into a reusable component for a real ML workflow, the main evolutions would be:

• adapt PipelineWithRowFilter so that transform does not depend on target labels, preventing target leakage and allowing unseen data to be processed;
• document the mapping between numerical target values and source classes explicitly;
• preserve interpretable names for all 43 final features;
• add automated tests for dimensions, missing values, and reproducibility;
• serialize the fitted pipeline as a versioned artifact.

A classifier, together with its metrics and validation procedures, would be a separate downstream phase rather than a missing requirement of this preprocessing project.

Technical FAQ

Why use three pipelines instead of one?

The branches address different goals: conditional transformation, supervised selection, and dimensionality reduction. Keeping them separate makes each strategy understandable and allows their outputs to be combined without mixing responsibilities.

Why use FeatureUnion?

FeatureUnion runs multiple transformers on the same input and concatenates their outputs by column. It is therefore suitable for building a final representation from features produced through different strategies.

Does the pipeline perform a diagnosis?

No. The project prepares a dataset for educational purposes. It includes no classifier, diagnostic metrics, or validation for clinical use.

What is the main limitation of the row filter?

The wrapper depends on the target used during training and reuses it to decide which rows to transform. This is sufficient to demonstrate the notebook logic, but it must be changed before using the pipeline with unseen data, train/test splits, or cross-validation.