← Back to projects

Wikipedia Article Classifier

2026-04-03 ยท Data Engineering

What this project is

Wikipedia Article Classifier is a PySpark NLP pipeline for preparing, vectorizing, and classifying Wikipedia articles into thematic categories. The project highlights data quality, Parquet persistence, TF-IDF feature engineering, a Naive Bayes baseline, and qualitative label analysis.

Technical context

A Wikipedia dataset containing more than 153 thousand articles includes title, category, summary, and document text, but it is not immediately ready for reliable classifier training. Some rows have missing text, some categories require qualitative inspection, and the textual content must be converted into numerical features before Spark MLlib can use it.

The challenge was not just building an NLP model. The most interesting part was designing a data pipeline able to load a large CSV in PySpark, check data quality, create a reproducible text representation, and prepare readable evaluation across 15 thematic classes.

Objective

The objective was to build an end-to-end NLP pipeline in PySpark to classify Wikipedia articles into thematic categories:

  • load the CSV from Google Drive in Google Colab;
  • run EDA with Spark SQL on schema, missing values, and category distribution;
  • remove unusable records with missing summary and documents;
  • persist the clean dataset in Parquet to avoid repeated processing;
  • combine summary and documents into a single text field;
  • transform text with Tokenizer, StopWordsRemover, HashingTF, and IDF;
  • train a Multinomial Naive Bayes classifier;
  • prepare aggregate metrics, per-class metrics, confusion matrix, and cross-validation.

The project therefore demonstrates a full workflow from raw data to model, with attention to both operational scalability and the readability of insights produced during analysis.

Architecture

The workflow is organized into eight main steps:

  1. wikipedia.csv is read from Google Drive with PySpark using header parsing, schema inference, multiline handling, quote, and escape options;
  2. the raw DataFrame is registered as the temporary view wikipedia_data for Spark SQL queries;
  3. the dataset_summary function checks schema, row count, columns, and null values;
  4. records with missing summary and documents are removed;
  5. the clean dataset is written to wikipedia_clean.parquet and reloaded when already available;
  6. summary and documents are concatenated into the text field;
  7. the Spark ML pipeline converts text into TF-IDF features and indexes the category;
  8. Naive Bayes produces predictions that are evaluated with multiclass metrics and category-level analysis.

This structure separates data preparation from training, reducing iteration cost when the model or evaluation layer changes.

Reasoning Map

The project was built as a sequence of technical decisions, not as a simple classification notebook. The underlying logic is to move data from raw CSV to an evaluable output while making explicit where quality, format, and model choices affect the result.

  1. Large semi-structured text dataset: the CSV contains 153,232 articles and 15 categories. PySpark frames the problem as a data pipeline through DataFrames, Spark SQL, and Spark MLlib instead of a local Pandas-only solution.
  2. Minimum quality before modeling: 928 records have missing summary and documents. Removing them avoids training the classifier on rows without useful textual information.
  3. Intermediate persistence: the clean dataset is saved as Parquet, keeping data preparation separate from training and preventing later iterations from repeating deterministic steps.
  4. Qualitative label checks: word clouds help identify 20 off-topic articles in the finance category. This shows that data quality also includes semantic consistency, not only nulls and duplicates.
  5. Scalable feature engineering: Tokenizer, StopWordsRemover, HashingTF, and IDF transform text into a 20,000-dimensional sparse representation without manually building a complete vocabulary.
  6. Interpretable baseline: Multinomial Naive Bayes fits TF-IDF features well and makes it possible to validate the full pipeline before introducing heavier models.
  7. Responsible evaluation: accuracy, F1, precision, recall, per-class metrics, confusion matrix, and cross-validation go beyond a single aggregate metric.

The core message is that the project demonstrates data-flow design: make the data trustworthy first, transform it next, and evaluate the model without confusing a working pipeline with final verified metrics.

Dataset and cleaning

The initial load contains 153,232 articles. Missing-value analysis identifies 928 rows where both summary and documents are null, while title and category are populated. These records do not provide usable text for the classifier and are removed.

After this cleaning step, the dataset contains 152,304 records. The notebook uses a dedicated load_or_create_clean_dataset function to write the result as Parquet on Google Drive and reuse it in later executions. This makes the workflow more efficient and closer to a real data pipeline than a purely in-memory analysis.

Qualitative data quality

Beyond numerical checks, the project uses word clouds to inspect category content. This step identified a problem in the finance category: 20 articles related to He-Man and the Masters of the Universe had been labeled as finance, likely because of ambiguous lexical overlap.

These records are filtered through a case-insensitive pattern matching terms such as he-man, masters of the universe, skeletor, grayskull, cartoon, toyline, and filmation. This highlights an important point in text-oriented Data Engineering: dataset quality is not limited to nulls and duplicates, but also requires semantic checks when labels may be noisy.

NLP feature engineering

The text field is transformed with a Spark ML pipeline made of:

  • Tokenizer to split text into words;
  • StopWordsRemover to remove frequent low-information words;
  • HashingTF with numFeatures=20000 to obtain a fixed-size sparse representation;
  • IDF with minDocFreq=5 to penalize terms that are too common;
  • StringIndexer to convert textual categories into numerical labels.

The HashingTF and IDF combination makes it possible to process a large text corpus without manually building a complete vocabulary, while keeping the representation suitable for multiclass classification.

Model and validation

The selected model is NaiveBayes, a strong baseline for text classification with frequency-based features. The dataset is split with randomSplit([0.8, 0.2], seed=42), keeping training and test data separate.

Evaluation is prepared through MulticlassClassificationEvaluator to compute weighted accuracy, F1, precision, and recall. The notebook also includes manual per-category precision, recall, and F1 calculations, plus a row-normalized confusion matrix.

To test baseline robustness, the project also configures a 3-fold CrossValidator with three Naive Bayes smoothing values: 0.5, 1.0, and 2.0. This structure makes it possible to compare configurations while keeping evaluation consistent and easy to document.

Category analysis

The project checks the distribution of 15 categories and observes a fairly balanced dataset. Categories are around roughly 10 thousand articles each, with politics as the most represented category and finance as the least represented after cleaning.

Word clouds are generated one category at a time to avoid Colab timeouts and keep inspection readable. This makes EDA operational: it is not just about producing charts, but about finding anomalies, thematic bias, and possible labeling issues.

Technical choices

Why PySpark?

PySpark allows the dataset to be handled as a distributed workflow even in Colab, using DataFrames, Spark SQL, and Spark MLlib in the same notebook. This makes the project closer to a Data Engineering workflow than a Pandas-only solution.

Why persist to Parquet?

The initial cleaning step is separated from training and saved as Parquet. Later executions can restart from the already validated dataset, reducing load time and avoiding repetition of deterministic steps.

Why Naive Bayes?

Naive Bayes is an effective baseline for text represented through frequency or TF-IDF features. The choice keeps the focus on the pipeline, data quality, and multiclass evaluation before introducing heavier models.

Why use word clouds?

Word clouds are not used as a model metric, but as a diagnostic tool. In this project, they helped identify noisy labels in the finance category, showing the value of qualitative checks during data preparation.

Results

  • 153,232 articles loaded from the source CSV;
  • 928 records removed because both summary and documents were missing;
  • 152,304 records retained after missing-value cleaning;
  • 15 thematic categories analyzed with a fairly balanced distribution;
  • 20 mislabeled articles identified in the finance category through qualitative analysis;
  • 20,000 TF-IDF dimensions generated through HashingTF and IDF;
  • complete Spark ML pipeline with tokenization, stopword removal, feature engineering, label indexing, and Naive Bayes;
  • evaluation prepared with aggregate metrics, per-class metrics, confusion matrix, and 3-fold cross-validation.

Future evolution

The most natural next steps focus on moving from an exploratory notebook to a more industrialized NLP pipeline while keeping the same Spark logic and qualitative checks.

  • publish a consolidated version of the final metrics in the README and project page;
  • version confusion matrix and per-class tables as evaluation artifacts;
  • document a reproducible procedure for loading the source dataset;
  • automate count checks after cleaning and label validation;
  • compare the Naive Bayes baseline with Logistic Regression and Linear SVM on the same TF-IDF pipeline;
  • serialize the Spark ML model for documented inference scenarios;
  • turn the notebook into a parameterized script for scheduled executions.

Technical FAQ

What metrics does the pipeline produce?

The pipeline prepares accuracy, F1, precision, recall, per-class metrics, confusion matrix, and cross-validation. The page focuses on the evaluation structure because it shows how the project can document model performance in a readable and reproducible way.

Why remove records with null summary and documents?

Because the classifier uses textual content as input. When both fields are absent, the row keeps a category but contains no useful information for learning linguistic patterns.

How is the source dataset handled?

The workflow uses wikipedia.csv as an external source loaded from Google Drive, which fits the size of the dataset. The notebook documents the entry point and then persists the cleaned result as Parquet to make later runs more efficient.

What role do word clouds play?

They act as qualitative category checks. In the finance category, they helped identify 20 off-topic articles that were then removed before training.

Leggi in italiano