NLP Low-Resource Classification: Comparative Analysis

1. Project Overview

This repository provides the comparative analysis of three low-resource text classification experiments:

• AG News — 4-class news topic classification
• DBpedia — 14-class ontology classification
• IMDb 50K — binary sentiment classification

The three projects follow the same progressive experimental framework, from classical NLP representations to sentence and word embeddings.

This repository brings their results together to analyze how the different approaches behave across datasets and training set sizes.

---

2. Analysis

The main analysis is contained in:

analysis.ipynb

The notebook:

• loads the results produced by the three dataset-specific projects,
• combines the experimental results,
• compares performance across datasets,
• analyzes the effect of training set size,
• compares classical NLP and embedding-based approaches,
• produces the final comparative visualizations.

The analysis focuses primarily on Macro F1, while also considering Accuracy and computational cost.

---

3. Embedding Comparison

The repository also contains:

compare_embeddings.py

This script provides a focused comparison of the embedding experiments:

• V6.1 — Sentence Embeddings + LinearSVC
• V6.2 — Sentence Embeddings + L2 normalization
• V6.3 — Sentence Embeddings + Nearest Centroid
• V7.1 — GloVe 100d + Mean Pooling
• V7.2 — GloVe 100d + TF-IDF Weighted Mean

It can be run on the results directory of each dataset:

python compare_embeddings.py <dataset_directory>

The script:

• loads the V6 and V7 embedding results,
• displays the combined results,
• generates an embedding performance comparison plot,
• saves the combined results as CSV and HTML files.

---

4. Experimental Framework

The three datasets follow the same general progression:

Baseline
   |
V1 - Stop Words
   |
V2 - N-grams
   |
V3 - Stemming
   |
V4 - Lemmatization
   |
V5 - Hyperparameter Optimization
   |
V6 - Sentence Embeddings
   |
V7 - Word Embeddings + Pooling

Training subsets are evaluated at different sizes to study performance under low-resource conditions.

---

5. Repository Structure

NLP-Low-Resource-Analysis/

├── analysis.ipynb
├── compare_embeddings.py
└── README.md

The result CSV files used by the analysis are generated by the three dataset-specific repositories.

---

6. Related Projects

The experiments analyzed in this repository are developed separately for:

• AG News (See also: https://github.com/BeaData/NLP-Low-Resource-experiments-AGNews/)
• DBpedia (See also: https://github.com/BeaData/NLP-Low-Resource-experiments-DBpedia)
• IMDb 50K (See also: https://github.com/BeaData/NLP-Low-Resource-experiments-IMDb50k)

Each dataset repository contains its own data preparation, experimental scripts, result files and dataset-specific visualizations.

---

7. Environment

Recommended Python environment:

Python >= 3.12

Main dependencies include:

numpy
pandas
matplotlib
scikit-learn

Installation:

uv sync

---

Acknowledgements

This project benefited from discussions and technical assistance provided by ChatGPT (OpenAI) for code organization and documentation structure.