README.md

Русская версия | English

Student Feedback Classification for VCIOM

A reproducible Russian NLP pipeline that classifies student social-media posts into four research categories for large-scale opinion analysis.

Built for the VCIOM “AI & Digital Technologies in Sociological Research” hackathon by team aCUtone!.

Results at a glance

Result	Value
Mean cross-validation accuracy	84.63% ± 5.62 pp
Mean macro-F1	0.844 ± 0.057
Evaluation protocol	10× repeated stratified 5-fold CV
Labeled data	160 posts, 40 per class
Posts classified	1,408
Batched inference	≈4,624 posts/s on Apple M1 CPU

The final model combines character-level TF-IDF features with a linear SVM. Keeping the vectorizer inside the scikit-learn pipeline prevents vocabulary leakage between cross-validation folds.

Problem

Large-scale sociological research relies on thousands of open-ended comments from university communities. Manual routing is slow and inconsistent, making it difficult to identify recurring academic, financial, living-condition, and loyalty signals.

The system assigns each post to one of four categories:

ID	Category	Examples of covered topics
0	Academic & extracurricular engagement	Teaching, exams, deadlines, clubs, volunteering, sports
1	Social & living conditions	Dormitories, food, healthcare, campus infrastructure
2	Financial conditions	Scholarships, tuition, grants, work, financial aid
3	University loyalty	Reputation, trust, pride, administration, recommendation intent

Solution

The repository contains a complete workflow rather than a notebook-only prototype:

1. validate labeled and prediction datasets;
2. build character TF-IDF features from Russian text;
3. evaluate the full pipeline under repeated stratified cross-validation;
4. train the final linear SVM on all labeled posts;
5. classify the 1,408-post research corpus;
6. export category IDs, readable labels, metrics, and diagnostic reports.

Character n-grams capture Russian morphology, misspellings, word fragments, and informal social-media language without requiring a large neural model. On this small balanced dataset they outperform the word-level Logistic Regression baseline by about 14 percentage points.

Model comparison

Model	Mean accuracy	Mean macro-F1
Most-frequent dummy	25.00%	0.100
Word TF-IDF + Logistic Regression	70.69%	0.701
Character TF-IDF + Linear SVM	84.63%	0.844

All reported model scores use the same 50 repeated stratified evaluations. The committed metrics report contains the full protocol, spread, and confusion matrix.

Classification output

The trained final model assigned all 1,408 research posts:

Predicted category	Posts	Share
Academic & extracurricular engagement	580	41.2%
Social & living conditions	476	33.8%
Financial conditions	154	10.9%
University loyalty	198	14.1%

These values describe model predictions, not additional ground-truth labels. The machine-readable distribution is available in reports/prediction_distribution.json.

Performance

After one warm-up pass, the fitted pipeline classified the full 1,408-post batch in a median 0.305 seconds across 50 runs: approximately 4,624 posts per second on an Apple M1 CPU with 16 GB RAM. This benchmark includes TF-IDF transformation and model inference; training time is excluded.

See reports/benchmark.json for the measured values.

Quick start

The project uses Python 3.11+ and uv.

git clone https://github.com/erstcl/student-comments-classification-vciom.git
cd student-comments-classification-vciom
uv sync --all-extras

Run the verified evaluation:

uv run vciom-classifier evaluate \
  --labeled data/labeled.csv \
  --output reports/metrics.json

Train and classify the complete corpus:

uv run vciom-classifier train \
  --labeled data/labeled.csv \
  --model-out artifacts/model.joblib

uv run vciom-classifier predict \
  --model artifacts/model.joblib \
  --input data/posts_to_classify.csv \
  --output reports/predictions.csv \
  --summary-output reports/prediction_distribution.json

Measure local inference throughput:

uv run vciom-classifier benchmark \
  --labeled data/labeled.csv \
  --input data/posts_to_classify.csv \
  --repeats 50 \
  --output reports/benchmark.json

Reproducible analysis

The executed notebook notebooks/student_feedback_classification.ipynb walks through data validation, model comparison, cross-validation, error analysis, prediction distribution, and throughput measurement. It runs top-to-bottom from the repository root without Google Drive or hidden state.

Project structure

.
├── src/vciom_classifier/       # reusable data, modeling, and CLI package
├── tests/                      # data, model, and CLI tests
├── notebooks/                  # executed reader-facing analysis
├── data/                       # labeled and prediction datasets
├── reports/                    # metrics, benchmark, and prediction distribution
├── assets/                     # README visualizations
├── scripts/                    # reproducible notebook/chart builders
├── docs/                       # hackathon presentation
├── pyproject.toml              # package and tool configuration
└── uv.lock                     # reproducible dependency lock

My contribution

• built the Russian text preprocessing and feature-engineering workflow;
• designed and tested thematic classification signals;
• trained, tuned, and evaluated the multi-class models;
• implemented the leakage-safe reproducible pipeline and CLI;
• analyzed category-level errors and prediction distribution;
• co-developed the final research use cases and team presentation.

Documentation

• Executed analysis notebook
• Cross-validation metrics
• Inference benchmark
• Prediction distribution
• Team presentation

Data and license

The datasets contain public social-media comments supplied for the hackathon. IDs are used only as stable record identifiers; the model does not use them as features.

The source code is released under the MIT License. Dataset contents are not covered by the software license and remain subject to their original source terms.