A minimal, educational implementation of language models specifically designed for Kazakh text. This project demonstrates how to build neural language models from scratch using both pure Python and PyTorch implementations.
kazakh-micro-lm is a lightweight language model implementation that showcases the fundamentals of modern language modeling techniques. It includes two different approaches:
- Pure Python Implementation - A minimal RNN-style model with no dependencies beyond standard library
- PyTorch GPT-2 Implementation - A small-scale GPT-2 architecture optimized for Kazakh text
Perfect for educational purposes, experimentation, and understanding the core concepts behind modern language models.
- Zero-Dependency Core: Pure Python implementation requires no external libraries
- Kazakh Language Support: Pre-trained tokenizer optimized for Kazakh text
- Educational Focus: Clear, readable code with extensive logging
- Multiple Implementations: Learn from both simple and advanced architectures
- Easy to Extend: Modular design makes it simple to experiment with new ideas
# Train a model on your text
python main.py --mode train --input abay-joly.txt --epochs 100 --lr 0.1
# Generate text
python main.py --mode infer --prompt "шла саша" --max_len 10# Install dependencies
pip install torch transformers tqdm requests
# Train the model
python gpt2_minimal_kazakh.pykazakh-micro-lm/
├── model.py # Pure Python RNN implementation
├── trainer.py # Training loop for pure Python model
├── inference.py # Text generation utilities
├── tokenizer.py # Simple word-level tokenizer
├── main.py # CLI interface for training/inference
├── gpt2_minimal_kazakh.py # PyTorch GPT-2 implementation
├── gpt2.c # C implementation (experimental)
├── abay-joly.txt # Sample Kazakh text for training
├── vocab.json # Pre-trained vocabulary
├── merges.txt # BPE merges for tokenization
├── tokenizer.json # Tokenizer configuration
├── model.json # Saved model weights
├── experiments/ # Training experiments and tools
│ ├── min-rnn.py # Minimal vanilla RNN (NumPy)
│ ├── min-rnn2.py # Minimal RNN with PyTorch
│ ├── tf-rnn.py # TensorFlow RNN implementation
│ ├── llm-small.py # Small LLM inference example
│ ├── mgpt_tuning.py # mGPT fine-tuning script
│ ├── prepare-data-for-nanogpt.py # Data preparation for nanoGPT
│ ├── prepare-data-for-nano-tokenize.py # Tokenization preparation
│ ├── show-tokenized.py # Tokenization visualization
│ └── clean-tokenize.py # BPE tokenization example
└── docs/
└── microLLM.md # Project planning and resource requirements
The pure Python implementation features:
- Simple embedding layer with one-hot encoding
- Matrix operations implemented from scratch
- Gradient descent optimization
- Cross-entropy loss
- JSON-based model serialization
The PyTorch implementation includes:
- Multi-head self-attention mechanism
- Transformer blocks with layer normalization
- Causal language modeling objective
- Mixed precision training support
- CUDA acceleration when available
- Prepare your Kazakh text file (UTF-8 encoded)
- Train with the pure Python implementation:
python main.py --mode train --input your_text.txt --epochs 200 --lr 0.05- Or use the PyTorch implementation for better results:
python gpt2_minimal_kazakh.py
# Modify the script to point to your training dataPure Python Model:
--epochs: Number of training iterations (default: 100)--lr: Learning rate (default: 0.1)--max_len: Maximum generation length (default: 5)
PyTorch GPT-2 Model:
ctx_len: Context length (default: 128)n_emb: Embedding dimension (default: 256)n_heads: Number of attention heads (default: 8)n_layers: Number of transformer layers (default: 6)batch_size: Training batch size (default: 64)
The experiments/ directory contains various training scripts and tools for exploring different model architectures and tokenization approaches:
- RNN Implementations: Compare vanilla RNN implementations across NumPy, PyTorch, and TensorFlow
- Tokenization Tools: Scripts for data preparation, BPE tokenization, and token visualization
- Fine-tuning Scripts: Examples for fine-tuning pre-trained models like mGPT on Kazakh text
- Data Preparation: Tools to prepare datasets in the format expected by nanoGPT and other frameworks
These scripts are ideal for experimentation and understanding different approaches to training language models.
- Education: Learn how language models work under the hood
- Research: Experiment with new architectures and training techniques
- Prototyping: Quickly test ideas before scaling up
- Kazakh NLP: Build applications for Kazakh language processing
- Low-Resource Settings: Train models without heavy computational requirements
- Python 3.6+
- No external dependencies
- Python 3.8+
- PyTorch 2.0+
- transformers
- tqdm
- requests
The pure Python implementation is intentionally minimal and not optimized for speed. It's designed for educational purposes to understand the fundamentals. For production use cases, the PyTorch implementation offers:
- GPU acceleration (10-100x faster)
- Mixed precision training
- Efficient batch processing
- Modern attention mechanisms
Contributions are welcome! This project is designed to be simple and educational, so please keep changes minimal and well-documented. Areas for contribution:
- Additional training examples
- Performance optimizations
- Better documentation
- Support for other Turkic languages
- Evaluation metrics and benchmarks
MIT License - see LICENSE file for details
If you use this project in your research or education, please cite:
@software{kazakh_micro_lm,
title = {Kazakh Micro Language Model},
author = {Stukenov, Saken},
year = {2024},
url = {https://github.com/stukenov/kazakh-micro-lm}
}- Inspired by Andrej Karpathy's educational materials on neural networks
- Tokenizer trained on Kazakh text corpus
- Built with educational simplicity in mind
- minGPT - Minimal PyTorch re-implementation of GPT
- nanoGPT - Simplest, fastest repository for training GPT
- micrograd - Tiny autograd engine
Made with focus on education and simplicity for the Kazakh language community.