Tokenization & Preprocessing
rasbt/LLMs-from-scratch
▲ +499 ★ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Nov 9, 2026
Implementation of the GBST block from the Charformer paper, in Pytorch
$ git clone https://github.com/lucidrains/charformer-pytorch.git▲ +499 ★ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
▲ +24 ★ this week
💫 Industrial-strength Natural Language Processing (NLP) in Python
▲ +1 ★ this week
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
Data from GitHub · snapshot Nov 16, 2026