Tokenization & Preprocessing
rasbt/LLMs-from-scratch🔥 active
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Sep 22, 2026
🏆 #2,929 overall#130 of 148 in Tokenization & Preprocessing
lucidrains /charformer-pytorch
Implementation of the GBST block from the Charformer paper, in Pytorch
$ git clone https://github.com/lucidrains/charformer-pytorch.gitStars
118
118
Forks
11
11
Language
Python
License
MIT
Created
Jun 30, 2021
5.2 years old
Last push
Jul 15, 2021
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
explosion/spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
🏷️ Named Entity Recognition🗂️ Text Classification🧠 Natural Language Processing
PythonMITupdated Aug 24, 2026
Tokenization & Preprocessing
analyticalrohit/llms-from-scratch
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
🤖 Language Models
Jupyter NotebookMITupdated Jun 8, 2026
Tokenization & Preprocessing
guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
RustApache-2.0updated Jan 22, 2026
Tokenization & Preprocessing
ImadSaddik/Train_Your_Language_Model_Course
Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.
Jupyter Notebookno licenseupdated Sep 26, 2025
Tokenization & Preprocessing
jshuadvd/LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
🤖 Language Models🧠 Natural Language Processing
Pythonno licenseupdated Jul 20, 2024
Data from GitHub · snapshot Sep 24, 2026