Tokenization & Preprocessing
explosion/spaCy
▲ +24 ★ this week
💫 Industrial-strength Natural Language Processing (NLP) in Python
🧠 Natural Language Processing🏷️ Named Entity Recognition🗂️ Text Classification
PythonMITupdated Oct 27, 2026
Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta
$ git clone https://github.com/kyegomez/MambaByte.git▲ +24 ★ this week
💫 Industrial-strength Natural Language Processing (NLP) in Python
▲ +1 ★ this week
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
▲ +499 ★ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
▲ +1 ★ this week
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
▲ +1 ★ this week
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
Data from GitHub · snapshot Nov 16, 2026