Tokenization & Preprocessing
sugarme/tokenizer
NLP tokenizers written in Go language
🧠 Natural Language Processing
GoApache-2.0updated May 26, 2026
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
$ git clone https://github.com/guillaume-be/rust-tokenizers.gitNLP tokenizers written in Go language
▲ +2 ★ this week
[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
Kanade is a single-layer disentangled speech tokenizer that extracts compact tokens suitable for both generative and discriminative modeling.
Implementation of the GBST block from the Charformer paper, in Pytorch
▲ +499 ★ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
▲ +24 ★ this week
💫 Industrial-strength Natural Language Processing (NLP) in Python
Data from GitHub · snapshot Nov 16, 2026