Kanade is a single-layer disentangled speech tokenizer that extracts compact tokens suitable for both generative and discriminative modeling.
$ git clone https://github.com/frothywater/kanade-tokenizer.gitRust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
NLP tokenizers written in Go language
[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
A 1.6B causal Transformer audio tokenizer with streaming, variable bitrates, and semantic alignment across speech, sound, and music
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
💫 Industrial-strength Natural Language Processing (NLP) in Python
Data from GitHub · snapshot Sep 24, 2026