Tokenization & Preprocessing
FoundationVision/UniTok
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
🤖 Language Models
PythonMITupdated Nov 14, 2025
🏆 #2,161 overall#74 of 148 in Tokenization & Preprocessing
amazon-far /deltatok
[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
$ git clone https://github.com/amazon-far/deltatok.git
Stars
261
261
Forks
7
7
Language
Python
License
Apache-2.0
Created
Mar 27, 2026
0.5 years old
Last push
Jul 17, 2026
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
RustApache-2.0updated Jan 22, 2026
Tokenization & Preprocessing
sugarme/tokenizer
NLP tokenizers written in Go language
🧠 Natural Language Processing
GoApache-2.0updated May 26, 2026
Tokenization & Preprocessing
AayushSameerShah/Neural-Net-Zero-to-Hero-with-Andrej
This repository contains the collection of explorative notebooks pure in python and in the language that we, humans can read. Have tried to compile all lectures from the Andrej Karpathy's 💎 playlist on Neural Networks - which we will end up with building GPT.
Jupyter NotebookMITupdated Apr 30, 2024
Tokenization & Preprocessing
frothywater/kanade-tokenizer
Kanade is a single-layer disentangled speech tokenizer that extracts compact tokens suitable for both generative and discriminative modeling.
Pythonno licenseupdated Jul 18, 2026
Tokenization & Preprocessing
rasbt/LLMs-from-scratch🔥 active
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Sep 22, 2026
Data from GitHub · snapshot Sep 24, 2026