Tokenization & Preprocessing
analyticalrohit/llms-from-scratch
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
🤖 Language Models
Jupyter NotebookMITupdated Jun 8, 2026
🏆 #1,517 overall#41 of 148 in Tokenization & Preprocessing
FoundationVision /UniTok
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
$ git clone https://github.com/FoundationVision/UniTok.gitStars
533
533
Forks
14
14
Language
Python
License
MIT
Created
Feb 27, 2025
1.6 years old
Last push
Nov 14, 2025
More in Tokenization & Preprocessing
Tokenization & Preprocessing
amazon-far/deltatok
[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
PythonApache-2.0updated Jul 17, 2026
Tokenization & Preprocessing
rasbt/LLMs-from-scratch🔥 active
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Sep 22, 2026
Tokenization & Preprocessing
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
🤖 Language Models
PythonMITupdated Sep 5, 2025
Tokenization & Preprocessing
cohere-ai/magikarp
Code for the paper "Fishing for Magikarp"
🤖 Language Models
PythonApache-2.0updated Jul 10, 2026
Tokenization & Preprocessing
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Data from GitHub · snapshot Sep 24, 2026