Tokenization & Preprocessing
analyticalrohit/llms-from-scratch
▲ +1 ★ this week
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
🤖 Language Models
Jupyter NotebookMITupdated Jun 8, 2026
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
$ git clone https://github.com/FoundationVision/UniTok.git▲ +1 ★ this week
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
▲ +2 ★ this week
[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens
▲ +499 ★ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
Code for the paper "Fishing for Magikarp"
▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Data from GitHub · snapshot Nov 16, 2026