🏆 #2,333 overall#48 of 178 in Tokenization & Preprocessing

FoundationVision /UniTok

[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding

$ git clone https://github.com/FoundationVision/UniTok.git
GitHub social preview for FoundationVision/UniTok
Stars
538
538
This week
+2 ★
+3 this month
Forks
16
16
Language
Python
License
MIT
Created
Feb 27, 2025
1.7 years old
Last push
Nov 14, 2025

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 535▲ +3★ 538

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

analyticalrohit/llms-from-scratch

▲ +1 ★ this week

Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.

🤖 Language Models
Jupyter NotebookMITupdated Jun 8, 2026
GitHub ↗★ 387⑂ 90
Tokenization & Preprocessing

amazon-far/deltatok

▲ +2 ★ this week

[CVPR 2026 Highlight] A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokens

PythonApache-2.0updated Jul 17, 2026
GitHub ↗★ 279⑂ 7
Tokenization & Preprocessing

rasbt/LLMs-from-scratch

▲ +499 ★ this week

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Nov 9, 2026
GitHub ↗★ 108.5K⑂ 16.8K
Tokenization & Preprocessing

AgentOps-AI/tokencost

▲ +1 ★ this week

Easy token price estimates for 400+ LLMs. TokenOps.

🤖 Language Models
PythonMITupdated Sep 5, 2025
GitHub ↗★ 2K⑂ 109
Tokenization & Preprocessing

cohere-ai/magikarp

Code for the paper "Fishing for Magikarp"

🤖 Language Models
PythonApache-2.0updated Jul 10, 2026
GitHub ↗★ 191⑂ 20
Tokenization & Preprocessing

theseer/tokenizer

▲ +2 ★ this week

A small library for converting tokenized PHP source code into XML (and potentially other formats)

PHPOtherupdated Feb 3, 2026
GitHub ↗★ 5.2K⑂ 25

Data from GitHub · snapshot Nov 16, 2026