Tokenization & Preprocessing
FoundationVision/OmniTokenizer
[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.
PythonMITupdated Jul 9, 2024
Single-stage End-to-End Training for Tokenization and Generation
$ git clone https://github.com/ShivamDuggal4/UNITE-tokenization-generation.git[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.
[CVPR '26] SceneTok: A Compressed, Diffusable Token Space for 3D Scenes
▲ +15 ★ this week
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
▲ +2 ★ this week
Language model tokenization at GB/s
▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
▲ +1 ★ this week
Ravencoin Core integration/staging tree
Data from GitHub · snapshot Nov 16, 2026