Tokenization & Preprocessing
ScrapeGraphAI/toonify
โฒ +1 โ this week
Toonify: Compact data format reducing LLM token usage by 30-60%
Pythonno licenseupdated Feb 6, 2026
๐ Token-Oriented Object Notation (TOON) โ compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
$ git clone https://github.com/toon-format/toon.gitโฒ +1 โ this week
Toonify: Compact data format reducing LLM token usage by 30-60%
โฒ +2 โ this week
Language model tokenization at GB/s
โฒ +1 โ this week
Easy token price estimates for 400+ LLMs. TokenOps.
Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.
Token-Oriented Object Notation for Go โ JSON for LLMs at half the token cost
โฒ +1 โ this week
Simple Byte pair Encoding mechanism used for tokenization process . written purely in C
Data from GitHub ยท snapshot Nov 16, 2026