Tokenization & Preprocessing
alpkeskin/gotoon
Token-Oriented Object Notation for Go – JSON for LLMs at half the token cost
GoMITupdated Nov 24, 2025
🏆 #628 overall#10 of 148 in Tokenization & Preprocessing
AgentOps-AI /tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
$ git clone https://github.com/AgentOps-AI/tokencost.gitStars
2K
2,005
Forks
106
106
Language
Python
License
MIT
Created
Dec 3, 2023
2.8 years old
Last push
Sep 5, 2025
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Apil-Shrestha/token-efficiency-lex🔥 active
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
🧠 Natural Language Processing
HTMLno licenseupdated Sep 24, 2026
Tokenization & Preprocessing
cohere-ai/magikarp
Code for the paper "Fishing for Magikarp"
🤖 Language Models
PythonApache-2.0updated Jul 10, 2026
Tokenization & Preprocessing
bminixhofer/zett
Code for Zero-Shot Tokenizer Transfer
🤖 Language Models
Pythonno licenseupdated Jan 14, 2025
Tokenization & Preprocessing
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
TypeScriptMITupdated Sep 3, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Data from GitHub · snapshot Sep 24, 2026