Tokenization & Preprocessing
AgentOps-AI/tokencost
▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
🤖 Language Models
PythonMITupdated Sep 5, 2025
Minimal, OpenSSL-less and super lightweight JWT library written in C.
$ git clone https://github.com/GlitchedPolygons/l8w8jwt.git▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
▲ +1 ★ this week
Ravencoin Core integration/staging tree
▲ +1 ★ this week
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
▲ +1 ★ this week
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
Token-Oriented Object Notation for Go – JSON for LLMs at half the token cost
Data from GitHub · snapshot Nov 16, 2026