Tokenization & Preprocessing
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
🤖 Language Models
PythonMITupdated Sep 5, 2025
🏆 #2,528 overall#101 of 148 in Tokenization & Preprocessing
GlitchedPolygons /l8w8jwt
Minimal, OpenSSL-less and super lightweight JWT library written in C.
$ git clone https://github.com/GlitchedPolygons/l8w8jwt.gitStars
176
176
Forks
50
50
Language
C
License
Apache-2.0
Created
Jan 15, 2020
6.7 years old
Last push
Aug 16, 2025
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Tokenization & Preprocessing
lionsoul2014/friso
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
CApache-2.0updated Oct 29, 2023
Tokenization & Preprocessing
macmade/ClangKit
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
Cno licenseupdated Aug 2, 2021
Tokenization & Preprocessing
Tokenization & Preprocessing
alpkeskin/gotoon
Token-Oriented Object Notation for Go – JSON for LLMs at half the token cost
GoMITupdated Nov 24, 2025
Data from GitHub · snapshot Sep 24, 2026