Tokenization & Preprocessing
🏆 #2,208 overall#80 of 148 in Tokenization & Preprocessing
SmartTokenLabs /TokenScript
TokenScript schema, specs and paper
$ git clone https://github.com/SmartTokenLabs/TokenScript.gitStars
249
249
Forks
68
68
Language
JavaScript
License
MIT
Created
Feb 6, 2019
7.6 years old
Last push
Jul 31, 2024
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
alasdairforsythe/tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
GoMITupdated Jul 10, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
TypeScriptMITupdated Sep 3, 2026
Tokenization & Preprocessing
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Data from GitHub · snapshot Sep 24, 2026