Tokenization & Preprocessing
🏆 #2,185 overall#77 of 148 in Tokenization & Preprocessing
CodeChain-io /codechain
CodeChain's official implementation in Rust.
$ git clone https://github.com/CodeChain-io/codechain.gitStars
256
256
Forks
50
50
Language
Rust
License
AGPL-3.0
Created
Jan 23, 2018
8.7 years old
Last push
Jan 7, 2023
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
WorksApplications/sudachi.rs🔥 active
Sudachi in Rust 🦀 and new generation of SudachiPy
RustApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
toon-format/toon
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
TypeScriptMITupdated Sep 3, 2026
Data from GitHub · snapshot Sep 24, 2026