Tokenization & Preprocessing
RavenProject/Ravencoin
▲ +1 ★ this week
Ravencoin Core integration/staging tree
CMITupdated Aug 18, 2026
CodeChain's official implementation in Rust.
$ git clone https://github.com/CodeChain-io/codechain.git▲ +1 ★ this week
Ravencoin Core integration/staging tree
▲ +2 ★ this week
Sudachi in Rust 🦀 and new generation of SudachiPy
▲ +2 ★ this week
🎤 vibrato: Viterbi-based accelerated tokenizer
▲ +2 ★ this week
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
▲ +15 ★ this week
🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
Data from GitHub · snapshot Nov 16, 2026