Tokenization & Preprocessing
daac-tools/vibratoπ₯ active
π€ vibrato: Viterbi-based accelerated tokenizer
π§ Natural Language Processing
RustApache-2.0updated Sep 19, 2026
π #2,718 overall#117 of 148 in Tokenization & Preprocessing
nooscraft /tokuin
CLI tool β estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.
$ git clone https://github.com/nooscraft/tokuin.gitStars
144
144
Forks
8
8
Language
Rust
License
Other
Created
Nov 6, 2025
0.9 years old
Last push
Jul 14, 2026
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
RustMITupdated Jul 1, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
ongjin/garuπ₯ active
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
π§ Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Data from GitHub Β· snapshot Sep 24, 2026