Tokenization & Preprocessing
daac-tools/vibrato
▲ +2 ★ this week
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
CLI tool – estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.
$ git clone https://github.com/nooscraft/tokuin.git▲ +2 ★ this week
🎤 vibrato: Viterbi-based accelerated tokenizer
▲ +2 ★ this week
Ready-made tokenizer library for working with GPT and tiktoken
▲ +2 ★ this week
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
▲ +1 ★ this week
초경량 한국어 형태소 분석기 — 1MB 모델로 브라우저에서 실행(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
Data from GitHub · snapshot Nov 16, 2026