🏆 #5,540 overall#138 of 178 in Tokenization & Preprocessing

nooscraft /tokuin

CLI tool – estimates LLM tokens/costs and runs provider-aware load tests for OpenAI, Anthropic, OpenRouter, or custom endpoints.

$ git clone https://github.com/nooscraft/tokuin.git
GitHub social preview for nooscraft/tokuin
Stars
146
146
This week
0 ★
+2 this month
Forks
8
8
Language
Rust
License
Other
Created
Nov 6, 2025
1.0 years old
Last push
Jul 14, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 144▲ +2★ 146

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

daac-tools/vibrato

▲ +2 ★ this week

🎤 vibrato: Viterbi-based accelerated tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub ↗★ 430⑂ 27
Tokenization & Preprocessing

zurawiki/tiktoken-rs

▲ +2 ★ this week

Ready-made tokenizer library for working with GPT and tiktoken

RustMITupdated Sep 24, 2026
GitHub ↗★ 412⑂ 76
Tokenization & Preprocessing

daac-tools/vaporetto

▲ +2 ★ this week

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Nov 3, 2026
GitHub ↗★ 302⑂ 12
Tokenization & Preprocessing

ongjin/garu🔥 active

▲ +1 ★ this week

초경량 한국어 형태소 분석기 — 1MB 모델로 브라우저에서 실행(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko

🧠 Natural Language Processing
TypeScriptApache-2.0updated Nov 16, 2026
GitHub ↗★ 183⑂ 9
Tokenization & Preprocessing

togatoga/kanpyo

Japanese Morphological Analyzer written in Rust

RustMITupdated Oct 9, 2026
GitHub ↗★ 108⑂ 1
Tokenization & Preprocessing

theseer/tokenizer

▲ +2 ★ this week

A small library for converting tokenized PHP source code into XML (and potentially other formats)

PHPOtherupdated Feb 3, 2026
GitHub ↗★ 5.2K⑂ 25

Data from GitHub · snapshot Nov 16, 2026