πŸ† #2,724 overall#58 of 178 in Tokenization & Preprocessing

daac-tools /vibrato

🎀 vibrato: Viterbi-based accelerated tokenizer

$ git clone https://github.com/daac-tools/vibrato.git
GitHub social preview for daac-tools/vibrato
Stars
430
430
This week
+2 β˜…
+3 this month
Forks
27
27
Language
Rust
License
Apache-2.0
Created
Jul 6, 2022
4.4 years old
Last push
Oct 25, 2026

Stars, last 30 days

Oct 15, 2026 β†’ Nov 16, 2026
β˜… 426β–² +4β˜… 430

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

daac-tools/vaporetto

β–² +2 β˜… this week

πŸ›₯ Vaporetto: Very accelerated pointwise prediction based tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Nov 3, 2026
GitHub β†—β˜… 302β‘‚ 12
Tokenization & Preprocessing

ikawaha/kagomeπŸ”₯ active

β–² +1 β˜… this week

Self-contained Japanese Morphological Analyzer written in pure Go

🧠 Natural Language Processing
GoMITupdated Nov 11, 2026
GitHub β†—β˜… 988β‘‚ 61
Tokenization & Preprocessing

ku-nlp/jumanpp

Juman++ (a Morphological Analyzer Toolkit)

🧠 Natural Language Processing
C++Apache-2.0updated Oct 13, 2026
GitHub β†—β˜… 413β‘‚ 47
Tokenization & Preprocessing

ongjin/garuπŸ”₯ active

β–² +1 β˜… this week

μ΄ˆκ²½λŸ‰ ν•œκ΅­μ–΄ ν˜•νƒœμ†Œ 뢄석기 β€” 1MB λͺ¨λΈλ‘œ λΈŒλΌμš°μ €μ—μ„œ μ‹€ν–‰(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko

🧠 Natural Language Processing
TypeScriptApache-2.0updated Nov 16, 2026
GitHub β†—β˜… 183β‘‚ 9
Tokenization & Preprocessing

WorksApplications/sudachi.rs

β–² +2 β˜… this week

Sudachi in Rust πŸ¦€ and new generation of SudachiPy

RustApache-2.0updated Oct 2, 2026
GitHub β†—β˜… 492β‘‚ 64
Tokenization & Preprocessing

marcelroed/gigatoken

β–² +2 β˜… this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub β†—β˜… 4.2Kβ‘‚ 225

Data from GitHub Β· snapshot Nov 16, 2026