πŸ† #3,462 overall#79 of 178 in Tokenization & Preprocessing

daac-tools /vaporetto

πŸ›₯ Vaporetto: Very accelerated pointwise prediction based tokenizer

$ git clone https://github.com/daac-tools/vaporetto.git
GitHub social preview for daac-tools/vaporetto
Stars
302
302
This week
+2 β˜…
+3 this month
Forks
12
12
Language
Rust
License
Apache-2.0
Created
Aug 18, 2021
5.2 years old
Last push
Nov 3, 2026

Stars, last 30 days

Oct 15, 2026 β†’ Nov 16, 2026
β˜… 299β–² +3β˜… 302

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

daac-tools/vibrato

β–² +2 β˜… this week

🎀 vibrato: Viterbi-based accelerated tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub β†—β˜… 430β‘‚ 27
Tokenization & Preprocessing

ikawaha/kagomeπŸ”₯ active

β–² +1 β˜… this week

Self-contained Japanese Morphological Analyzer written in pure Go

🧠 Natural Language Processing
GoMITupdated Nov 11, 2026
GitHub β†—β˜… 988β‘‚ 61
Tokenization & Preprocessing

ku-nlp/jumanpp

Juman++ (a Morphological Analyzer Toolkit)

🧠 Natural Language Processing
C++Apache-2.0updated Oct 13, 2026
GitHub β†—β˜… 413β‘‚ 47
Tokenization & Preprocessing

ongjin/garuπŸ”₯ active

β–² +1 β˜… this week

μ΄ˆκ²½λŸ‰ ν•œκ΅­μ–΄ ν˜•νƒœμ†Œ 뢄석기 β€” 1MB λͺ¨λΈλ‘œ λΈŒλΌμš°μ €μ—μ„œ μ‹€ν–‰(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko

🧠 Natural Language Processing
TypeScriptApache-2.0updated Nov 16, 2026
GitHub β†—β˜… 183β‘‚ 9
Tokenization & Preprocessing

WorksApplications/sudachi.rs

β–² +2 β˜… this week

Sudachi in Rust πŸ¦€ and new generation of SudachiPy

RustApache-2.0updated Oct 2, 2026
GitHub β†—β˜… 492β‘‚ 64
Tokenization & Preprocessing

marcelroed/gigatoken

β–² +2 β˜… this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub β†—β˜… 4.2Kβ‘‚ 225

Data from GitHub Β· snapshot Nov 16, 2026