Tokenization & Preprocessing
daac-tools/vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Jul 20, 2026
π #1,705 overall#49 of 148 in Tokenization & Preprocessingπ₯ active this week
daac-tools /vibrato
π€ vibrato: Viterbi-based accelerated tokenizer
$ git clone https://github.com/daac-tools/vibrato.gitStars
423
423
Forks
26
26
Language
Rust
License
Apache-2.0
Created
Jul 6, 2022
4.2 years old
Last push
Sep 19, 2026
π₯ this week
More in Tokenization & Preprocessing
Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
π§ Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
Tokenization & Preprocessing
ongjin/garuπ₯ active
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
π§ Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
ikawaha/kagomeπ₯ active
Self-contained Japanese Morphological Analyzer written in pure Go
GoMITupdated Sep 19, 2026
Tokenization & Preprocessing
WorksApplications/sudachi.rsπ₯ active
Sudachi in Rust π¦ and new generation of SudachiPy
RustApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
π§ Natural Language Processing
RustMITupdated Sep 2, 2026
Data from GitHub Β· snapshot Sep 24, 2026