Tokenization & Preprocessing
daac-tools/vibratoπ₯ active
π€ vibrato: Viterbi-based accelerated tokenizer
π§ Natural Language Processing
RustApache-2.0updated Sep 19, 2026
π #2,025 overall#68 of 148 in Tokenization & Preprocessing
daac-tools /vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
$ git clone https://github.com/daac-tools/vaporetto.gitStars
299
299
Forks
12
12
Language
Rust
License
Apache-2.0
Created
Aug 18, 2021
5.1 years old
Last push
Jul 20, 2026
More in Tokenization & Preprocessing
Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
π§ Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
Tokenization & Preprocessing
ongjin/garuπ₯ active
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
π§ Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
ikawaha/kagomeπ₯ active
Self-contained Japanese Morphological Analyzer written in pure Go
GoMITupdated Sep 19, 2026
Tokenization & Preprocessing
WorksApplications/sudachi.rsπ₯ active
Sudachi in Rust π¦ and new generation of SudachiPy
RustApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
π§ Natural Language Processing
RustMITupdated Sep 2, 2026
Data from GitHub Β· snapshot Sep 24, 2026