Tokenization & Preprocessing
daac-tools/vibratoπ₯ active
π€ vibrato: Viterbi-based accelerated tokenizer
π§ Natural Language Processing
RustApache-2.0updated Sep 19, 2026
π #1,602 overall#45 of 148 in Tokenization & Preprocessingπ₯ active this week
WorksApplications /sudachi.rs
Sudachi in Rust π¦ and new generation of SudachiPy
$ git clone https://github.com/WorksApplications/sudachi.rs.gitStars
481
481
Forks
60
60
Language
Rust
License
Apache-2.0
Created
Nov 23, 2019
6.8 years old
Last push
Sep 24, 2026
π₯ this week
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
daac-tools/vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
ikawaha/kagomeπ₯ active
Self-contained Japanese Morphological Analyzer written in pure Go
GoMITupdated Sep 19, 2026
Tokenization & Preprocessing
ongjin/garuπ₯ active
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
π§ Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
π§ Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
Tokenization & Preprocessing
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
π Machine Translationπ§ Natural Language Processing
C++MITupdated Jan 10, 2026
Data from GitHub Β· snapshot Sep 24, 2026