Tokenization & Preprocessing
daac-tools/vibratoπ₯ active
π€ vibrato: Viterbi-based accelerated tokenizer
π§ Natural Language Processing
RustApache-2.0updated Sep 19, 2026
π #1,042 overall#20 of 148 in Tokenization & Preprocessingπ₯ active this week
ikawaha /kagome
Self-contained Japanese Morphological Analyzer written in pure Go
$ git clone https://github.com/ikawaha/kagome.gitStars
983
983
Forks
60
60
Language
Go
License
MIT
Created
Jun 26, 2014
12.2 years old
Last push
Sep 19, 2026
π₯ this week
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
π§ Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
ongjin/garuπ₯ active
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
π§ Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
WorksApplications/sudachi.rsπ₯ active
Sudachi in Rust π¦ and new generation of SudachiPy
RustApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
taishi-i/nagisaπ₯ active
A Japanese tokenizer based on recurrent neural networks
π§ Natural Language Processing
PythonMITupdated Sep 22, 2026
Data from GitHub Β· snapshot Sep 24, 2026