Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
🏆 #3,008 overall#136 of 148 in Tokenization & Preprocessing
togatoga /kanpyo
Japanese Morphological Analyzer written in Rust
$ git clone https://github.com/togatoga/kanpyo.gitStars
109
109
Forks
1
1
Language
Rust
License
MIT
Created
Oct 11, 2023
3.0 years old
Last push
Sep 5, 2026
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
ikawaha/kagome🔥 active
Self-contained Japanese Morphological Analyzer written in pure Go
GoMITupdated Sep 19, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
🧠 Natural Language Processing
C++MITupdated Oct 24, 2025
Tokenization & Preprocessing
taishi-i/nagisa🔥 active
A Japanese tokenizer based on recurrent neural networks
🧠 Natural Language Processing
PythonMITupdated Sep 22, 2026
Data from GitHub · snapshot Sep 24, 2026