Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
🏆 #1,513 overall#39 of 148 in Tokenization & Preprocessing
polm /fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
$ git clone https://github.com/polm/fugashi.gitStars
535
535
Forks
40
40
Language
C++
License
MIT
Created
Oct 14, 2019
6.9 years old
Last push
Oct 24, 2025
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
taishi-i/nagisa🔥 active
A Japanese tokenizer based on recurrent neural networks
🧠 Natural Language Processing
PythonMITupdated Sep 22, 2026
Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
🧠 Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
Kensuke-Mitsuzawa/JapaneseTokenizers
aim to use JapaneseTokenizer as easy as possible
🧠 Natural Language Processing
PythonMITupdated Mar 25, 2019
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Data from GitHub · snapshot Sep 24, 2026