Tokenization & Preprocessing
daac-tools/vibratoπ₯ active
π€ vibrato: Viterbi-based accelerated tokenizer
π§ Natural Language Processing
RustApache-2.0updated Sep 19, 2026
π #1,735 overall#51 of 148 in Tokenization & Preprocessing
ku-nlp /jumanpp
Juman++ (a Morphological Analyzer Toolkit)
$ git clone https://github.com/ku-nlp/jumanpp.gitStars
413
413
Forks
47
47
Language
C++
License
Apache-2.0
Created
Oct 11, 2016
10.0 years old
Last push
Apr 17, 2026
More in Tokenization & Preprocessing
Tokenization & Preprocessing
taishi-i/nagisaπ₯ active
A Japanese tokenizer based on recurrent neural networks
π§ Natural Language Processing
PythonMITupdated Sep 22, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
π₯ Vaporetto: Very accelerated pointwise prediction based tokenizer
π§ Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
ongjin/garuπ₯ active
μ΄κ²½λ νκ΅μ΄ ννμ λΆμκΈ° β 1MB λͺ¨λΈλ‘ λΈλΌμ°μ μμ μ€ν(WASM). Ultra-lightweight Korean morphological analyzer in the browser. npm: garu-ko
π§ Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
ikawaha/kagomeπ₯ active
Self-contained Japanese Morphological Analyzer written in pure Go
GoMITupdated Sep 19, 2026
Tokenization & Preprocessing
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
π§ Natural Language Processing
C++MITupdated Oct 24, 2025
Data from GitHub Β· snapshot Sep 24, 2026