Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
🧠 Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
🏆 #1,713 overall#50 of 148 in Tokenization & Preprocessing🔥 active this week
taishi-i /nagisa
A Japanese tokenizer based on recurrent neural networks
$ git clone https://github.com/taishi-i/nagisa.gitStars
420
420
Forks
23
23
Language
Python
License
MIT
Created
Feb 14, 2018
8.6 years old
Last push
Sep 22, 2026
🔥 this week
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Tokenization & Preprocessing
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
🧠 Natural Language Processing
C++MITupdated Oct 24, 2025
Tokenization & Preprocessing
daac-tools/vibrato🔥 active
🎤 vibrato: Viterbi-based accelerated tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Sep 19, 2026
Tokenization & Preprocessing
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
🧠 Natural Language Processing
RustApache-2.0updated Jul 20, 2026
Tokenization & Preprocessing
bnosac/udpipe
R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
🧠 Natural Language Processing💎 Text Mining
C++MPL-2.0updated Mar 2, 2026
Data from GitHub · snapshot Sep 24, 2026