Tokenization & Preprocessing
polm/fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
π§ Natural Language Processing
C++MITupdated Oct 24, 2025
π #2,759 overall#119 of 148 in Tokenization & Preprocessing
Kensuke-Mitsuzawa /JapaneseTokenizers
aim to use JapaneseTokenizer as easy as possible
$ git clone https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers.gitStars
138
138
Forks
21
21
Language
Python
License
MIT
Created
Sep 1, 2015
11.1 years old
Last push
Mar 25, 2019
More in Tokenization & Preprocessing
Tokenization & Preprocessing
ku-nlp/jumanpp
Juman++ (a Morphological Analyzer Toolkit)
π§ Natural Language Processing
C++Apache-2.0updated Apr 17, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
π§ Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
π·οΈ Named Entity Recognitionπ§ Natural Language Processing
PythonMITupdated Apr 13, 2026
Tokenization & Preprocessing
lovit/soynlp
νκ΅μ΄ μμ°μ΄μ²λ¦¬λ₯Ό μν νμ΄μ¬ λΌμ΄λΈλ¬λ¦¬μ λλ€. λ¨μ΄ μΆμΆ/ ν ν¬λμ΄μ / νμ¬νλ³/ μ μ²λ¦¬μ κΈ°λ₯μ μ 곡ν©λλ€.
π§ Natural Language Processing
PythonOtherupdated Mar 10, 2026
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
π§ Natural Language Processing
PythonMITupdated Jun 2, 2025
Data from GitHub Β· snapshot Sep 24, 2026