🏆 #5,711 overall#140 of 178 in Tokenization & Preprocessing

Kensuke-Mitsuzawa /JapaneseTokenizers

aim to use JapaneseTokenizer as easy as possible

$ git clone https://github.com/Kensuke-Mitsuzawa/JapaneseTokenizers.git
GitHub social preview for Kensuke-Mitsuzawa/JapaneseTokenizers
Stars
138
138
This week
0 ★
0 this month
Forks
21
21
Language
Python
License
MIT
Created
Sep 1, 2015
11.2 years old
Last push
Mar 25, 2019

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 138▲ +0★ 138

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

polm/fugashi

▲ +1 ★ this week

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

🧠 Natural Language Processing
C++MITupdated Oct 24, 2025
GitHub ↗★ 538⑂ 40
Tokenization & Preprocessing

ku-nlp/jumanpp

Juman++ (a Morphological Analyzer Toolkit)

🧠 Natural Language Processing
C++Apache-2.0updated Oct 13, 2026
GitHub ↗★ 413⑂ 47
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

natasha/natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Oct 14, 2026
GitHub ↗★ 1.4K⑂ 122
Tokenization & Preprocessing

lovit/soynlp

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

🧠 Natural Language Processing
PythonOtherupdated Oct 4, 2026
GitHub ↗★ 996⑂ 181
Tokenization & Preprocessing

ikawaha/kagome🔥 active

▲ +1 ★ this week

Self-contained Japanese Morphological Analyzer written in pure Go

🧠 Natural Language Processing
GoMITupdated Nov 11, 2026
GitHub ↗★ 988⑂ 61

Data from GitHub · snapshot Nov 16, 2026