🏆 #3,175 overall#72 of 178 in Tokenization & Preprocessing

sugarme /tokenizer

NLP tokenizers written in Go language

$ git clone https://github.com/sugarme/tokenizer.git
GitHub social preview for sugarme/tokenizer
Stars
342
342
This week
0 ★
+2 this month
Forks
69
69
Language
Go
License
Apache-2.0
Created
Aug 3, 2020
6.3 years old
Last push
May 26, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 340▲ +2★ 342

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

natasha/natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Oct 14, 2026
GitHub ↗★ 1.4K⑂ 122
Tokenization & Preprocessing

lovit/soynlp

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

🧠 Natural Language Processing
PythonOtherupdated Oct 4, 2026
GitHub ↗★ 996⑂ 181
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91
Tokenization & Preprocessing

polm/fugashi

▲ +1 ★ this week

A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.

🧠 Natural Language Processing
C++MITupdated Oct 24, 2025
GitHub ↗★ 538⑂ 40
Tokenization & Preprocessing

hplt-project/sacremoses

▲ +1 ★ this week

Python port of Moses tokenizer, truecaser and normalizer

🌍 Machine Translation🧠 Natural Language Processing
PythonMITupdated Sep 7, 2026
GitHub ↗★ 498⑂ 60

Data from GitHub · snapshot Nov 16, 2026