marcelroed/gigatoken
▲ +2 ★ this week
Language model tokenization at GB/s
NLP tokenizers written in Go language
$ git clone https://github.com/sugarme/tokenizer.git▲ +2 ★ this week
Language model tokenization at GB/s
Solves basic Russian NLP tasks, API for lower level Natasha projects
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
▲ +1 ★ this week
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
▲ +1 ★ this week
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
▲ +1 ★ this week
Python port of Moses tokenizer, truecaser and normalizer
Data from GitHub · snapshot Nov 16, 2026