Tokenization & Preprocessing
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Apr 13, 2026
🏆 #2,066 overall#69 of 148 in Tokenization & Preprocessing
natasha /razdel
Rule-based token, sentence segmentation for Russian language
$ git clone https://github.com/natasha/razdel.gitStars
288
288
Forks
34
34
Language
Python
License
MIT
Created
Nov 10, 2018
7.9 years old
Last push
Apr 13, 2026
More in Tokenization & Preprocessing
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
jfilter/clean-text
🧹 Python package for text cleaning
🧠 Natural Language Processing
PythonOtherupdated May 15, 2026
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🧠 Natural Language Processing
PythonMITupdated Jun 2, 2025
Tokenization & Preprocessing
yooper/php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
🧠 Natural Language Processing💎 Text Mining
PHPMITupdated Dec 28, 2024
Tokenization & Preprocessing
NLPOptimize/flash-tokenizer
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
🧠 Natural Language Processing
C++no licenseupdated Feb 2, 2026
Data from GitHub · snapshot Sep 24, 2026