🏆 #3,560 overall#85 of 178 in Tokenization & Preprocessing

natasha /razdel

Rule-based token, sentence segmentation for Russian language

$ git clone https://github.com/natasha/razdel.git
GitHub social preview for natasha/razdel
Stars
290
290
This week
+1 ★
Forks
34
34
Language
Python
License
MIT
Created
Nov 10, 2018
8.0 years old
Last push
Apr 13, 2026

Stars, last 18 days

Oct 27, 2026 → Nov 16, 2026
★ 289▲ +1★ 290

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

natasha/natasha

Solves basic Russian NLP tasks, API for lower level Natasha projects

🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Oct 14, 2026
GitHub ↗★ 1.4K⑂ 122
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

jfilter/clean-text

▲ +1 ★ this week

🧹 Python package for text cleaning

🧠 Natural Language Processing
PythonOtherupdated Oct 26, 2026
GitHub ↗★ 1K⑂ 82
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91
Tokenization & Preprocessing

yooper/php-text-analysis

▲ +1 ★ this week

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

🧠 Natural Language Processing💎 Text Mining
PHPMITupdated Dec 28, 2024
GitHub ↗★ 535⑂ 91
Tokenization & Preprocessing

NLPOptimize/flash-tokenizer

▲ +1 ★ this week

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

🧠 Natural Language Processing
C++no licenseupdated Feb 2, 2026
GitHub ↗★ 458⑂ 11

Data from GitHub · snapshot Nov 16, 2026