Tokenization & Preprocessing
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Apr 13, 2026
🏆 #1,652 overall#48 of 148 in Tokenization & Preprocessing
NLPOptimize /flash-tokenizer
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
$ git clone https://github.com/NLPOptimize/flash-tokenizer.gitStars
458
458
Forks
11
11
Language
C++
License
None
Created
Mar 9, 2025
1.5 years old
Last push
Feb 2, 2026
More in Tokenization & Preprocessing
Tokenization & Preprocessing
jfilter/clean-text
🧹 Python package for text cleaning
🧠 Natural Language Processing
PythonOtherupdated May 15, 2026
Tokenization & Preprocessing
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
Tokenization & Preprocessing
sugarme/tokenizer
NLP tokenizers written in Go language
🧠 Natural Language Processing
GoApache-2.0updated May 26, 2026
Tokenization & Preprocessing
natasha/razdel
Rule-based token, sentence segmentation for Russian language
🧠 Natural Language Processing
PythonMITupdated Apr 13, 2026
Tokenization & Preprocessing
aymara/lima
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
🏷️ Named Entity Recognition⛏️ Information Extraction🧠 Natural Language Processing
C++Otherupdated Jul 8, 2026
Data from GitHub · snapshot Sep 24, 2026