Tokenization & Preprocessing
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonMITupdated Oct 14, 2026
A non-validating SQL parser module for Python
$ git clone https://github.com/andialbrecht/sqlparse.gitSolves basic Russian NLP tasks, API for lower level Natasha projects
▲ +1 ★ this week
Fast and customizable text tokenization library with BPE and SentencePiece support
A tool that allow you to run SQL-like query on local files instead of database files using the GitQL SDK.
▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
▲ +2 ★ this week
Language model tokenization at GB/s
▲ +2 ★ this week
Parser Building Toolkit for JavaScript
Data from GitHub · snapshot Nov 16, 2026