Tokenization & Preprocessing
theseer/tokenizer
▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
ACM MM 2026 Oral
$ git clone https://github.com/wjf5203/TokBench.git▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
▲ +2 ★ this week
Language model tokenization at GB/s
A non-validating SQL parser module for Python
▲ +2 ★ this week
Parser Building Toolkit for JavaScript
Solves basic Russian NLP tasks, API for lower level Natasha projects
Data from GitHub · snapshot Nov 16, 2026