Tokenization & Preprocessing
theseer/tokenizer
▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
:heart_eyes_cat: Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer
$ git clone https://github.com/bevacqua/megamark.git▲ +2 ★ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
▲ +2 ★ this week
Language model tokenization at GB/s
A non-validating SQL parser module for Python
▲ +2 ★ this week
Parser Building Toolkit for JavaScript
Solves basic Russian NLP tasks, API for lower level Natasha projects
Data from GitHub · snapshot Nov 16, 2026