Tokenization & Preprocessing
glayzzle/php-parser
▲ +1 ★ this week
NodeJS PHP Parser - extract AST or tokens
JavaScriptBSD-3-Clauseupdated Oct 30, 2026
A small library for converting tokenized PHP source code into XML (and potentially other formats)
$ git clone https://github.com/theseer/tokenizer.git▲ +1 ★ this week
NodeJS PHP Parser - extract AST or tokens
Query Translator is a search query translator with AST representation
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
▲ +1 ★ this week
A WHATWG-compliant HTML5 tokenizer and tag soup parser
▲ +2 ★ this week
Language model tokenization at GB/s
A non-validating SQL parser module for Python
Data from GitHub · snapshot Nov 16, 2026