Tokenization & Preprocessing
glayzzle/php-parser
▲ +1 ★ this week
NodeJS PHP Parser - extract AST or tokens
JavaScriptBSD-3-Clauseupdated Oct 30, 2026
Tokenizer (lexer) for anything
$ git clone https://github.com/bzick/tokenizer.git▲ +1 ★ this week
NodeJS PHP Parser - extract AST or tokens
▲ +1 ★ this week
A WHATWG-compliant HTML5 tokenizer and tag soup parser
▲ +2 ★ this week
Parser Building Toolkit for JavaScript
▲ +2 ★ this week
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
▲ +3 ★ this week
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
Data from GitHub · snapshot Nov 16, 2026