Tokenization & Preprocessing
glayzzle/php-parser
:herb: NodeJS PHP Parser - extract AST or tokens
JavaScriptBSD-3-Clauseupdated Sep 10, 2026
🏆 #2,726 overall#118 of 148 in Tokenization & Preprocessing
bzick /tokenizer
Tokenizer (lexer) for golang
$ git clone https://github.com/bzick/tokenizer.gitStars
142
142
Forks
11
11
Language
Go
License
MIT
Created
Oct 19, 2021
4.9 years old
Last push
Jul 26, 2026
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Tokenization & Preprocessing
Tokenization & Preprocessing
Chevrotain/chevrotain🔥 active
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
no-context/moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
JavaScriptBSD-3-Clauseupdated Feb 27, 2026
Tokenization & Preprocessing
alasdairforsythe/tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript
GoMITupdated Jul 10, 2026
Data from GitHub · snapshot Sep 24, 2026