Tokenization & Preprocessing
tlaceby/guide-to-interpreters-series
Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.
TypeScriptno licenseupdated May 2, 2024
🏆 #483 overall#9 of 148 in Tokenization & Preprocessing🔥 active this week
Chevrotain /chevrotain
Parser Building Toolkit for JavaScript
$ git clone https://github.com/Chevrotain/chevrotain.gitStars
2.8K
2,804
Forks
222
222
Language
TypeScript
License
Apache-2.0
Created
Apr 15, 2015
11.4 years old
Last push
Sep 24, 2026
🔥 this week
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
no-context/moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
JavaScriptBSD-3-Clauseupdated Feb 27, 2026
Tokenization & Preprocessing
glayzzle/php-parser
:herb: NodeJS PHP Parser - extract AST or tokens
JavaScriptBSD-3-Clauseupdated Sep 10, 2026
Tokenization & Preprocessing
mck89/peast
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
PHPBSD-3-Clauseupdated Sep 15, 2026
Tokenization & Preprocessing
botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
TypeScriptMITupdated Jan 27, 2023
Tokenization & Preprocessing
Data from GitHub · snapshot Sep 24, 2026