Tokenization & Preprocessing
Chevrotain/chevrotain🔥 active
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Sep 24, 2026
🏆 #1,833 overall#57 of 148 in Tokenization & Preprocessing
macmade /ClangKit
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
$ git clone https://github.com/macmade/ClangKit.gitStars
367
367
Forks
45
45
Language
C
License
None
Created
Aug 12, 2012
14.1 years old
Last push
Aug 2, 2021
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
lionsoul2014/friso
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
CApache-2.0updated Oct 29, 2023
Tokenization & Preprocessing
mck89/peast
JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification
PHPBSD-3-Clauseupdated Sep 15, 2026
Tokenization & Preprocessing
GlitchedPolygons/l8w8jwt
Minimal, OpenSSL-less and super lightweight JWT library written in C.
CApache-2.0updated Aug 16, 2025
Tokenization & Preprocessing
Tokenization & Preprocessing
rasbt/LLMs-from-scratch🔥 active
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Sep 22, 2026
Data from GitHub · snapshot Sep 24, 2026