Tokenization & Preprocessing
belladoreai/llama3-tokenizer-js
JS tokenizer for LLaMA 3 and LLaMA 3.1
JavaScriptMITupdated Jul 28, 2025
JS tokenizer for LLaMA 1 and 2
$ git clone https://github.com/belladoreai/llama-tokenizer-js.gitJS tokenizer for LLaMA 3 and LLaMA 3.1
▲ +2 ★ this week
Language model tokenization at GB/s
▲ +2 ★ this week
Parser Building Toolkit for JavaScript
▲ +2 ★ this week
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
▲ +1 ★ this week
NodeJS PHP Parser - extract AST or tokens
Data from GitHub · snapshot Nov 16, 2026