Tokenization & Preprocessing
belladoreai/llama3-tokenizer-js
JS tokenizer for LLaMA 3 and LLaMA 3.1
JavaScriptMITupdated Jul 28, 2025
🏆 #1,844 overall#58 of 148 in Tokenization & Preprocessing
belladoreai /llama-tokenizer-js
JS tokenizer for LLaMA 1 and 2
$ git clone https://github.com/belladoreai/llama-tokenizer-js.gitStars
363
363
Forks
23
23
Language
JavaScript
License
MIT
Created
Jun 11, 2023
3.3 years old
Last push
Jun 27, 2024
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
Chevrotain/chevrotain🔥 active
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
no-context/moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
JavaScriptBSD-3-Clauseupdated Feb 27, 2026
Tokenization & Preprocessing
glayzzle/php-parser
:herb: NodeJS PHP Parser - extract AST or tokens
JavaScriptBSD-3-Clauseupdated Sep 10, 2026
Tokenization & Preprocessing
Data from GitHub · snapshot Sep 24, 2026