Tokenization & Preprocessing
🏆 #2,547 overall#102 of 148 in Tokenization & Preprocessing
botisan-ai /gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
$ git clone https://github.com/botisan-ai/gpt3-tokenizer.gitStars
171
171
Forks
16
16
Language
TypeScript
License
MIT
Created
Dec 27, 2021
4.7 years old
Last push
Jan 27, 2023
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Chevrotain/chevrotain🔥 active
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Sep 24, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
tlaceby/guide-to-interpreters-series
Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.
TypeScriptno licenseupdated May 2, 2024
Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Tokenization & Preprocessing
no-context/moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
JavaScriptBSD-3-Clauseupdated Feb 27, 2026
Data from GitHub · snapshot Sep 24, 2026