Tokenization & Preprocessing
🏆 #5,012 overall#122 of 178 in Tokenization & Preprocessing
botisan-ai /gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
$ git clone https://github.com/botisan-ai/gpt3-tokenizer.gitStars
171
171
This week
0 ★
0 this month
Forks
16
16
Language
TypeScript
License
MIT
Created
Dec 27, 2021
4.9 years old
Last push
Jan 27, 2023
Stars, last 30 days
Oct 15, 2026 → Nov 16, 2026Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Chevrotain/chevrotain🔥 active
▲ +2 ★ this week
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Nov 14, 2026
Tokenization & Preprocessing
Tokenization & Preprocessing
tlaceby/guide-to-interpreters-series
Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.
TypeScriptno licenseupdated May 2, 2024
Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Tokenization & Preprocessing
no-context/moo
▲ +2 ★ this week
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
JavaScriptBSD-3-Clauseupdated Feb 27, 2026
Data from GitHub · snapshot Nov 16, 2026