🏆 #5,012 overall#122 of 178 in Tokenization & Preprocessing

botisan-ai /gpt3-tokenizer

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.

$ git clone https://github.com/botisan-ai/gpt3-tokenizer.git
GitHub social preview for botisan-ai/gpt3-tokenizer
Stars
171
171
This week
0 ★
0 this month
Forks
16
16
Language
TypeScript
License
MIT
Created
Dec 27, 2021
4.9 years old
Last push
Jan 27, 2023

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 171▲ +0★ 171

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

dluc/openai-tools

A collection of tools for working with OpenAI

C#CC0-1.0updated May 4, 2023
GitHub ↗★ 102⑂ 14
Tokenization & Preprocessing

Chevrotain/chevrotain🔥 active

▲ +2 ★ this week

Parser Building Toolkit for JavaScript

TypeScriptApache-2.0updated Nov 14, 2026
GitHub ↗★ 2.8K⑂ 222
Tokenization & Preprocessing

dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScriptMITupdated Apr 24, 2025
GitHub ↗★ 1.7K⑂ 188
Tokenization & Preprocessing

tlaceby/guide-to-interpreters-series

Contains source-code for viewers following along with my Beginners Guide To Building Interpreters series on my Youtube Channel.

TypeScriptno licenseupdated May 2, 2024
GitHub ↗★ 286⑂ 37
Tokenization & Preprocessing

dmitry-brazhenko/SharpToken

▲ +1 ★ this week

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

C#MITupdated Mar 25, 2026
GitHub ↗★ 264⑂ 19
Tokenization & Preprocessing

no-context/moo

▲ +2 ★ this week

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

JavaScriptBSD-3-Clauseupdated Feb 27, 2026
GitHub ↗★ 886⑂ 74

Data from GitHub · snapshot Nov 16, 2026