Tokenization & Preprocessing
botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
TypeScriptMITupdated Jan 27, 2023
🏆 #725 overall#11 of 148 in Tokenization & Preprocessing
dqbd /tiktokenizer
Online playground for OpenAPI tokenizers
$ git clone https://github.com/dqbd/tiktokenizer.gitStars
1.7K
1,697
Forks
187
187
Language
TypeScript
License
MIT
Created
Mar 2, 2023
3.6 years old
Last push
Apr 24, 2025
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
RustMITupdated Jul 1, 2026
Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Tokenization & Preprocessing
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
C#MITupdated Sep 10, 2026
Tokenization & Preprocessing
johannschopplich/tokenx
📐 Fast token estimation with 95%+ average accuracy in a 2kB bundle
TypeScriptMITupdated Sep 3, 2026
Tokenization & Preprocessing
Data from GitHub · snapshot Sep 24, 2026