Tokenization & Preprocessing
botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
TypeScriptMITupdated Jan 27, 2023
🏆 #3,080 overall#145 of 148 in Tokenization & Preprocessing
dluc /openai-tools
A collection of tools for working with OpenAI
$ git clone https://github.com/dluc/openai-tools.gitStars
102
102
Forks
14
14
Language
C#
License
CC0-1.0
Created
Oct 17, 2022
3.9 years old
Last push
May 4, 2023
Categories
GitHub topics
More in Tokenization & Preprocessing
Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
🤖 Language Models
PythonMITupdated Sep 5, 2025
Tokenization & Preprocessing
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🧠 Natural Language Processing
PythonMITupdated Jun 2, 2025
Data from GitHub · snapshot Sep 24, 2026