botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
A collection of tools for working with OpenAI
$ git clone https://github.com/dluc/openai-tools.gitIsomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
▲ +2 ★ this week
Language model tokenization at GB/s
▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
▲ +1 ★ this week
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Data from GitHub · snapshot Nov 16, 2026