🏆 #6,893 overall#175 of 178 in Tokenization & Preprocessing

dluc /openai-tools

A collection of tools for working with OpenAI

$ git clone https://github.com/dluc/openai-tools.git
GitHub social preview for dluc/openai-tools
Stars
102
102
This week
0 ★
0 this month
Forks
14
14
Language
C#
License
CC0-1.0
Created
Oct 17, 2022
4.1 years old
Last push
May 4, 2023

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 102▲ +0★ 102

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

botisan-ai/gpt3-tokenizer

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.

TypeScriptMITupdated Jan 27, 2023
GitHub ↗★ 171⑂ 16
Tokenization & Preprocessing

dmitry-brazhenko/SharpToken

▲ +1 ★ this week

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

C#MITupdated Mar 25, 2026
GitHub ↗★ 264⑂ 19
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

AgentOps-AI/tokencost

▲ +1 ★ this week

Easy token price estimates for 400+ LLMs. TokenOps.

🤖 Language Models
PythonMITupdated Sep 5, 2025
GitHub ↗★ 2K⑂ 109
Tokenization & Preprocessing

dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScriptMITupdated Apr 24, 2025
GitHub ↗★ 1.7K⑂ 188
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91

Data from GitHub · snapshot Nov 16, 2026