🏆 #3,792 overall#93 of 178 in Tokenization & Preprocessing

dmitry-brazhenko /SharpToken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

$ git clone https://github.com/dmitry-brazhenko/SharpToken.git
GitHub social preview for dmitry-brazhenko/SharpToken
Stars
264
264
This week
+1 ★
+1 this month
Forks
19
19
Language
C#
License
MIT
Created
Mar 28, 2023
3.6 years old
Last push
Mar 25, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 263▲ +1★ 264

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

microsoft/Tokenizer

▲ +1 ★ this week

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

C#MITupdated Oct 19, 2026
GitHub ↗★ 215⑂ 36
Tokenization & Preprocessing

botisan-ai/gpt3-tokenizer

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.

TypeScriptMITupdated Jan 27, 2023
GitHub ↗★ 171⑂ 16
Tokenization & Preprocessing

dluc/openai-tools

A collection of tools for working with OpenAI

C#CC0-1.0updated May 4, 2023
GitHub ↗★ 102⑂ 14
Tokenization & Preprocessing

dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScriptMITupdated Apr 24, 2025
GitHub ↗★ 1.7K⑂ 188
Tokenization & Preprocessing

niieani/gpt-tokenizer🔥 active

▲ +4 ★ this week

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

TypeScriptMITupdated Nov 10, 2026
GitHub ↗★ 861⑂ 65
Tokenization & Preprocessing

zurawiki/tiktoken-rs

▲ +2 ★ this week

Ready-made tokenizer library for working with GPT and tiktoken

RustMITupdated Sep 24, 2026
GitHub ↗★ 412⑂ 76

Data from GitHub · snapshot Nov 16, 2026