🏆 #2,806 overall#61 of 178 in Tokenization & Preprocessing

zurawiki /tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

$ git clone https://github.com/zurawiki/tiktoken-rs.git
GitHub social preview for zurawiki/tiktoken-rs
Stars
412
412
This week
+2 ★
+4 this month
Forks
76
76
Language
Rust
License
MIT
Created
Feb 2, 2023
3.8 years old
Last push
Sep 24, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 408▲ +4★ 412

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScriptMITupdated Apr 24, 2025
GitHub ↗★ 1.7K⑂ 188
Tokenization & Preprocessing

daac-tools/vibrato

▲ +2 ★ this week

🎤 vibrato: Viterbi-based accelerated tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub ↗★ 430⑂ 27
Tokenization & Preprocessing

OpenNMT/Tokenizer

▲ +1 ★ this week

Fast and customizable text tokenization library with BPE and SentencePiece support

🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
GitHub ↗★ 344⑂ 83
Tokenization & Preprocessing

daac-tools/vaporetto

▲ +2 ★ this week

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Nov 3, 2026
GitHub ↗★ 302⑂ 12
Tokenization & Preprocessing

dmitry-brazhenko/SharpToken

▲ +1 ★ this week

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

C#MITupdated Mar 25, 2026
GitHub ↗★ 264⑂ 19
Tokenization & Preprocessing

microsoft/Tokenizer

▲ +1 ★ this week

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

C#MITupdated Oct 19, 2026
GitHub ↗★ 215⑂ 36

Data from GitHub · snapshot Nov 16, 2026