🏆 #4,319 overall#106 of 178 in Tokenization & Preprocessing

microsoft /Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

$ git clone https://github.com/microsoft/Tokenizer.git
GitHub social preview for microsoft/Tokenizer
Stars
215
215
This week
+1 ★
+3 this month
Forks
36
36
Language
C#
License
MIT
Created
Mar 28, 2023
3.6 years old
Last push
Oct 19, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 212▲ +3★ 215

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

dmitry-brazhenko/SharpToken

▲ +1 ★ this week

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

C#MITupdated Mar 25, 2026
GitHub ↗★ 264⑂ 19
Tokenization & Preprocessing

jshuadvd/LongRoPE

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

🧠 Natural Language Processing🤖 Language Models
Pythonno licenseupdated Jul 20, 2024
GitHub ↗★ 154⑂ 13
Tokenization & Preprocessing

rasbt/LLMs-from-scratch

▲ +499 ★ this week

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

🤖 Language Models🧠 Natural Language Processing
Jupyter NotebookOtherupdated Nov 9, 2026
GitHub ↗★ 108.5K⑂ 16.8K
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

AgentOps-AI/tokencost

▲ +1 ★ this week

Easy token price estimates for 400+ LLMs. TokenOps.

🤖 Language Models
PythonMITupdated Sep 5, 2025
GitHub ↗★ 2K⑂ 109
Tokenization & Preprocessing

dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScriptMITupdated Apr 24, 2025
GitHub ↗★ 1.7K⑂ 188

Data from GitHub · snapshot Nov 16, 2026