🏆 #1,630 overall#32 of 178 in Tokenization & Preprocessing🔥 active this week

niieani /gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

$ git clone https://github.com/niieani/gpt-tokenizer.git
GitHub social preview for niieani/gpt-tokenizer
Stars
861
861
This week
+4 ★
+6 this month
Forks
65
65
Language
TypeScript
License
MIT
Created
Mar 22, 2023
3.7 years old
Last push
Nov 10, 2026
🔥 this week

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 854▲ +7★ 861

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

dmitry-brazhenko/SharpToken

▲ +1 ★ this week

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

C#MITupdated Mar 25, 2026
GitHub ↗★ 264⑂ 19
Tokenization & Preprocessing

explosion/spaCy

▲ +24 ★ this week

💫 Industrial-strength Natural Language Processing (NLP) in Python

🧠 Natural Language Processing🏷️ Named Entity Recognition🗂️ Text Classification
PythonMITupdated Oct 27, 2026
GitHub ↗★ 34.1K⑂ 4.7K
Tokenization & Preprocessing

jbesomi/texthero

Text preprocessing, representation and visualization from zero to hero.

🧬 Embeddings🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Aug 29, 2023
GitHub ↗★ 2.9K⑂ 235
Tokenization & Preprocessing

explosion/spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

🧠 Natural Language Processing🏷️ Named Entity Recognition🗂️ Text Classification
PythonMITupdated Jul 29, 2024
GitHub ↗★ 864⑂ 111
Tokenization & Preprocessing

nlp-uoregon/trankit

▲ +1 ★ this week

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

🧠 Natural Language Processing🤖 Language Models
PythonApache-2.0updated Jul 22, 2025
GitHub ↗★ 797⑂ 107
Tokenization & Preprocessing

zurawiki/tiktoken-rs

▲ +2 ★ this week

Ready-made tokenizer library for working with GPT and tiktoken

RustMITupdated Sep 24, 2026
GitHub ↗★ 412⑂ 76

Data from GitHub · snapshot Nov 16, 2026