Tokenization & Preprocessing
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
C#MITupdated Mar 25, 2026
🏆 #1,170 overall#25 of 148 in Tokenization & Preprocessing
niieani /gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
$ git clone https://github.com/niieani/gpt-tokenizer.gitStars
849
849
Forks
59
59
Language
TypeScript
License
MIT
Created
Mar 22, 2023
3.5 years old
Last push
Aug 16, 2026
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
explosion/spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
🏷️ Named Entity Recognition🗂️ Text Classification🧠 Natural Language Processing
PythonMITupdated Aug 24, 2026
Tokenization & Preprocessing
jbesomi/texthero
Text preprocessing, representation and visualization from zero to hero.
🧬 Embeddings🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Aug 29, 2023
Tokenization & Preprocessing
explosion/spacy-streamlit
👑 spaCy building blocks and visualizers for Streamlit apps
🏷️ Named Entity Recognition🗂️ Text Classification🧠 Natural Language Processing
PythonMITupdated Jul 29, 2024
Tokenization & Preprocessing
nlp-uoregon/trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
🤖 Language Models🧠 Natural Language Processing
PythonApache-2.0updated Jul 22, 2025
Tokenization & Preprocessing
zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
RustMITupdated Jul 1, 2026
Data from GitHub · snapshot Sep 24, 2026