dmitry-brazhenko/SharpToken
▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
$ git clone https://github.com/niieani/gpt-tokenizer.git▲ +1 ★ this week
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
▲ +24 ★ this week
💫 Industrial-strength Natural Language Processing (NLP) in Python
Text preprocessing, representation and visualization from zero to hero.
👑 spaCy building blocks and visualizers for Streamlit apps
▲ +1 ★ this week
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
▲ +2 ★ this week
Ready-made tokenizer library for working with GPT and tiktoken
Data from GitHub · snapshot Nov 16, 2026