๐Ÿ† #4,683 overall#115 of 178 in Tokenization & Preprocessing๐Ÿ”ฅ active this week

johannschopplich /tokenx

๐Ÿ“ Fast token estimation with 95%+ average accuracy in a 2kB bundle

$ git clone https://github.com/johannschopplich/tokenx.git
GitHub social preview for johannschopplich/tokenx
Stars
190
190
This week
+2 โ˜…
+7 this month
Forks
8
8
Language
TypeScript
License
MIT
Created
Nov 27, 2023
3.0 years old
Last push
Nov 14, 2026
๐Ÿ”ฅ this week

Stars, last 30 days

Oct 15, 2026 โ†’ Nov 16, 2026
โ˜… 181โ–ฒ +9โ˜… 190

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

marcelroed/gigatoken

โ–ฒ +2 โ˜… this week

Language model tokenization at GB/s

๐Ÿง  Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub โ†—โ˜… 4.2Kโ‘‚ 225
Tokenization & Preprocessing

dqbd/tiktokenizer

Online playground for OpenAPI tokenizers

TypeScriptMITupdated Apr 24, 2025
GitHub โ†—โ˜… 1.7Kโ‘‚ 188
Tokenization & Preprocessing

cbaziotis/ekphrasis

โ–ฒ +1 โ˜… this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

๐Ÿง  Natural Language Processing๐Ÿ’Ž Text Mining
PythonMITupdated Jun 2, 2025
GitHub โ†—โ˜… 672โ‘‚ 91
Tokenization & Preprocessing

alasdairforsythe/tokenmonster

โ–ฒ +3 โ˜… this week

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

GoMITupdated Jul 10, 2026
GitHub โ†—โ˜… 627โ‘‚ 23
Tokenization & Preprocessing

daac-tools/vibrato

โ–ฒ +2 โ˜… this week

๐ŸŽค vibrato: Viterbi-based accelerated tokenizer

๐Ÿง  Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub โ†—โ˜… 430โ‘‚ 27
Tokenization & Preprocessing

OpenNMT/Tokenizer

โ–ฒ +1 โ˜… this week

Fast and customizable text tokenization library with BPE and SentencePiece support

๐ŸŒ Machine Translation๐Ÿง  Natural Language Processing
C++MITupdated Jan 10, 2026
GitHub โ†—โ˜… 344โ‘‚ 83

Data from GitHub ยท snapshot Nov 16, 2026