🏆 #4,828 overall#118 of 178 in Tokenization & Preprocessing

gautierdag /bpeasy

Fast bare-bones BPE for modern tokenizer training

$ git clone https://github.com/gautierdag/bpeasy.git
GitHub social preview for gautierdag/bpeasy
Stars
181
181
This week
+1 ★
+1 this month
Forks
6
6
Language
Python
License
MIT
Created
Nov 9, 2023
3.0 years old
Last push
Jun 23, 2025

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 180▲ +1★ 181

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

OpenNMT/Tokenizer

▲ +1 ★ this week

Fast and customizable text tokenization library with BPE and SentencePiece support

🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
GitHub ↗★ 344⑂ 83
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91
Tokenization & Preprocessing

alasdairforsythe/tokenmonster

▲ +3 ★ this week

Ungreedy subword tokenizer and vocabulary trainer for Python, Go, C++ & Javascript

GoMITupdated Jul 10, 2026
GitHub ↗★ 627⑂ 23
Tokenization & Preprocessing

daac-tools/vibrato

▲ +2 ★ this week

🎤 vibrato: Viterbi-based accelerated tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub ↗★ 430⑂ 27
Tokenization & Preprocessing

zurawiki/tiktoken-rs

▲ +2 ★ this week

Ready-made tokenizer library for working with GPT and tiktoken

RustMITupdated Sep 24, 2026
GitHub ↗★ 412⑂ 76

Data from GitHub · snapshot Nov 16, 2026