🏆 #490 overall#8 of 178 in Tokenization & Preprocessing

marcelroed /gigatoken

Language model tokenization at GB/s

$ git clone https://github.com/marcelroed/gigatoken.git
GitHub social preview for marcelroed/gigatoken
Stars
4.2K
4,155
This week
+2 ★
+27 this month
Forks
225
225
Language
Rust
License
MIT
Created
Nov 10, 2025
1.0 years old
Last push
Nov 2, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 4.1K▲ +31★ 4.2K

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91
Tokenization & Preprocessing

daac-tools/vibrato

▲ +2 ★ this week

🎤 vibrato: Viterbi-based accelerated tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Oct 25, 2026
GitHub ↗★ 430⑂ 27
Tokenization & Preprocessing

daac-tools/vaporetto

▲ +2 ★ this week

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

🧠 Natural Language Processing
RustApache-2.0updated Nov 3, 2026
GitHub ↗★ 302⑂ 12
Tokenization & Preprocessing

clipperhouse/uax29

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

🧠 Natural Language Processing
GoMITupdated Feb 16, 2026
GitHub ↗★ 125⑂ 8
Tokenization & Preprocessing

Apil-Shrestha/token-efficiency-lex🔥 active

▲ +1 ★ this week

Token Cost Parity: Multilingual LLM Efficiency Analysis 2026

🧠 Natural Language Processing
HTMLno licenseupdated Nov 16, 2026
GitHub ↗★ 116⑂ 0
Tokenization & Preprocessing

adbar/trafilatura🔥 active

▲ +43 ★ this week

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

🔍 Search & Retrieval🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Nov 16, 2026
GitHub ↗★ 7.1K⑂ 510

Data from GitHub · snapshot Nov 16, 2026