Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
🏆 #2,949 overall#134 of 148 in Tokenization & Preprocessing🔥 active this week
Apil-Shrestha /token-efficiency-lex
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
$ git clone https://github.com/Apil-Shrestha/token-efficiency-lex.gitStars
116
116
Forks
0
0
Language
HTML
License
None
Created
Jun 28, 2026
0.2 years old
Last push
Sep 24, 2026
🔥 this week
More in Tokenization & Preprocessing
Tokenization & Preprocessing
AgentOps-AI/tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
🤖 Language Models
PythonMITupdated Sep 5, 2025
Tokenization & Preprocessing
bminixhofer/zett
Code for Zero-Shot Tokenizer Transfer
🤖 Language Models
Pythonno licenseupdated Jan 14, 2025
Tokenization & Preprocessing
adbar/trafilatura🔥 active
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Sep 21, 2026
Tokenization & Preprocessing
cbaziotis/ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
🧠 Natural Language Processing
PythonMITupdated Jun 2, 2025
Tokenization & Preprocessing
yooper/php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
🧠 Natural Language Processing💎 Text Mining
PHPMITupdated Dec 28, 2024
Data from GitHub · snapshot Sep 24, 2026