Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
🧠 Natural Language Processing
RustMITupdated Sep 2, 2026
🏆 #209 overall#4 of 148 in Tokenization & Preprocessing🔥 active this week
adbar /trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
$ git clone https://github.com/adbar/trafilatura.gitStars
6.9K
6,860
Forks
436
436
Language
Python
License
Apache-2.0
Created
Apr 8, 2019
7.5 years old
Last push
Sep 21, 2026
🔥 this week
More in Tokenization & Preprocessing
Tokenization & Preprocessing
artitw/text2text
Text2Text Language Modeling Toolkit
🔍 Search & Retrieval🧠 Natural Language Processing
PythonOtherupdated Jan 14, 2025
Tokenization & Preprocessing
adbar/simplemma🔥 active
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
🧠 Natural Language Processing
PythonMITupdated Sep 21, 2026
Tokenization & Preprocessing
gil-son/language-ai-engineering-lab
Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.
🧠 Natural Language Processing
Jupyter NotebookGPL-3.0updated Jun 24, 2026
Tokenization & Preprocessing
Apil-Shrestha/token-efficiency-lex🔥 active
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
🧠 Natural Language Processing
HTMLno licenseupdated Sep 24, 2026
Tokenization & Preprocessing
jbesomi/texthero
Text preprocessing, representation and visualization from zero to hero.
🧬 Embeddings🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Aug 29, 2023
Data from GitHub · snapshot Sep 24, 2026