🏆 #6,367 overall#163 of 178 in Tokenization & Preprocessing🔥 active this week

Apil-Shrestha /token-efficiency-lex

Token Cost Parity: Multilingual LLM Efficiency Analysis 2026

$ git clone https://github.com/Apil-Shrestha/token-efficiency-lex.git
GitHub social preview for Apil-Shrestha/token-efficiency-lex
Stars
116
116
This week
+1 ★
0 this month
Forks
0
0
Language
HTML
License
None
Created
Jun 28, 2026
0.4 years old
Last push
Nov 16, 2026
🔥 this week

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 116▲ +0★ 116

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

AgentOps-AI/tokencost

▲ +1 ★ this week

Easy token price estimates for 400+ LLMs. TokenOps.

🤖 Language Models
PythonMITupdated Sep 5, 2025
GitHub ↗★ 2K⑂ 109
Tokenization & Preprocessing

bminixhofer/zett

Code for Zero-Shot Tokenizer Transfer

🤖 Language Models
Pythonno licenseupdated Jan 14, 2025
GitHub ↗★ 147⑂ 11
Tokenization & Preprocessing

adbar/trafilatura🔥 active

▲ +43 ★ this week

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

🔍 Search & Retrieval🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Nov 16, 2026
GitHub ↗★ 7.1K⑂ 510
Tokenization & Preprocessing

JohnSnowLabs/spark-nlp🔥 active

▲ +4 ★ this week

State of the Art Natural Language Processing

🧠 Natural Language Processing🏷️ Named Entity Recognition💖 Sentiment Analysis🗂️ Text Classification❓ Question Answering🌍 Machine Translation
ScalaApache-2.0updated Nov 14, 2026
GitHub ↗★ 4.2K⑂ 743
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91

Data from GitHub · snapshot Nov 16, 2026