marcelroed/gigatoken
▲ +2 ★ this week
Language model tokenization at GB/s
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
$ git clone https://github.com/Apil-Shrestha/token-efficiency-lex.git▲ +2 ★ this week
Language model tokenization at GB/s
▲ +1 ★ this week
Easy token price estimates for 400+ LLMs. TokenOps.
Code for Zero-Shot Tokenizer Transfer
▲ +43 ★ this week
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
▲ +4 ★ this week
State of the Art Natural Language Processing
▲ +1 ★ this week
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
Data from GitHub · snapshot Nov 16, 2026