Tokenization & Preprocessing
rasbt/LLMs-from-scratch๐ฅ active
โฒ +441 โ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
๐ค Language Models๐ง Natural Language Processing
Jupyter NotebookOtherupdated Nov 9, 2026
๐ฅ Trending this week in Tokenization & Preprocessing
Tokenization & Preprocessing
adbar/trafilatura๐ฅ active
โฒ +43 โ this week
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
๐ Search & Retrieval๐ง Natural Language Processing๐ Text Mining
PythonApache-2.0updated Nov 11, 2026
Tokenization & Preprocessing
snchimata/tokenfold๐ฅ active
โฒ +31 โ this week
Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.
RustApache-2.0updated Nov 12, 2026
Tokenization & Preprocessing
explosion/spaCy
โฒ +21 โ this week
๐ซ Industrial-strength Natural Language Processing (NLP) in Python
๐ง Natural Language Processing๐ท๏ธ Named Entity Recognition๐๏ธ Text Classification
PythonMITupdated Oct 27, 2026
Tokenization & Preprocessing
toon-format/toon๐ฅ active
โฒ +18 โ this week
๐ Token-Oriented Object Notation (TOON) โ compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
TypeScriptMITupdated Nov 11, 2026
Tokenization & Preprocessing
myylogic/cevahir-ai
โฒ +9 โ this week
Full-stack open-source AI engine for building language models โ tokenizer training, transformer architecture, cognitive reasoning and chat pipeline.
๐ Search & Retrieval๐ค Language Models๐ง Natural Language Processing
PythonApache-2.0updated Oct 13, 2026
Top 30 tokenizers and text preprocessing tools by stars
Tokenization & Preprocessing๐ฅ#1
rasbt/LLMs-from-scratch๐ฅ active
โฒ +441 โ this week
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
๐ค Language Models๐ง Natural Language Processing
Jupyter NotebookOtherupdated Nov 9, 2026
Tokenization & Preprocessing๐ฅ#2
explosion/spaCy
โฒ +21 โ this week
๐ซ Industrial-strength Natural Language Processing (NLP) in Python
๐ง Natural Language Processing๐ท๏ธ Named Entity Recognition๐๏ธ Text Classification
PythonMITupdated Oct 27, 2026
Tokenization & Preprocessing๐ฅ#3
toon-format/toon๐ฅ active
โฒ +18 โ this week
๐ Token-Oriented Object Notation (TOON) โ compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.
TypeScriptMITupdated Nov 11, 2026
Tokenization & Preprocessing#4
adbar/trafilatura๐ฅ active
โฒ +43 โ this week
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
๐ Search & Retrieval๐ง Natural Language Processing๐ Text Mining
PythonApache-2.0updated Nov 11, 2026
Tokenization & Preprocessing#5
theseer/tokenizer
โฒ +1 โ this week
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Tokenization & Preprocessing#6
Tokenization & Preprocessing#7
JohnSnowLabs/spark-nlp๐ฅ active
โฒ +3 โ this week
State of the Art Natural Language Processing
๐ง Natural Language Processing๐ท๏ธ Named Entity Recognition๐ Sentiment Analysis๐๏ธ Text Classificationโ Question Answering๐ Machine Translation
ScalaApache-2.0updated Nov 12, 2026
Tokenization & Preprocessing#8
marcelroed/gigatoken
โฒ +4 โ this week
Language model tokenization at GB/s
๐ง Natural Language Processing
RustMITupdated Nov 2, 2026
Tokenization & Preprocessing#9
andialbrecht/sqlparse
A non-validating SQL parser module for Python
PythonBSD-3-Clauseupdated Nov 1, 2026
Tokenization & Preprocessing#10
jbesomi/texthero
Text preprocessing, representation and visualization from zero to hero.
๐งฌ Embeddings๐ง Natural Language Processing๐ Text Mining
PythonMITupdated Aug 29, 2023
Tokenization & Preprocessing#11
Chevrotain/chevrotain๐ฅ active
โฒ +1 โ this week
Parser Building Toolkit for JavaScript
TypeScriptApache-2.0updated Nov 12, 2026
Tokenization & Preprocessing#12
hunspell/hunspell๐ฅ active
The most popular spellchecking library.
๐ง Natural Language Processing
C++LGPL-2.1updated Nov 8, 2026
Tokenization & Preprocessing#13
AgentOps-AI/tokencost
โฒ +3 โ this week
Easy token price estimates for 400+ LLMs. TokenOps.
๐ค Language Models
PythonMITupdated Sep 5, 2025
Tokenization & Preprocessing#14
dqbd/tiktokenizer
โฒ +2 โ this week
Online playground for OpenAPI tokenizers
TypeScriptMITupdated Apr 24, 2025
Tokenization & Preprocessing#15
Tokenization & Preprocessing#16
securitybunker/databunker๐ฅ active
โฒ +3 โ this week
Secure Vault for Customer PII/PHI/PCI/KYC Records
GoMITupdated Nov 10, 2026
Tokenization & Preprocessing#17
lunasec-io/lunasec
โฒ +1 โ this week
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
TypeScriptOtherupdated May 2, 2024
Tokenization & Preprocessing#18
roshan-research/hazm
โฒ +1 โ this week
Persian NLP Toolkit
๐ง Natural Language Processing๐ Text Mining
PythonMITupdated Apr 1, 2026
Tokenization & Preprocessing#19
natasha/natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
๐ท๏ธ Named Entity Recognition๐ง Natural Language Processing
PythonMITupdated Oct 14, 2026
Tokenization & Preprocessing#20
SeanLee97/xmnlp
xmnlp๏ผๆไพไธญๆๅ่ฏ, ่ฏๆงๆ ๆณจ, ๅฝๅไฝ่ฏๅซ๏ผๆ ๆๅๆ๏ผๆๆฌ็บ ้๏ผๆๆฌ่ฝฌๆผ้ณ๏ผๆๆฌๆ่ฆ๏ผๅๆ้จ้ฆ๏ผๅฅๅญ่กจๅพๅๆๆฌ็ธไผผๅบฆ่ฎก็ฎ็ญๅ่ฝ
๐ Sentiment Analysis๐ท๏ธ Named Entity Recognition๐งฌ Embeddings๐ง Natural Language Processing
PythonApache-2.0updated Nov 12, 2022
Tokenization & Preprocessing#21
uhop/stream-json๐ฅ active
A micro-library of stream components for building custom JSON and JSONC processing pipelines with a minimal memory footprint โ parse, filter, and transform JSON far larger than available memory with a SAX-inspired token API, on Node.js or Web Streams.
JavaScriptBSD-3-Clauseupdated Nov 9, 2026
Tokenization & Preprocessing#22
RavenProject/Ravencoin
โฒ +1 โ this week
Ravencoin Core integration/staging tree
CMITupdated Aug 18, 2026
Tokenization & Preprocessing#23
sylvainhalle/textidote
Spelling, grammar and style checking on LaTeX documents
JavaGPL-3.0updated Sep 9, 2026
Tokenization & Preprocessing#24
jfilter/clean-text
โฒ +2 โ this week
๐งน Python package for text cleaning
๐ง Natural Language Processing
PythonOtherupdated Oct 26, 2026
Tokenization & Preprocessing#25
lovit/soynlp
ํ๊ตญ์ด ์์ฐ์ด์ฒ๋ฆฌ๋ฅผ ์ํ ํ์ด์ฌ ๋ผ์ด๋ธ๋ฌ๋ฆฌ์ ๋๋ค. ๋จ์ด ์ถ์ถ/ ํ ํฌ๋์ด์ / ํ์ฌํ๋ณ/ ์ ์ฒ๋ฆฌ์ ๊ธฐ๋ฅ์ ์ ๊ณตํฉ๋๋ค.
๐ง Natural Language Processing
PythonOtherupdated Oct 4, 2026
Tokenization & Preprocessing#26
ikawaha/kagome๐ฅ active
Self-contained Japanese Morphological Analyzer written in pure Go
๐ง Natural Language Processing
GoMITupdated Nov 11, 2026
Tokenization & Preprocessing#27
JohnSnowLabs/nlu
1 line for thousands of State of The Art NLP models in hundreds of languages The fastest and most accurate way to solve text problems.
๐ Sentiment Analysis๐ท๏ธ Named Entity Recognition๐ Text Summarization๐๏ธ Text Classification๐งฌ Embeddings
PythonOtherupdated Jan 28, 2025
Tokenization & Preprocessing#28
AmoDinho/datacamp-python-data-science-track
โฒ +2 โ this week
All the slides, accompanying code and exercises all stored in this repo. ๐
๐ง Natural Language Processing
PythonMITupdated Jul 17, 2023
Tokenization & Preprocessing#29
no-context/moo
โฒ +2 โ this week
Optimised tokenizer/lexer generator! ๐ Uses /y for performance. Moo.
JavaScriptBSD-3-Clauseupdated Feb 27, 2026
Tokenization & Preprocessing#30
wangfenjin/simple
โฒ +1 โ this week
ๆฏๆไธญๆๅๆผ้ณ็ SQLite fts5 ๅ จๆๆ็ดขๆฉๅฑ ๏ฝ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
C++Otherupdated May 17, 2026
๐ New research on Tokenization & Preprocessing
๐ arXiv:2611.13990 ยท cs.CL2d ago
On the information rate of phones, syllables and words
Herman Kamper
๐ฌ 5 pages, 1 table, 2 figures
Which linguistic units minimise the bitrate required to encode speech content: phones, syllables, words, maybe larger units, or something in between? This question has implications for the development of speech and language systems and theories of human language processing. Using a transcribed corpus of 960 hours of English audiobooks, I compare the information rates of phones, syllables, words, multi-word combinations and intermediate tokenisations. While all units are equivalent under a perfect language model, it makes sense to compare these under limited modelling budgets; I use $n$-gram and transformer language models. The lowest bitrates are achieved using words, with slightly longer units sometimes performing best. The results show that, when modelling capacity is limited, units larger than syllables but smaller than long multi-word phrases provide the most efficient representation in terms of information rate.โผ more
โ๏ธ Tokenization & Preprocessing๐๏ธ Speech Recognition๐ค Language Models
๐ arXiv:2611.13607 ยท cs.CL2d ago
One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data
Liwei Zhang, ZhiMing Tong
๐ฌ Full text in English followed by the full text in Chinese. 39 pages, 2 figures, 7 tables. Code and task suite: https://huggingface.co/datasets/leiwhere/ld3-byte-eval
Byte-level BPE fragments Chinese text (for Qwen3.8 on Chinese Wikipedia, 3.84 bytes per token on average; 64% of tokens are single characters) and hides spelling from the model. We ask whether an already-trained 27B subword model can be converted directly into a pure byte-level model (one byte, one token), without a patching structure or a local encoder, while keeping most of its ability. Starting from Qwen3.8-27B (48 Gated DeltaNet layers + 16 full-attention layers), we shrink the vocabulary from 248,320 to 384 entries (the 256 byte symbols already in the BPE vocabulary plus special ids), leave every other weight untouched, and continue training in a two-stage curriculum (encoding first, then reading) on 4.77 GB of supervised bytes: all dialogue/behavioral supervision is generated on-policy by the original model itself (character dictionary, word dictionary, general dialogue, coding sessions), and knowledge reading uses encyclopedia text (54% of supervised bytes). Training takes 178.6 h on 8xH200. On the same GPU and serving stack, the byte model's gap to the original falls within the paired bootstrap interval on MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K, and it loses 4-6 points on C-Eval, thinking-mode GSM8K and non-thinking IFEval. Without thinking it leads on every character-level task (CUTE 75.2 -> 84.0, EXECUTE-zho 81.6 -> 93.9, our Chinese character suite 54.4 -> 64.0), but lookup knowledge about Chinese characters such as pronunciation and stroke count drops, and thinking does not recover it. The cost is throughput: per-step speed is identical, but each step yields 3-5x less output; Chinese characters per second is about 1/4 of the original, and prefill at 64K bytes is 6.5x slower. We also document four byte-model failure modes (greedy runaway, sampling breaking UTF-8, ineffective penalty terms on a small vocabulary, repetition) and their fixes.โผ more
โ๏ธ Tokenization & Preprocessing๐ค Language Models
๐ arXiv:2611.13568 ยท cs.DS2d ago
Pair-Merge Tokenisation Is Hard to Approximate on Binary Strings and Strongly NP-Hard on Unary Datasets
Kundan Baliga, Aiden Xie, Aditya Byju
๐ฌ 34 pages, 11 figures. Code and certificates: https://doi.org/10.5281/zenodo.24023730
Tokenisers such as Byte-Pair Encoding (BPE) shorten text by repeatedly merging an adjacent pair of symbols into a new one; fewer tokens mean less computation. We ask whether an efficient algorithm can choose $k$ merges that leave a string of length $n$ almost as short as possible; doing nothing is already within a factor $n$. We prove that, unless P = NP, no polynomial-time algorithm comes within a factor $c\,n/\log n$, for some constant $c>0$, already for one binary string and for BPE-style merges that replace every occurrence. The key idea is that $k$ merges form a grammar with $k$ rules: a vertex-cover gap becomes a grammar-size gap, which we amplify with relabelled copies, encode in binary and realise with explicit merges. For merges that may skip occurrences, a simple algorithm achieves $O(n/\log\log n)$, and no Karp reduction can prove hardness beyond order $n/\log n$ unless coNP $\subseteq$ NP/poly. Over a one-letter alphabet, both tokenisation problems left open by Kastreva et al. are strongly NP-complete on datasets with repeated strings.โผ more
โ๏ธ Tokenization & Preprocessing
๐ arXiv:2611.13564 ยท cs.CL2d ago
A Concept-Based Interpretable Framework for Clinical Text-based Prediction via Token-Concept Attention without Concept Supervision
Jifan Gao, Junjie Hu, John Caskey +6
Objective: Concept-based interpretation has emerged as a promising approach for explaining blackbox models using human-understandable, high-level concepts. However, most existing methods require annotated concept labels to train the model to identify concepts in the input, and obtaining such annotations in clinical text is often expensive. This study aims to enable concept-based interpretation for language models without concept supervision on clinical prediction tasks. Methods: We propose a novel framework that enables concept-based interpretation without concept supervision through a tailored cross-attention bottleneck. Clinical text and the text descriptions of medical concepts are first tokenized and embedded by a pretrained language model. A cross-attention module then estimates the presence of each concept for each text token, producing concept relevance scores that capture how concepts are involved throughout the note. These scores are pooled and used in a linear prediction layer to model clinical outcomes in an interpretable manner. Results: The performance of the proposed framework was compared against blackbox models across two tasks: chest trauma severity stratification using a private dataset and an in-hospital mortality prediction task using the publicly available MIMIC-III dataset. The proposed framework achieved AUROC values within 0.02 of the corresponding blackbox models. The framework also enabled cohort- and patient-level interpretation, capturing the overall concept importance across the cohort and patient-specific concept contributions. Conclusion: Our proposed framework realizes concept-based interpretation without requiring ground-truth concept presence labels. It improves the transparency of clinical text predictions while maintaining strong performance in clinical prediction tasks.โผ more
โ๏ธ Tokenization & Preprocessing๐ค Language Models