🏆 #3,847 overall#95 of 178 in Tokenization & Preprocessing🔥 active this week

snchimata /tokenfold

Private, provider-neutral context compression for LLMs. Reversibly compress schemas and logs inside your perimeter with zero third-party calls.

$ git clone https://github.com/snchimata/tokenfold.git
GitHub social preview for snchimata/tokenfold
Stars
259
259
This week
+158 ★
+157 this month
Forks
30
30
Language
Rust
License
Apache-2.0
Created
Jul 14, 2026
0.3 years old
Last push
Nov 12, 2026
🔥 this week

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 102▲ +157★ 259

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

AgentOps-AI/tokencost

▲ +1 ★ this week

Easy token price estimates for 400+ LLMs. TokenOps.

🤖 Language Models
PythonMITupdated Sep 5, 2025
GitHub ↗★ 2K⑂ 109
Tokenization & Preprocessing

microsoft/Tokenizer

▲ +1 ★ this week

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

C#MITupdated Oct 19, 2026
GitHub ↗★ 215⑂ 36
Tokenization & Preprocessing

toon-format/toon🔥 active

▲ +15 ★ this week

🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

TypeScriptMITupdated Nov 11, 2026
GitHub ↗★ 25.6K⑂ 1.1K
Tokenization & Preprocessing

adbar/trafilatura🔥 active

▲ +43 ★ this week

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

🔍 Search & Retrieval🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Nov 16, 2026
GitHub ↗★ 7.1K⑂ 510
Tokenization & Preprocessing

crate-ci/typos🔥 active

Source code spell checker

RustApache-2.0updated Nov 16, 2026
GitHub ↗★ 4.2K⑂ 192
Tokenization & Preprocessing

JohnSnowLabs/spark-nlp🔥 active

▲ +4 ★ this week

State of the Art Natural Language Processing

🧠 Natural Language Processing🏷️ Named Entity Recognition💖 Sentiment Analysis🗂️ Text Classification❓ Question Answering🌍 Machine Translation
ScalaApache-2.0updated Nov 14, 2026
GitHub ↗★ 4.2K⑂ 743

Data from GitHub · snapshot Nov 16, 2026