🏆 #6,265 overall#157 of 178 in Tokenization & Preprocessing

belladoreai /llama3-tokenizer-js

JS tokenizer for LLaMA 3 and LLaMA 3.1

$ git clone https://github.com/belladoreai/llama3-tokenizer-js.git
GitHub social preview for belladoreai/llama3-tokenizer-js
Stars
119
119
This week
0 ★
0 this month
Forks
5
5
Language
JavaScript
License
MIT
Created
Apr 19, 2024
2.6 years old
Last push
Jul 28, 2025

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 119▲ +0★ 119

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

artitw/text2text

▲ +1 ★ this week

Text2Text Language Modeling Toolkit

🔍 Search & Retrieval🧠 Natural Language Processing
PythonOtherupdated Jan 14, 2025
GitHub ↗★ 306⑂ 40
Tokenization & Preprocessing

microsoft/Tokenizer

▲ +1 ★ this week

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

C#MITupdated Oct 19, 2026
GitHub ↗★ 215⑂ 36
Tokenization & Preprocessing

toon-format/toon🔥 active

▲ +15 ★ this week

🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

TypeScriptMITupdated Nov 11, 2026
GitHub ↗★ 25.6K⑂ 1.1K
Tokenization & Preprocessing

adbar/trafilatura🔥 active

▲ +43 ★ this week

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

🔍 Search & Retrieval🧠 Natural Language Processing💎 Text Mining
PythonApache-2.0updated Nov 16, 2026
GitHub ↗★ 7.1K⑂ 510

Data from GitHub · snapshot Nov 16, 2026