🏆 #4,152 overall#100 of 178 in Tokenization & Preprocessing

daulet /tokenizers

Go, Wasm bindings for HF Tokenizers and Tiktoken

$ git clone https://github.com/daulet/tokenizers.git
GitHub social preview for daulet/tokenizers
Stars
229
229
This week
0 ★
+1 this month
Forks
43
43
Language
Rust
License
MIT
Created
Mar 31, 2023
3.6 years old
Last push
Mar 30, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 228▲ +1★ 229

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

clipperhouse/uax29

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.

🧠 Natural Language Processing
GoMITupdated Feb 16, 2026
GitHub ↗★ 125⑂ 8
Tokenization & Preprocessing

clipperhouse/jargon

Tokenizers and lemmatizers for Go

🧠 Natural Language Processing
GoMITupdated Sep 2, 2025
GitHub ↗★ 120⑂ 4
Tokenization & Preprocessing

theseer/tokenizer

▲ +2 ★ this week

A small library for converting tokenized PHP source code into XML (and potentially other formats)

PHPOtherupdated Feb 3, 2026
GitHub ↗★ 5.2K⑂ 25
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

andialbrecht/sqlparse

A non-validating SQL parser module for Python

PythonBSD-3-Clauseupdated Nov 1, 2026
GitHub ↗★ 4K⑂ 781

Data from GitHub · snapshot Nov 16, 2026