🏆 #6,297 overall#160 of 178 in Tokenization & Preprocessing

taishan1994 /sentencepiece_chinese_bpe

使用sentencepiece中BPE训练中文词表,并在transformers中进行使用。

$ git clone https://github.com/taishan1994/sentencepiece_chinese_bpe.git
GitHub social preview for taishan1994/sentencepiece_chinese_bpe
Stars
118
118
This week
0 ★
0 this month
Forks
17
17
Language
Python
License
None
Created
Jun 24, 2023
3.4 years old
Last push
Jun 24, 2023

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 118▲ +0★ 118

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

OpenNMT/Tokenizer

▲ +1 ★ this week

Fast and customizable text tokenization library with BPE and SentencePiece support

🌍 Machine Translation🧠 Natural Language Processing
C++MITupdated Jan 10, 2026
GitHub ↗★ 344⑂ 83
Tokenization & Preprocessing

toon-format/toon🔥 active

▲ +15 ★ this week

🎒 Token-Oriented Object Notation (TOON) – compact, human-readable serialization of JSON data for LLM prompts. TypeScript SDK, CLI, benchmarks.

TypeScriptMITupdated Nov 11, 2026
GitHub ↗★ 25.6K⑂ 1.1K
Tokenization & Preprocessing

marcelroed/gigatoken

▲ +2 ★ this week

Language model tokenization at GB/s

🧠 Natural Language Processing
RustMITupdated Nov 2, 2026
GitHub ↗★ 4.2K⑂ 225
Tokenization & Preprocessing

AgentOps-AI/tokencost

▲ +1 ★ this week

Easy token price estimates for 400+ LLMs. TokenOps.

🤖 Language Models
PythonMITupdated Sep 5, 2025
GitHub ↗★ 2K⑂ 109
Tokenization & Preprocessing

RavenProject/Ravencoin

▲ +1 ★ this week

Ravencoin Core integration/staging tree

CMITupdated Aug 18, 2026
GitHub ↗★ 1.1K⑂ 727
Tokenization & Preprocessing

cbaziotis/ekphrasis

▲ +1 ★ this week

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Jun 2, 2025
GitHub ↗★ 672⑂ 91

Data from GitHub · snapshot Nov 16, 2026