Tokenization & Preprocessing
zhenye234/xcodec
AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
๐ค Language Models
PythonMITupdated Oct 12, 2025
๐ #2,179 overall#76 of 148 in Tokenization & Preprocessing
OpenMOSS /MOSS-Audio-Tokenizer
A 1.6B causal Transformer audio tokenizer with streaming, variable bitrates, and semantic alignment across speech, sound, and music
$ git clone https://github.com/OpenMOSS/MOSS-Audio-Tokenizer.gitStars
258
258
Forks
19
19
Language
Python
License
Apache-2.0
Created
Feb 3, 2026
0.6 years old
Last push
Jun 16, 2026
Categories
More in Tokenization & Preprocessing
Tokenization & Preprocessing
Voine/Bert-VITS2-MNN
A practical on-device BertVITS2 runtime for Android, powered by alibaba-MNN engine. ้ขๅ็งปๅจ็ซฏ็ฆป็บฟ่ฏญ้ณๅๆ็่ฝป้ๅ BertVITS2 ๅทฅ็จๅฎ็ฐ๏ผๅบไบ้ฟ้ MNN ๆจ็ๆกๆถใ
C++Apache-2.0updated Aug 5, 2026
Tokenization & Preprocessing
frothywater/kanade-tokenizer
Kanade is a single-layer disentangled speech tokenizer that extracts compact tokens suitable for both generative and discriminative modeling.
Pythonno licenseupdated Jul 18, 2026
Tokenization & Preprocessing
theseer/tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
PHPOtherupdated Feb 3, 2026
Tokenization & Preprocessing
marcelroed/gigatoken
Language model tokenization at GB/s
๐ง Natural Language Processing
RustMITupdated Sep 2, 2026
Tokenization & Preprocessing
andialbrecht/sqlparse
A non-validating SQL parser module for Python
PythonBSD-3-Clauseupdated Aug 13, 2026
Data from GitHub ยท snapshot Sep 24, 2026