🏆 #2,622 overall#929 of 2,559 in Natural Language Processing

NLPOptimize /flash-tokenizer

EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING

$ git clone https://github.com/NLPOptimize/flash-tokenizer.git
GitHub social preview for NLPOptimize/flash-tokenizer
Stars
458
458
This week
+1 ★
0 this month
Forks
11
11
Language
C++
License
None
Created
Mar 9, 2025
1.7 years old
Last push
Feb 2, 2026

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 458▲ +0★ 458

Categories

GitHub topics

More in Natural Language Processing

Natural Language Processing

candle-org/MindAct

▲ +1 ★ this week

MindSpore + 🤗Huggingface: Run any Transformers/Diffusers model on MindSpore with seamless compatibility and acceleration.

🤖 Language Models
PythonApache-2.0updated Sep 1, 2026
GitHub ↗★ 921⑂ 269
Natural Language Processing

EricFillion/happy-transformer

▲ +1 ★ this week

Happy Transformer makes it easy to fine-tune and perform inference with NLP Transformer models.

🗂️ Text Classification❓ Question Answering
PythonApache-2.0updated Jan 10, 2026
GitHub ↗★ 548⑂ 69
Natural Language Processing

nlpodyssey/cybertron

▲ +1 ★ this week

Cybertron: the home planet of the Transformers in Go

🏷️ Named Entity Recognition📝 Text Summarization🗂️ Text Classification❓ Question Answering🌍 Machine Translation🔍 Search & Retrieval
GoBSD-2-Clauseupdated Jun 8, 2024
GitHub ↗★ 332⑂ 26
Natural Language Processing

extreme-bert/extreme-bert

ExtremeBERT is a toolkit that accelerates the pretraining of customized language models on customized datasets, described in the paper “ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT”.

🤖 Language Models
PythonApache-2.0updated Mar 5, 2023
GitHub ↗★ 268⑂ 14
Natural Language Processing

natasha/slovnet

Deep Learning based NLP modeling for Russian language

🏷️ Named Entity Recognition
PythonMITupdated Jul 24, 2023
GitHub ↗★ 255⑂ 23

Data from GitHub · snapshot Nov 16, 2026