πŸ† #213 overall#61 of 2,559 in Natural Language ProcessingπŸ”₯ active this week

huggingface /tokenizers

πŸ’₯ Fast State-of-the-Art Tokenizers optimized for Research and Production

$ git clone https://github.com/huggingface/tokenizers.git
GitHub social preview for huggingface/tokenizers
Stars
11.2K
11,249
This week
+21 β˜…
+68 this month
Forks
1.3K
1,261
Language
Rust
License
Apache-2.0
Created
Nov 1, 2019
7.0 years old
Last push
Nov 16, 2026
πŸ”₯ this week

Stars, last 30 days

Oct 15, 2026 β†’ Nov 16, 2026
β˜… 11.2Kβ–² +78β˜… 11.2K

Categories

GitHub topics

More in Natural Language Processing

Natural Language Processing

explosion/spacy-transformers

πŸ›Έ Use pretrained transformers like BERT, XLNet and GPT-2 in spaCy

πŸ€– Language Models
PythonMITupdated Nov 3, 2026
GitHub β†—β˜… 1.4Kβ‘‚ 178
Natural Language Processing

ashishpatel26/Treasure-of-Transformers

πŸ’ Awesome Treasure of Transformers Models for Natural Language processing contains papers, videos, blogs, official repo along with colab Notebooks. πŸ›«β˜‘οΈ

πŸŽ™οΈ Speech RecognitionπŸ€– Language Models
Jupyter NotebookMITupdated Aug 1, 2025
GitHub β†—β˜… 1.2Kβ‘‚ 240
Natural Language Processing

SKTBrain/KoBERT

β–² +1 β˜… this week

Korean BERT pre-trained cased (KoBERT)

πŸ€– Language Models
PythonApache-2.0updated Jun 14, 2025
GitHub β†—β˜… 1.4Kβ‘‚ 374
Natural Language Processing

ymcui/MacBERT

β–² +1 β˜… this week

Revisiting Pre-trained Models for Chinese Natural Language Processing (MacBERT)

πŸ€– Language Models
OtherApache-2.0updated Apr 19, 2026
GitHub β†—β˜… 728β‘‚ 61
Natural Language Processing

charles9n/bert-sklearn

a sklearn wrapper for Google's BERT model

🏷️ Named Entity RecognitionπŸ€– Language Models
Jupyter NotebookApache-2.0updated Oct 26, 2022
GitHub β†—β˜… 302β‘‚ 70
Natural Language Processing

extreme-bert/extreme-bert

ExtremeBERT is a toolkit that accelerates the pretraining of customized language models on customized datasets, described in the paper β€œExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT”.

πŸ€– Language Models
PythonApache-2.0updated Mar 5, 2023
GitHub β†—β˜… 268β‘‚ 14

Data from GitHub Β· snapshot Nov 16, 2026