🏆 #1,738 overall#33 of 178 in Tokenization & Preprocessing

nlp-uoregon /trankit

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

$ git clone https://github.com/nlp-uoregon/trankit.git
GitHub social preview for nlp-uoregon/trankit
Stars
797
797
This week
+1 ★
-1 this month
Forks
107
107
Language
Python
License
Apache-2.0
Created
Jan 8, 2021
5.9 years old
Last push
Jul 22, 2025

Stars, last 30 days

Oct 15, 2026 → Nov 16, 2026
★ 798▼ 1★ 797

Categories

GitHub topics

More in Tokenization & Preprocessing

Tokenization & Preprocessing

huspacy/huspacy🔥 active

HuSpaCy: industrial-strength Hungarian natural language processing

🧠 Natural Language Processing🏷️ Named Entity Recognition⛏️ Information Extraction💎 Text Mining
PythonApache-2.0updated Nov 16, 2026
GitHub ↗★ 194⑂ 17
Tokenization & Preprocessing

jshuadvd/LongRoPE

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

🧠 Natural Language Processing🤖 Language Models
Pythonno licenseupdated Jul 20, 2024
GitHub ↗★ 154⑂ 13
Tokenization & Preprocessing

explosion/spaCy

▲ +24 ★ this week

💫 Industrial-strength Natural Language Processing (NLP) in Python

🧠 Natural Language Processing🏷️ Named Entity Recognition🗂️ Text Classification
PythonMITupdated Oct 27, 2026
GitHub ↗★ 34.1K⑂ 4.7K
Tokenization & Preprocessing

explosion/spacy-streamlit

👑 spaCy building blocks and visualizers for Streamlit apps

🧠 Natural Language Processing🏷️ Named Entity Recognition🗂️ Text Classification
PythonMITupdated Jul 29, 2024
GitHub ↗★ 864⑂ 111
Tokenization & Preprocessing

CogComp/cogcomp-nlp

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.

🏷️ Named Entity Recognition🧠 Natural Language Processing⛏️ Information Extraction
JavaOtherupdated Jul 7, 2023
GitHub ↗★ 479⑂ 143
Tokenization & Preprocessing

janlukasschroeder/nlp-cheat-sheet-python

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition

🧠 Natural Language Processing🏷️ Named Entity Recognition
Jupyter Notebookno licenseupdated Feb 11, 2023
GitHub ↗★ 261⑂ 77

Data from GitHub · snapshot Nov 16, 2026