Tokenization & Preprocessing
jshuadvd/LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
🤖 Language Models🧠 Natural Language Processing
Pythonno licenseupdated Jul 20, 2024
🏆 #1,223 overall#26 of 148 in Tokenization & Preprocessing
nlp-uoregon /trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
$ git clone https://github.com/nlp-uoregon/trankit.gitStars
799
799
Forks
106
106
Language
Python
License
Apache-2.0
Created
Jan 8, 2021
5.7 years old
Last push
Jul 22, 2025
More in Tokenization & Preprocessing
Tokenization & Preprocessing
explosion/spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
🏷️ Named Entity Recognition🗂️ Text Classification🧠 Natural Language Processing
PythonMITupdated Aug 24, 2026
Tokenization & Preprocessing
explosion/spacy-streamlit
👑 spaCy building blocks and visualizers for Streamlit apps
🏷️ Named Entity Recognition🗂️ Text Classification🧠 Natural Language Processing
PythonMITupdated Jul 29, 2024
Tokenization & Preprocessing
CogComp/cogcomp-nlp
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
🏷️ Named Entity Recognition⛏️ Information Extraction🧠 Natural Language Processing
JavaOtherupdated Jul 7, 2023
Tokenization & Preprocessing
janlukasschroeder/nlp-cheat-sheet-python
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
🏷️ Named Entity Recognition🧠 Natural Language Processing
Jupyter Notebookno licenseupdated Feb 11, 2023
Tokenization & Preprocessing
mit-ccc/TweebankNLP
[LREC 2022] An off-the-shelf pre-trained Tweet NLP Toolkit (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset
🏷️ Named Entity Recognition🧠 Natural Language Processing
PythonApache-2.0updated Jan 24, 2024
Data from GitHub · snapshot Sep 24, 2026