Tokenization & Preprocessing
roshan-research/hazm
▲ +1 ★ this week
Persian NLP Toolkit
🧠 Natural Language Processing💎 Text Mining
PythonMITupdated Apr 1, 2026
R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
$ git clone https://github.com/bnosac/udpipe.git▲ +1 ★ this week
Persian NLP Toolkit
[LREC 2022] An off-the-shelf pre-trained Tweet NLP Toolkit (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset
Fast, Consistent Tokenization of Natural Language Text
▲ +1 ★ this week
A Japanese tokenizer based on recurrent neural networks
This repository consists of all my NLP Projects
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
Data from GitHub · snapshot Nov 16, 2026