🏆 #5,737 overall#2085 of 2,559 in Natural Language Processing

shjwudp /c4-dataset-script

Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

$ git clone https://github.com/shjwudp/c4-dataset-script.git
GitHub social preview for shjwudp/c4-dataset-script
Stars
137
137
This week
—
Forks
18
18
Language
Python
License
MIT
Created
May 27, 2022
4.5 years old
Last push
Jun 7, 2023

Categories

GitHub topics

More in Natural Language Processing

Natural Language Processing

ademakdogan/ChatSQL

Convert the given plain text to MySQL query by ChatGPT

Pythonno licenseupdated May 27, 2023
GitHub ↗★ 146⑂ 27
Natural Language Processing

google/langextract🔥 active

▲ +37 ★ this week

A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.

🤖 Language Models
PythonApache-2.0updated Nov 10, 2026
GitHub ↗★ 39.3K⑂ 2.8K
Natural Language Processing

NandhaKishorM/laya🔥 active

▲ +409 ★ this week

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

PythonApache-2.0updated Nov 10, 2026
GitHub ↗★ 35.5K⑂ 3.2K
Natural Language Processing

nltk/nltk🔥 active

▲ +3 ★ this week

NLTK Source

PythonApache-2.0updated Nov 11, 2026
GitHub ↗★ 14.8K⑂ 3.1K
Natural Language Processing

neuml/txtai🔥 active

▲ +310 ★ this week

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

🔍 Search & Retrieval🤖 Language Models🧬 Embeddings
PythonApache-2.0updated Nov 13, 2026
GitHub ↗★ 13.4K⑂ 1K

Data from GitHub · snapshot Nov 13, 2026