Repo

LLM-Sanitizer-Pro

@yogsothoby Yog-Sotho YogSotho · Independent

0

Clean, de-identify and curate LLM training data at scale: PII & secrets redaction (regex + GLiNER NER), exact/fuzzy/semantic dedup, language ID, FineWeb-Edu/DCLM quality scoring, Gopher/C4/FineWeb rules, benchmark decontamination, chat validation. Streaming, parallel, resumable CLI + Python API with run manifests & dataset cards

Built withPythonaiai-toolsclidata-cleaningdatasetdeduplication

See what other members are building and what each project is looking for.