dedup

Best-Effort Extent-Same, a btrfs dedupe agent

📧 CLI to deduplicate mails from mail boxes

Check your files for data corruption and run quick file deduplication

CLI utility to find near duplicate images and remove all but the best copy.

High performance rsync backup utilising BTRFS / ZFS filesystem features

🍉基于Hadoop和HBase的大规模海量数据去重

Golang structured logging (slog) deduplication and sorting for use with json logging

String deduplication package for Go

Parallel Patterns Implementation of PARSEC Benchmark Applications

A C++ reimplementation of Near Duplicate Video Detection - Get a 64-bit comparable hash-value for any video (Video Hash).

distill large scale web page text

Hard Link Deduplicator

Find and remove duplicate rows in Excel spreadsheets by selected columns. Works with .xlsx and .csv - no Excel installation required. Generates a remo

Sift duplicate whitespaces away!

A CLI tool for images analysis: checking image integrity, images deduplication, image retrieval.

Find (partial content) duplicate files.

📄【优爱酷可视化网站网页数据采集系统】 采用先进的可视化采集技术，智能识别网页元素类型，如：图片、文字、链接、HTML 、文件等，支持运行Javascript脚本、应用正则表达式、自动滚屏、自动翻页、打开弹出窗口并采集数据，支持数据自动去重、仿人工间歇暂停防IP阻塞、自动保存等采集设置；支持浏览器Cookie和缓存等浏览器设置；支持代理轮换科学上网采集；支持“类别/关键字”；支持图像重命名等； 更可支持多线程采集等高级采集选项设置，vip版还可支持定时计划采集。

Local music library manager: tag editing, genre classification, duplicate detection & safe dedup. Roon-compatible, DRY-RUN-first, never deletes, fully reversible. 本地音乐库管家：标签编辑·流派分类·重复查找与安全去重。

Print FastCDC rolling hash chunks and checksums.

Easy Deduplication