Dedup a folder in 3 easy steps

Imagine we want to dedup these files:

example01
├── beethoven.mp3
├── haydn.mp3
├── mozart (copy 1).mp3
├── mozart (copy 2).mp3
├── mozart (copy 3).mp3
├── mozart.mp3
├── pdf_1
│   ├── different-dimensions.pdf
│   ├── ocr.pdf
│   └── scans.pdf
├── pdf_2
│   ├── different-dimensions.pdf
│   ├── ocr.pdf
│   └── scans.pdf
└── pdf_3
    ├── different-dimensions.pdf
    ├── ocr.pdf
    └── scans.pdf

Step 1: Tell BigTrees to scan them and generate a dedup script.

bigtrees dupes example01 \
    --output dedup.sh \
    --dupes-out-fmt dedup-script

Step 2: Skim the script to make sure it’s categorized the duplicates the way you want. BigTrees sorts each set of dupes, putting its best guess about which copy you’d prefer to keep at the top. The script will confirm that one still exists before deleting the others.

Here are the relevant lines for our example:

dupe_set 'yefChv7y4LiL0Lw+vjmXOm' 'dirs'
dupe 'example01/pdf_1'
dupe 'example01/pdf_2'
dupe 'example01/pdf_3'

dupe_set 'xRddK/EUyJ+AdIJCRZM2ib' 'files'
dupe 'example01/mozart.mp3'
dupe 'example01/mozart (copy 1).mp3'
dupe 'example01/mozart (copy 2).mp3'
dupe 'example01/mozart (copy 3).mp3'

You can edit the script now if you want to pick specific files to keep. Just move them to the tops of their sets.

Step 3: When you’re ready, go ahead and run it:

bash dedup.sh

yefChv7y4LiL0Lw+vjmXOm 3 dirs: skip 0, rm 2, keep 'example01/pdf_1'
xRddK/EUyJ+AdIJCRZM2ib 4 files: skip 0, rm 3, keep 'example01/mozart.mp3'

Total: 5 removed, 0 errors

example01
├── beethoven.mp3
├── haydn.mp3
├── mozart.mp3
└── pdf_1
    ├── different-dimensions.pdf
    ├── ocr.pdf
    └── scans.pdf