There's software out there to find and remove duplicate files. I doubt that's going to be very satisfying though. Probably not smart enough to dedupe inside archive files (and would you trust them to do it right?).
Pains me to say it, but semantic web tools would really help to collaboratively manage retro files. Probably less than 5% of your retro files are unique to you.