Cleanup 2022/2023

[jbl007]

Originally Posted by TCD

'Collection/Compilation/Fred_TheGang_2014_Amiga_Collection/Amiga - Not TOSEC/Demos - DMS' contained 291 DMS files. After converting them to ADF 4 files remained that couldn't be converted to ADF. That makes 287 ADF files. Out of those only 1 (one) wasn't in TOSEC.

Nice.

Did you delete those?
Should files from collections be deleted?


I spotted some 17-byte-sized .dms/.lha files. Should probably be removed.

[TCD]

Originally Posted by jbl007

Did you delete those?
Should files from collections be deleted?

The problem with these collections is that people got the files in those collections from the same sources. A cracked game or a demo got spread and ended up in multiple collections. In the end I think that it's more important to keep the unique files and not x copies of the same file in various collections.

Originally Posted by jbl007

I spotted some 17-byte-sized .dms/.lha files. Should probably be removed.

Thank you I'll have a look at those.

[chip]

Are you saying you want to keep an unique copy of each file ?

Collections are many, it's hard job to remove all the dupes i guess

[TCD]

Originally Posted by chip

Are you saying you want to keep an unique copy of each file ?

Collections are many, it's hard job to remove all the dupes i guess

Well, yes and yes Realistically we'll never get to 'only one version of each unique file', but if we can find duplicates we should remove them.

[TCD]

Originally Posted by jbl007

I spotted some 17-byte-sized .dms/.lha files. Should probably be removed.

The contents of those files are "Nuked by Byteandi" btw. I assume that is a virus?

[Dan]

Hmm, here is a suggestion:
if you delete files from a collection, maybe add a text file with deleted filenames and crc ?
In that way, if people want to assemble the collection, they would have the information needed, because the duplicates are found somewhere else.

[TCD]

Maybe once Turran has his script for MD5 sorted this can be done, but I won't do it manually. Just to make sure that people understand that it was an idea in the main thread (http://eab.abime.net/showpost.php?p=...postcount=3166) and I checked if it is possible and makes sense (as in are there a lot of duplicate files). So far I've deleted a few files that can be restored if it's really important that the collections stay intact.

[chip]

Collections should remain intact IMHO

I tell this as a collector

For me there's no problem in your final decision, i have already two backups of the various collections

I just want to say my personal point of view

[rygar]

There are a lot of duplicates in both IPF catalogs:
Non TOSEC IPFs - Official
Non TOSEC IPFs - Unofficial

[TCD]

Yep, already removed over 2000 of those (there are actually three folders besides the official TOSEC ones that need to be checked) I'm cleaning the IPF folders up and will make a thread about the missing official ones afterwards.

[TCD]

Speaking of removing files: I've uploaded a list of duplicates to the root of the server called 'Duplicate_Files.txt'. If a file you are looking for seems to be missing on the server please have a look at the file and check if it wasn't a duplicate of another file.

As an example: You are looking for the file '17bit-1574b.dms'. The file isn't available but in the list of duplicates you'll find this entry when searching for the file:

== File with md5 d2e91da94549e9554b1287ef5b5b161d, 902776 bytes, found in 2 locations ==
Collection/Compilation/Fred_TheGang_2014_Amiga_Collection/Amiga - Not TOSEC/Demos - DMS/Odyssey (1991-12)(Alcatraz)(Disk 2 of 5)[TP#1].dms
Collection/Compilation/BTTR/17bit/dms/15xx/17bit-1574b.dms

So the file 'Odyssey (1991-12)(Alcatraz)(Disk 2 of 5)[TP#1].dms' is the exact same file as '17bit-1574b.dms'.

[TCD]

After a bit over two months I'll call it done for now. I'll still remove some duplicates in the next weeks, but the main cleaning up is over. Feel free to report any bits that you spot that could use some TLC and I'll get to it.

[Turran]

Once again, BIG thanks. Looks like little over 500GiB has been cleared up.

[peo]

Originally Posted by chip

Collections should remain intact IMHO

I tell this as a collector

For me there's no problem in your final decision, i have already two backups of the various collections

I just want to say my personal point of view

Maybe the collections could be made into 'virtual collections': keep unique files (index, texts, images etc) belonging to the collection itself, and then link in the actual content from other places on the FTP server (if downloading links are supported)..


Another way to do it would be to create a wget/curl compatible file list for batch downloading a collection as it was put together.