14 January 2023, 19:44 | #61 | |
Registered User
Join Date: Mar 2013
Location: Leipzig/Germany
Posts: 471
|
Quote:
Did you delete those? Should files from collections be deleted? I spotted some 17-byte-sized .dms/.lha files. Should probably be removed. |
|
15 January 2023, 07:34 | #62 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
The problem with these collections is that people got the files in those collections from the same sources. A cracked game or a demo got spread and ended up in multiple collections. In the end I think that it's more important to keep the unique files and not x copies of the same file in various collections.
Thank you I'll have a look at those. |
15 January 2023, 10:49 | #63 |
Registered User
Join Date: Oct 2012
Location: Italy
Age: 49
Posts: 2,977
|
Are you saying you want to keep an unique copy of each file ?
Collections are many, it's hard job to remove all the dupes i guess |
15 January 2023, 13:00 | #64 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
Well, yes and yes Realistically we'll never get to 'only one version of each unique file', but if we can find duplicates we should remove them.
|
15 January 2023, 16:34 | #65 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
The contents of those files are "Nuked by Byteandi" btw. I assume that is a virus?
Last edited by TCD; 15 January 2023 at 16:50. |
15 January 2023, 18:26 | #66 |
Registered User
Join Date: Nov 2004
Location: Germany
Posts: 629
|
Hmm, here is a suggestion:
if you delete files from a collection, maybe add a text file with deleted filenames and crc ? In that way, if people want to assemble the collection, they would have the information needed, because the duplicates are found somewhere else. |
15 January 2023, 18:59 | #67 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
Maybe once Turran has his script for MD5 sorted this can be done, but I won't do it manually. Just to make sure that people understand that it was an idea in the main thread (http://eab.abime.net/showpost.php?p=...postcount=3166) and I checked if it is possible and makes sense (as in are there a lot of duplicate files). So far I've deleted a few files that can be restored if it's really important that the collections stay intact.
|
16 January 2023, 10:38 | #68 |
Registered User
Join Date: Oct 2012
Location: Italy
Age: 49
Posts: 2,977
|
Collections should remain intact IMHO
I tell this as a collector For me there's no problem in your final decision, i have already two backups of the various collections I just want to say my personal point of view |
28 January 2023, 20:49 | #69 |
Registered User
Join Date: Nov 2007
Location: Poland
Posts: 1,417
|
There are a lot of duplicates in both IPF catalogs:
Non TOSEC IPFs - Official Non TOSEC IPFs - Unofficial |
28 January 2023, 21:38 | #70 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
Yep, already removed over 2000 of those (there are actually three folders besides the official TOSEC ones that need to be checked) I'm cleaning the IPF folders up and will make a thread about the missing official ones afterwards.
|
29 January 2023, 10:43 | #71 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
Speaking of removing files: I've uploaded a list of duplicates to the root of the server called 'Duplicate_Files.txt'. If a file you are looking for seems to be missing on the server please have a look at the file and check if it wasn't a duplicate of another file.
As an example: You are looking for the file '17bit-1574b.dms'. The file isn't available but in the list of duplicates you'll find this entry when searching for the file: Code:
== File with md5 d2e91da94549e9554b1287ef5b5b161d, 902776 bytes, found in 2 locations == Collection/Compilation/Fred_TheGang_2014_Amiga_Collection/Amiga - Not TOSEC/Demos - DMS/Odyssey (1991-12)(Alcatraz)(Disk 2 of 5)[TP#1].dms Collection/Compilation/BTTR/17bit/dms/15xx/17bit-1574b.dms |
03 February 2023, 09:30 | #72 |
HOL/FTP busy bee
Join Date: Sep 2006
Location: Germany
Age: 46
Posts: 32,409
|
After a bit over two months I'll call it done for now. I'll still remove some duplicates in the next weeks, but the main cleaning up is over. Feel free to report any bits that you spot that could use some TLC and I'll get to it.
|
06 February 2023, 13:24 | #73 |
Moderator
Join Date: May 2012
Location: Stockholm / Sweden
Age: 49
Posts: 1,576
|
Once again, BIG thanks. Looks like little over 500GiB has been cleared up.
|
12 February 2023, 13:05 | #74 | |
Registered User
Join Date: Dec 2008
Location: Ursviken
Posts: 146
|
Quote:
Maybe the collections could be made into 'virtual collections': keep unique files (index, texts, images etc) belonging to the collection itself, and then link in the actual content from other places on the FTP server (if downloading links are supported).. Another way to do it would be to create a wget/curl compatible file list for batch downloading a collection as it was put together. |
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Guess the Screenshot - 2023 | malko | EAB's competition | 1210 | 14 February 2024 10:53 |
Amiga Ireland 2023 | Daedalus | Amiga scene | 19 | 30 March 2023 16:04 |
Super League 2023: Round 1 - Xenon 2 | lifeschool | EAB's competition | 11 | 29 January 2023 01:23 |
Super League 2023: Round 1 - Cast Your Votes | lifeschool | EAB's competition | 27 | 08 January 2023 01:15 |
Passione Amiga #11 (January 2023) is published! | passioneamiga | News | 0 | 16 December 2022 11:25 |
|
|