The sorry state of Amiga software preservation

[absence]

I was poking around at the WHD slaves for Lotus 2 and 3, and noticed that there are four versions of Lotus 2 supported (not including the cracked NTSC one) and three versions of Lotus 3. A web search suggests a fourth version of Lotus 3 with a fix or something that could be obtained by sending the original disks back to Gremlin. I wondered what the differences between all these versions were, and realised that there's basically no way of finding out, because almost none of them are preserved!

While SPS had a promising start, they've long since ground to a halt. They have one single version each of Lotus 2 and 3, and the others aren't even on their wanted list. We're missing three out of four versions for each game! And that's just the tip of the iceberg. Where can you find a clean set of original floppy images of Directory Opus older than 4.0? Deluxe Paint 2? Sonix 1.0? Certainly not in TOSEC.

Is there something the community can do about the situation? Are there missing floppies in the hands of collectors? Could stuff like Kryoflux, MFMWarp, etc. help? Are the images already dumped, but circulated privately instead of being added to curated collections? Or are they simply lost for all time?

[Minuous]

Presumably the relevant WHDLoad slave authors have had copies of each version, otherwise they could not be supported by those slaves.

[absence]

Even if the authors had the floppies when they wrote slaves back in the 90s, both authors and floppies could be long gone by now.

[jotd]

long gone like ... dead? I keep the MFM/copies of the disks I supported. My disks could still work and I have a working A1200 to warp them if needed.

I'm not saying I have all versions of Lotus II. I did not fix that one (except a crack, one of my first patches)

Make me a wishlist. Not sure I have everything or even anything. I may have the exact preserved versions of SPS...

[idrougge]

I've ripped lots of old original productivity software and submitted to TOSEC maintainers, but they just aren't interested.

[McTrinsic]

SPS and TOSEC are a completely different thing.

SPS is more of a preservation of the very original.

TOSEC tries to provide the content in whatever way is necessary, including cracks and such.
Cheers,
McT

[Thalion]

Originally Posted by idrougge

I've ripped lots of old original productivity software and submitted to TOSEC maintainers, but they just aren't interested.

Thats not true. Don't forget TOSEC is a "hobby" project and affords a lot of time. Unfortunately not everybody who once volunteered to maintain a certain section of Software is still active. In my opinion TOSEC needs some fresh blood to keep up the sections which could be called vacant. Thats all. But unfortunately not easy to recruit some new Maintainers.

[absence]

Originally Posted by jotd

long gone like ... dead?

While that's always a possibility, more commonly people move on to other interests.

Originally Posted by idrougge

I've ripped lots of old original productivity software and submitted to TOSEC maintainers, but they just aren't interested.

That's my experience as well. Even if new submissions were accepted, I don't think TOSEC is a good model for preservation. There's too little documentation of the process, and over the years any old image has been added, and now it's a huge mess. Stricter curation is needed. A lot more metadata is needed, for example which versions of software are known to exist, how many floppies were there in the package, which versions are preserved and which have missing disks, are there issues with existing images (unfortunately people accidently or knowingly modified original floppies), modification history (e.g. technical details when an image is replaced by a better one), known serial numbers or keys, source of the disk image (i.e. who has access to the physical floppy the data came from), etc. Also there should only be one good (or as close to good as possible) image for each version, not like in TOSEC where you have ten alternative versions and all of them are obviously modified in drastic ways.

[Thalion]

In the past i have contributed much to TOSEC and always was in good contact with the guys. Of course it takes a lot of time and the project is transforming over the time as many stuff is being sorted out now but don`t forget TOSEC is about collecting as much as possible.
And adding more meta information is nearly impossible for most of the stuff. The stuff simply is too old and there are simply no informations available.
Of course i don't say everything is optimal but considering the work done TOSEC simply is the best project we have around.
For more specific collections like ony working versions there are other projects like No-Intro. So if you look around a bit you will find the collection most suited for your needs for sure.

[BarryB]

The SPS games database isn't up to date! I've submitted games on various platforms that are not listed in their database, but they may already have dumps of those games but the database doesn't reflect that! Also, some games modify the disk before you can play so while SPS may have the game dumped, all dumps may be from modified versions so another reason they are not listed. Plus, like Thalion says about TOSEC, SPS is a 'hobby' project and their focus isn't on updating the database every day!

As for 'preserving' games, people have to be aware that their games need preserving in the first place, if people don't know this how can they possibly preserve them? Don't forget many Amiga users sold up long ago and all the games with them, some just binned all their games and others just left them rotting in garages or cellars so lord knows what condition they will be in! Some people, like me, kept most of their games in good condition and have submitted them all to SPS but I don't have every game or sent off for every update or newer version so another reason these updates/versions are not common.

[absence]

Originally Posted by Thalion

don`t forget TOSEC is about collecting as much as possible.

Yes, that's one of the main reasons I don't think TOSEC is suitable for preservation. When modified images have as much value as unmodified ones, and there's no metadata to tell them apart, it becomes difficult to notice that e.g. Deluxe Paint 2 is missing. People's spare time is limited, and at a quick glance it looks like it's already preserved. If it was clearly labelled MISSING in big red blinking letters, collectors of original floppy disks would be more likely to submit what they have.

Originally Posted by Thalion

And adding more meta information is nearly impossible for most of the stuff. The stuff simply is too old and there are simply no informations available.

Maybe I'm optimistic, but I still think there are collectors with boxed copies of a lot of Amiga software around. Obtaining valuable metadata is as easy as opening the box, counting the floppies, and writing down what it says on the labels. If they can be dumped as well, that's even better!

Originally Posted by Thalion

considering the work done TOSEC simply is the best project we have around.

I agree, and I think it's important to bring awareness to just how poor the preservation situation is, otherwise people could believe it's a solved issue with nothing more to do.

Originally Posted by Thalion

For more specific collections like ony working versions there are other projects like No-Intro. So if you look around a bit you will find the collection most suited for your needs for sure.

No-Intro only contains kickstart roms and the IPF images from SPS, so it doesn't add anything. I think it's wishful thinking that such collections exist.

[idrougge]

Making a list of what is missing is obviously something that can be done outside of both SPS and TOSEC, and cross-referenced to their collections.

TOSEC does have metadata, and it is being refreshed from time to time — it's just that the starting point was not optimal. TOSEC seems to have originated as a way to keep track of console dumps, where the number of altered images, cracks and variations is much, much smaller than on computers with rewritable media. The remaining Amiga maintainers seem to acknowledge this and aren't as interested in cataloguing every dump of Workbench 1.3 with an altered file date.
Nevertheless, due to the nature of Amiga dumps, a project like TOSEC is necessary simply so that someone interested in preserving only originals or known working copies has a way to tell bad and good dumps apart; it's just that most people don't realise that that's what it's for.

MrDolby, for instance, has done a lot of work to isolate original dumps of Workbench disks, which was previously impossible to tell apart in TOSEC because verified dumps wasn't a priority when the project started out.

But if TOSEC/Amiga is lacking in manpower, it needs to reach out to the community to recruit more people. And yes, more metadata is necessary if veracity is a goal.

[absence]

Originally Posted by idrougge

Nevertheless, due to the nature of Amiga dumps, a project like TOSEC is necessary simply so that someone interested in preserving only originals or known working copies has a way to tell bad and good dumps apart; it's just that most people don't realise that that's what it's for.

My impression is that Kryoflux dumps can show if a professionally duplicated floppy has been modified in a computer. As such, a library of variously modified images in ADF format (which lacks all the low-level data) isn't necessary to establish if a dump is good or not. Just dump the available Workbench 1.3 floppies with Kryoflux and check if they've been written to, and the painstalking work of comparing dates and modifications isn't necessary. Except for rare stuff where all the available floppies are modified of course.

[BarryB]

Kryoflux can't tell you if an Amiga disk has been modified or not, only by analysing the streams it produces can you tell if it's modified, but you need to KNOW how to interpret the stream data and also have a means to analyse it as well!

[emufan]

some serious intervention: stop polluting the application section with SPS/CAPS/IPF.
we dont need another section, we are not allowed to:

» Do not ask for SPS (formerly CAPS) releases

» Do not respond to messages that ask for SPS (formerly CAPS) releases

» Any posts that do so, risk being deleted!

» Do not Upload SPS (formerly CAPS) Image to the Zone

this has nothing to do with preservation.


#1) how many games you are still looking for from the CAPS list? 100+ 500+ how many ?

[absence]

Originally Posted by BarryB

Kryoflux can't tell you if an Amiga disk has been modified or not, only by analysing the streams it produces can you tell if it's modified, but you need to KNOW how to interpret the stream data and also have a means to analyse it as well!

I didn't suppose it would do so automatically, but I did think the SPS analyser was available to the public. Upon closer inspection, that doesn't seem to be the case. I don't know if any of the third-party analysis tools are advanced enough.

It's unfortunate that SPS is such a closed entity, because they obviously lack the capacity to get things done (Killing Game Show is a good example), and can use all the help they can get. While it's no doubt very technical work, I'm sure there are skilled and knowledgeable people in the community who would be interested in learning and contributing.

Originally Posted by emufan

some serious intervention: stop polluting the application section with SPS/CAPS/IPF.

Nobody has asked for SPS releases, so no forum rules are broken. It's not forbidden to talk about SPS or what they do.

[BarryB]

No, their analyser is not in the public domain so you have to use other tools like the HxC floppy emulator software to load the streams and analyse them but you still need to know what you're looking for!! The GUI that comes with the Kryoflux software has a scatter plot that can also help in identifying suspect modified disks, but I've never used it and wouldn't know what to look for if I did!

[emufan]

thanks for pointing out, BarryB.
since all talk about this so called preservation thing, you should consider using SuperCardPro by Jim Drew.

#1) the question is, does it really need this kind of copy at all, there is almost no application, which uses some kind of protection, which needs a hardware copier to make a genuine copy. i dont know of any, offhand.

[dlfrsilver]

Originally Posted by absence

I didn't suppose it would do so automatically, but I did think the SPS analyser was available to the public. Upon closer inspection, that doesn't seem to be the case. I don't know if any of the third-party analysis tools are advanced enough.

It's unfortunate that SPS is such a closed entity, because they obviously lack the capacity to get things done (Killing Game Show is a good example), and can use all the help they can get. While it's no doubt very technical work, I'm sure there are skilled and knowledgeable people in the community who would be interested in learning and contributing.


Nobody has asked for SPS releases, so no forum rules are broken. It's not forbidden to talk about SPS or what they do.

Ok, let me enlight a bit about all those questions and interrogations :

The main problem is that almost nobody even right now has the technical level to make IPFs.

Let me go in details : outputting IPFs is not a "put a stream, make a CTraw and click blindingly on processing requester, and tadaaa here's an IPF.

I see that more or less a certain amount of people would like to make themselves the IPF.

Problem : it's not an automatised process, and people are fantasming over their own hability to make them.

The main reason i think for the tool to not be accessible as people would love it too is that it's more easy to make broken IPF than using the analyser correctly to make them. It requires to know a truckload of copy protections.

In this sense, it's better and more secure to have people making dumps on one side (anybody can make them), and people to process on the other side.

It looks annoying, but that's a security measure which avoid us to deal with broken files.

There can't be a tool for preservation that would encode ANY disk format blindly, and therefore usable by the public.

That's why some protections are not yet supported. Killing Game Show use a specific disk format requiring a big update on the system. Istvan is very busy by its day life.....

So for the moment, just telling you that we have already good dumps of the unsupported games as stream files.

That's what counts. Once ready, those will be processed. During this time, you have all these games available to play with whdload.

SPS only needs right now people dumping their softwares, as participation to the preservation process.

The floppies are dying more and more. That's the priority, extracting the datas out of the floppies before it's too late.

Sorry for the offtopic.

[absence]

Thank you for the insight. I've written software to convert WWarp dumps of copy protected floppies to extended ADF format, so I'm well aware that producing IPF files from a flux stream isn't a simple automatic process. I also agree that releasing a tool that attempts to hide the complexity in order to let laymen convert to IPF without understanding what goes on would cause more harm than good. A tool for analysis would be useful for learning and research however, especially if people can be taught how to identify modified floppes.

If almost nobody has the technical skills to produce IPF files, it's better to share that knowledge and get other technical people who understand copy protection involved. What if the Istvan guy you mention gets killed in a traffic accident? I'm sure it's possible to work in a way where outsiders assist with the heavy lifting, and SPS can review the work and have the final say in whether it's good enough for an official release. If they worry about poor third-party releases, they can even cryptographically sign their own releases to remove any doubt about authenticity.

There's also the problem of SPS being very opaque. It's good to hear they have a flux dump of Killing Game Show, but how can the public know? It's not listed on their page. Are both PAL and NTSC versions dumped? Similarly, regarding the games that triggered this thread in the first place, does SPS have flux dumps of more versions of Lotus 2 and 3 as well? An up-to-date list of secured flux dumps would be useful, even if it takes another 15 years for them to be converted to IPF.