Have these Amiga games been dumped?

[TheCyberDruid]

@Interceptor
I'm just curious: Is there a way to get kaffer's IPFs approved? I mean since the way that you would do it is 'too tedious for just a few disks' it sounds like an idea just to check his IPFs and approve them. Just my 2 cent.

[Interceptor]

Originally Posted by kaffer

I had a discussion with SPS about this. It is not my intention to have my generated IPFs be confused with SPS-generated and -blessed archival quality images. I stamp my IPF headers with some magic values to make them particularly easy to spot. Further more, of course, SPS keeps CRC32 values for all their blessed images, so this is another way to distinguish between them. This could be beefed up to sha1sum or similar if collisions (accidental or forged) were a concern.

IPF can be a general-purpose container *and* the preservation format of choice.

I know

i just wanted to clarify this to people who suddenly wonder why SPS can't do something for years and then someone else can. SPS could have banged out 'unauthentic' images for pretty much every game by now, long ago in fact.

hopefully i've acheived that.

personally, i would have preferred you work with SPS to make an official version if that were possible, rather then spend your efforts on unofficial ones, but i dont know the specifics in this case, maybe you already tried that in it wasnt possible for these disks, i don't recall. but this paragraph is aimed at answering TCD's post somewhat.

[kaffer]

Ah well, this all came about from a personal project to investigate the old Amiga disk formats and copy protection schemes. Most of which are total crap it turns out, but there are sure a lot of them.

Then IPF support was an afterthought when the decoder sources became available. And indeed I just viewed it as a handy container format supported by lots of emulators, *and* supported for writing by Kryoflux! Super nice.

The preservation quality image thing was not my thing. Also I haven't seen any process by which SPS gets others to join in on that. I kind of got the impression that SPS's preservation process (the technical side at least) was IFW.

[Interceptor]

Originally Posted by kaffer

Ah well, this all came about from a personal project to investigate the old Amiga disk formats and copy protection schemes. Most of which are total crap it turns out, but there are sure a lot of them.

Then IPF support was an afterthought when the decoder sources became available. And indeed I just viewed it as a handy container format supported by lots of emulators, *and* supported for writing by Kryoflux! Super nice.

The preservation quality image thing was not my thing. Also I haven't seen any process by which SPS gets others to join in on that. I kind of got the impression that SPS's preservation process (the technical side at least) was IFW.

Its IFW because no people both capable of doing it and willing to be so thorrough have approached him with a genuine offer of help i guess...

[kaffer]

Originally Posted by Interceptor

Its IFW because no people both capable of doing it and willing to be so thorrough have approached him with a genuine offer of help i guess...

Well, this may be true. But offers of help for this sort of thing are rare unless they're solicited.

Anyhow, this is all making presumptions about what IFW thinks and wants.

[BarryB]

Well, with kaffers help, i've been able to dump and write IPF's of two of those Pysgnosis games that SPS haven't had time to release yet, so being able to backup my own streams to IPF is a welcome bonus.

As for the 'authenticity', code something into the IPF library that stops 'none authentic' or 'tampered' IPFs being used and only allows SPS or kaffers to load? Or, just warn if not an SPS IPF?

[mr.vince]

If it's clearly defined as this, all is fine. It's just misleading and bringing people to the conclusion that stories have been told if technology implemented is presented as the "glorious" result of good reverse engineering. Since the technology is available in source format this is obviously as exciting as cracking a game you have the commented source for.

I have not seen any of these IPFs, but I am pretty sure this is just raw data stored in the IPF container. This has been in the format since it was conceived over a decade ago, but of course does not fulfil the intention of a Interchangeable Preservation Format file. It's like an extended ADF, with raw data and absolutely no control over it. There's no separation of content and context. I have no idea about what will happen if such data is written back to disk. While this may work for some disks, it won't work for anything that deals with advanced protection where it is important how data is written.

Again, I appreciate your efforts, I just discourage it if such development leads to conclusions as above. It's impossible to store these four Psygnosis games _properly_ in an IPF. IPFs can be verified to be error free by looking at transportation layer. These can't. I know that Denis will ignore this because it's not important to him (no offence, just being plain honest here from what I noticed according to the posts he made), but we actually do care about the good reputation the format has gotten.

Denis: It's ok to ask, but your wording above is really insulting. Yes, we know what we do. We did not have to open the tech, but we did because we felt it was about time and important.

I think it is ok to ask for this kind of respect.

[kaffer]

Mr Vince, I think noone is insulting you. I think maybe you are insulting me however. You obviously know nothing about my IPFs. You even sound confused about IPFs in general to be frank, they are not some high-level scripted form of disk data, actually they are quite raw, just represent tracks as streams of raw bit cells, mfm-encoded bit cells, and gap filler bytes. I'm sure the analyser that produces the SPS IPFs contains the high-level scripting smarts, and can validate IPFs as it produces them, according to the script, but this higher-level form, separation of content/context, is *not* represented in the resulting IPF.

[dlfrsilver]

Denis: It's ok to ask, but your wording above is really insulting. Yes, we know what we do. We did not have to open the tech, but we did because we felt it was about time and important.

I think it is ok to ask for this kind of respect.

I'm sorry if you felt it like this, i never meant to be disrespectful. I just wanted to get some clues since Kaffer has been able to generate IPF.

Obitus, Nitro, armour geddon and killing game show use very long tracks.

Just check the source code of the format description Kaffer made :

https://github.com/keirf/Amiga-Disk-.../psygnosis_c.c

Please enlight, and even comment it, tell us your opinion about it

[kaffer]

The raw tracks are not that long, only ~105500 bit cells, like most other long track formats. It's just that they can contain more data than usual, because of the more efficient non-mfm encoding scheme. Typically a long track contains no more than 6kB of decoded data, whereas this format could have up to 9kB (but more typically around 7-8kB).

[Harry]

Personally, I dont see the problem with the data in the extended ADFs / unofficial IPFs of the Psygnosis games. At least for Nitro and KGS i am sure that they check the integrity of the whole disk when imaging with the WHDLoad-Installer, because I analysed and verified that as I made the install. So if they install in WHDLoad, their data is complete and errorfree (ok, with the probability that a simple ADD.L checksum can give).

And finally I want to ask everyone to chose their wording carefully and think twice before posting stuff which someone could get as the wrong end of the stick.

[IFW]

Guys... I've been away developing my family as well as KryoFlux tools for a few months and this is what I have to read when I come back

So, just a few very quick answers:
1, If you use IPF as a container format and store those games as raw streams it is possible to have them in IPF. Note, the raw data is marked as "test only" in the source I believe there is even a comment on that storage mode in there
2, If you actually do what any other official IPFs do and have an encoder as well as a decoder for the encoding being used it is not possible to store those games in IPF yet, as the codec has to be added to the analyser as well as the library.
3, IPF files do have information on track and disk geometry, timing etc, more over it is perfectly possible to compare them as long as they use a format that is actually encoded and not stored as raw data. Therefore they do have context information. A format that only stores raw data (e.g. extended adf etc) makes this very difficult as it lacks this kind of information.
4, Not many people offered (read: 0) genuine and useful help in any kind of enhancement; it's really very domain specific knowledge especially once you use stream dumps from a device like KryoFlux.
However, let's not forget all the people who do help when and where they can, either by improving things that are not domain specific (e.g. porting KryoFlux tools to other platforms, GUI work), or getting hold of games, dumping them etc.
5, There are tons of work that would require contributors that are not domain specific including searchable databases etc.

Now everyone be a good boy or girl and start getting hold of dumping undumped games for all platforms including Amiga

Thanks

[mr.vince]

Originally Posted by Harry

Personally, I dont see the problem with the data in the extended ADFs / unofficial IPFs of the Psygnosis games. At least for Nitro and KGS i am sure that they check the integrity of the whole disk when imaging with the WHDLoad-Installer, because I analysed and verified that as I made the install. So if they install in WHDLoad, their data is complete and errorfree (ok, with the probability that a simple ADD.L checksum can give).

This can be done, but the intention of IPFs is to be able to verify them at the format level, e.g. loading the IPF and then having the designated decoder verify the integrity. Passing this on to an external solution would mean having to install each and every IPF via WHDLoad or run some custom scripted loader in emulation.

That's what I actually tried to point out above; it's possible but not practical, especially if you think of archivists at some library or museum which don't have knowledge or tools to actually install games to verify their assets.

Originally Posted by Harry

And finally I want to ask everyone to chose their wording carefully and think twice before posting stuff which someone could get as the wrong end of the stick.

Which easily happens, as it happended with my post regarding Keir's work.

@Keir: My apologies if this was a sour pill to swallow, it was not meant to be rude. Maybe IFW's comments help making clear where I drew the line.

What I actually thought about later that evening... what if Keir would want to come aboard? I mean... doing it properly, adding the encoder, add some other stuff as well. Some things are just on hold because of the lack of manpower. And it's really hard finding people that think bitcells and stuff are sexy.

[kaffer]

Yeah, I will agree there is more context than in an extended ADF. Primarily, you know where the data is, where the gap is, and crucially where the write splice should go. I'm still not convinced that storing the actual data raw, rather than decoded, really matters that much. But yes since every other official IPF stores decoded data I can see it would be nice to do the same for these disks too.

I can help, since I have C code for the encoding and decoding now. Tbh that isn't really the hard bit for some of these games. Obitus and Armour-Geddon both have mastered but apparently empty/unused tracks. And Armour-Geddon has a track format with no validation info at all. (Not the specially-encoded tracks, some other crap and clearly title-specific MFM-based format.) The attention to detail required to get these kinds of bodged-together disks preserved correctly is the time consuming bit, and one reason why IFW gets plenty of respect from me.

[mr.vince]

What might complicate things is that the transport layer would need to be created and then scripts would need to be written which the Analyser would use to correctly form the data and produce the IPF.

Will email you at the address stored on the KryoFlux forums.

[kaffer]

Thanks mr v!

[kaffer]

Armour-Geddon now supported by mfmparse. At least to the extent it can be (no validation info on some tracks).

[prowler]

Originally Posted by mr.vince

And it's really hard finding people that think bitcells and stuff are sexy.

It shouldn't be. Just keep your eye on the Currently Active Users Viewing This Thread panel.

[dlfrsilver]

Yes, this thread is very interesting !

[BarryB]

Jeez, I hope I don't start a Flame War

I know any IPF's I create with kaffers tools are staying soley on my drive, no way am I upsetting kaffer or SPS by distributing them, too much repsect for both of them to do something stupid like that!!

[kaffer]

Originally Posted by BarryB

I know any IPF's I create with kaffers tools are staying soley on my drive, no way am I upsetting kaffer or SPS by distributing them, too much repsect for both of them to do something stupid like that!!

The ones you create yourself from your own dumps, you can do as you like with as far as I'm concerned. They're not "my" images, you merely used my analyser on your own dump data. They are not manually-checked preservation-quality images however, hence wide distribution wouldn't really be advisable.

[BarryB]

Originally Posted by kaffer

They are not manually-checked preservation-quality images however, hence wide distribution wouldn't really be advisable.

And that's the reason why they will never be uploaded anywhere or given to anyone!

The idea was to make working images of MY disks, and so far your tools have exceeded my expectations and continue to amaze me. They give me an alternative legal avenue for obtaining IPFs that SPS have but simply can't release.

[kipper2k]

It would be nice to have Kaffers tool converted to Windows

[Codetapper]

Since only 4 games were created with that format is it possible that you can just include some kind of CRC32 of all tracks in the stream and then you can at least see if the data matches? It's a hack but at least you could read the track back and compare the CRC couldn't you?

[BarryB]

Originally Posted by kipper2k

It would be nice to have Kaffers tool converted to Windows

That's why I installed Fedora 17 in a VM and downloaded kaffers repository. Took same faffing about to get it working I'll tell you, but now I can dump and create IPFs provided the relevant disk format is supported.

kaffer

[mr.vince]

Originally Posted by Codetapper

Since only 4 games were created with that format is it possible that you can just include some kind of CRC32 of all tracks in the stream and then you can at least see if the data matches? It's a hack but at least you could read the track back and compare the CRC couldn't you?

You can't compare the CRC of two RAW dumps. Well you can, but they don't match. You need decoded data, with the encoding scheme scripted. You will have to implement the decoder in the comversion tool AND in the IPF support lib. That's why we said it needs the transport layer...

The rest is taking shortcuts, with the problems and obstacles you just come
across.

Kaffer... I am in the middle of your mail, but kids keep me entertained...

[kaffer]

Need to distinguish between raw dumps, and raw aligned data in an IPF. Of course raw dumps present an unaligned and undistinguished stream of data, gap, and write splice. But if the data component is pulled out by an analyser and encoded into, say, an IPF, there's no reason that can't be CRC-checked, or written reliably back to disk, or whatever.

In any case, the track format that kicked all this off does contain validation info (albeit a weak ADD.L based checksum, but that's par for the course with Amiga track formats). Yes the analyser needs to be able to decode the raw data to check it, but that doesn't particularly mean the decoded data then has to be written into IPF. If the decoded data has been checked, then of course the original raw data is known good too, and can legitimately be dumped into an IPF (correctly aligned, and separated from gap, of course).

I will keep an eye out for your email mr vince.

[Rixa]

Originally Posted by mr.vince

The Sony is a good choice (only 82 tracks, but has all the bells and whistles needed otherwise)

My MPF920 does 84 tracks. I wonder how much it differs from the E model otherwise. (perhaps it's otherwise worse? )

[mr.vince]

Lucky Rixa then. Great drive. Enjoy.

[BarryB]

My Sony 'E' only dumps up to 81 according to 'dtc -c2', so what happens to the drive if it's trying to dump with preservation setting 'i2'?

@Rixa: When you say '84' you mean 0-83 = 84 tracks?

[Rixa]

Yes, BarryB, just that. 0-83.

The sticker on the drive says "E/131 SEP2003", probably indicating date of manufacture.

[BarryB]

Yep, mine says E/131 APR2001 and has created some very good dumps that kaffer has created working IPFs from

[kaffer]

Was looking forward to declaring BarryB's Dungeon Master dump successfully imaged. But only now I realise it can't be done (as an IPF (yet)). Not a waste of time though. This one actually educated me some.

[BarryB]

"If you know the enemy and know yourself you need not fear the results of a hundred battles."

So wrote Sun Tzu! And you already know the enemy kaffer....

How amazing that a quarter of century old protection is still causing problems for modern man!

[BarryB]

Just an update, have dumped and uploaded the following to SPS using their dump script:

Amiga Challenge - Indy 500, Cool Croc Twins (Both AmigaDOS) and Paradroid '90 with custom disk format, no Copylock.
Crystals of Arborea - Long Track protection on track 79
Gold of the Realm - AmigaDOS
The Munsters - Retail with PROTEC track, not unprotected as SPS 1265
TIME - English version, Copylock on Disk 2 (Same type as F-29 Retaliator)

All protections identified via mfmparse!!

[dlfrsilver]

yes time use indeed a copylock on disk 2

[kaffer]

Paradroid '90 is Graftgold format, which is AmigaDOS but with cylinder# where the track# would normally be in the sector header. Copylock on TIME is the old hidden sector type.

[BarryB]

Too many disk formats to keep track off!! I also think I put copylock down for Munsters when submitting the dump script when it should be PROTEC!!

BTW: Denis, did you get your disks 'cleaned'?

[dlfrsilver]

i finally found that it is in fact the demo Hardwired from the silents

[mr.vince]

No worries, Barry, our Analyser will know what it is. It understands 500 main formats and some thousands if counting variants in as well.