New IPF decoder ("capslib") supports loading of raw files

[BarryB]

Tried Dungeon Master dumped on my A1200/060 and it loads in WinUAE, not played enough to see if it crashes though! Uploaded to the zone as 'Dungeon Master.rar'

Deep Core I couldn't get working, Disk 1 loads but when Disk 2 is inserted it just keeps checking track 0 in WinUAE and won't load.

Using this parameter to convert all images:

DTC -m1 -fstreamfile -i0 -fdiskname.ct -i2

Is that correct?

[mark_k]

It's good news that raw dumps are finally useable (to anyone other than SPS ). I downloaded the Kryoflux free archive and have a couple of questions...

The ctprog program version is 3.7, whereas version 3.8 was on the SPS dumping disk. Was there some bug in 3.8 which makes it preferable to use version 3.7?

In LICENSE.txt, it says:
"- Redistributions that are modified from the original source must include the complete source code, including the source code for all components used by a binary built from the modified sources. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable."

Does that mean you're releasing the source code to ctprog?

[IFW]

BarryB,
Yes, that is correct.
However I did load Deep Core without any problem like that, and that's certainly your dump, so maybe retry? The reason I asked you guys to check it is because it is supposed to be randomly crashing or something like that, and you need to play the game - but that's the very last thing I have time for... I tried both passwords for later levels from HOL, and they loaded fine as well.
Hmmm.... it has just occurred to me... could you post the CTR files generated?
I am always using the development version of DTC, so the output might be different... just want to make sure you use the same.


Mark_k,
As for the source: it's the MAME licence, without any change apart from replacing product names, as discussed with the licence originator. We may or may not release the source code, but it's a possibility if there is some interest.
The last stable version is definitely this one, as it was written by me - it might have had some changes that had been reverted though.

I think you asked about sources for CT Raw, which is somewhat related. The source code for the entire CODEC will come with the source release of the new library, once it's stable - currently lots of updates go into it very frequently.

As for SPS:
You know all this was made possible by SPS and it was our decision to do so, right...?
I think this is a fantastic thing for everyone: SPS as well as contributors - please see below why.

So to just quote myself here's why - taken from my post on Atari forum.

To clarify the purpose:

CT Raw has been the format before KryoFlux that everyone used to submit dumps to CAPS/SPS, so it's like over 10,000 games dumped for various platforms, including ST/E. There is a large number of existing dumps in this format, and KryoFlux also supports creating/converting images to it natively.
The format is also very compact when you take into account that it contains 5 complete revolutions sampled for each track as well as bitcell timing data.
This has been achieved with a very heavy delta compression specifically designed for bitcell data that allows further compression using a traditional archiver like zip afterwards - those wouldn't work very well on the exact same data before the CT Raw processing.
So it's the perfect choice for supporting both old (before KryoFlux) and new (after KryoFlux) images at the same time, as a first choice, and there is more to come

Please note there are existing images that are duplicatess, modified or bad and there are others which require work in the toolchain in order to release as IPF - not to mention we've been quite busy with software development recently.
In any of these cases there is no IPF, but of course for example a game being modified does not stop you playing it, it's just not suitable for preservation itself.
To resolve this conflict, and let people play any game regardless of preservation status and most importantly instantly, we decided to make it possible to use every dump ever made with any of our tools or hardware in all emulators using the library while we'll keep the very strict criteria for IPFs - so we hope it's a good solution for everyone and every need now

In short:
- Use your dumps instantly while you submit them to SPS for preservation
- Many games are modified etc, and could not be used to create IPFs - but we understand that you'd still love to use the images anyway.
- There is a very huge number of games in CT Raw format as old dumps and all new KryoFlux dumps can be converted into this format in a few seconds as well.
- Have some fun 8)

Enjoy

ps: These are raw files, and come with no warranty due to their nature. So PLEASE do not bug any emulator author if they do not work. It can happen, as it is the nature of the beast - but they do tend to work

[dlfrsilver]

Deep Core is using weakbits. I have noticed that Trex Warrior, my own RAW dump of it crash like if it was a copy.

[mr.vince]

Guess that uses weakbits and needs more than five different results. Don't think that will work from a raw dump.

[dlfrsilver]

well there is a possibility you're right

[mr.vince]

I know. Try Magic Lines. Will also fail. But work from IPF...

[Supamax]

Originally Posted by BarryB

Tried Dungeon Master dumped on my A1200/060 and it loads in WinUAE, not played enough to see if it crashes though! Uploaded to the zone as 'Dungeon Master.rar'

In the Zone? (I don't see it )

[BarryB]

Could have been removed for copyright reasons? Check PM!

[dlfrsilver]

Originally Posted by mr.vince

I know. Try Magic Lines. Will also fail. But work from IPF...

indeed, or otherwise Leavin teramis from thalion this bastard is doing actually
50 tests of track 13, you bet it's an intelligent idea.....

[IFW]

There is a very specific reason why many (all?) Thalion games using weak bits will not work from any kind of raw file. They read 50 (!!!) revolutions of a track containing weak bits, and expect to have different data read from ALL of the previous revolutions.
In other words: during the 50 revolutions sampled no data should repeat at all.
The reason this mostly works on the real hardware is that the AGC in the drive produces a fairly random pattern - but even that can fail to be random enough for this depending on your drive...
When you use a raw image, the emulator effectively plays back what has been recorded from the disk. If you for example record 5 revolutions, 5 different values will be read of at the places where the weak bits are, then the same 5 patterns will repeat again and again...
Whatever you do to get the raw images reading, such as software PLL simulation etc. is deterministic in nature - it's programmed to be like that.
The only way to get a protection like this working is pre-processing the data.
With a format like IPF the CODEC is capable of generating random enough data that satisfies the 50 different values criterion.
With a raw format this is of course impossible - the emulator sees what's been read from the disk once, and that won't change.
Under emulation you'd have to process the entire raw track data sampled, detect weak bit areas, and then instead of playing back the sampled values, randomized data should be played back.
Not exactly worth the effort... and can increase load time of a raw file (or track access time) significantly, ie introduce glitches and frame skips when the track scanning happens. So it's better to do it when loading the image, but then it takes time to find weak areas - and this can be very expensive processing if it's foolproof enough to not turn legit real data into weak bits

Hope this helps understanding what happens and why

[BarryB]

When we get scripting for DTC maybe that issue will go away?

[Retroplay]

Originally Posted by BarryB

Deep Core I couldn't get working, Disk 1 loads but when Disk 2 is inserted it just keeps checking track 0 in WinUAE and won't load.

Disk 1 loads perfectly fine and disk 2 is detected and loads but gets stuck on track 18 for me.

I tried the WHD installer using my raw dumps, disk 1 installs fine however it seems I have a bad disk 2 image as I get a checksum error on track 37. Oh well.

[BarryB]

I haven't got the disks to try another dump, have to wait till the weekend or next week!

Tried the WHDInstall on the CT dumps I made from the streams but get a CAPSLockTrack failed error?

[Galahad/FLT]

Great work guys, as always.

[dlfrsilver]

my own dump loads on disk 2 and 3. Have a look on these graphics which will tell you :

[IFW]

BarryB,
Scripting: Yes, since you'll create user-generated IPF files, and the IPF codec is capable of randomizing the data to the level needed for this to work.
Whdload: What are you exactly trying to do? Whdload uses IPF lib directly, but that's genuine Amiga code. You will have to use:
- genuine v5.0 Amiga library on real hardware
- use normal disk imaging with emulation instead of direct IPF access of WHDload, then the kind of image used will be completely transparent

If it's some other issue please post an exact repro and details

Denis,
So did you try them as CTR with WinUAE?
I've checked and your dump was not on the FTP - you uploaded about a dozen stream dumps in total...

[IFW]

Ok, I've checked BarryB's CTR files for Deep Core - they are completely different from the ones generated by my development version of DTC... and indeed Disk#2 is not recognized from the old one (my guess is the protection track), and everything loads perfectly from the CTRs generated by the new DTC

As for Dungeon Master: it's a copy, not an original disk. The only thing original on it is the protection track on 0.1
I guess someone tried to fix a broken original using another one... so I recommend trying to get hold of a CTR made from another disk

[Supamax]

Originally Posted by IFW

As for Dungeon Master: it's a copy, not an original disk. The only thing original on it is the protection track on 0.1
I guess someone tried to fix a broken original using another one... so I recommend trying to get hold of a CTR made from another disk

I tested it in WinUAE (last release).
It loads fine (I didn't try to save yet, though) but at a certain point it gave me the "Dungeon Master System Error #60".
Any idea if it's a read error or something else?

[mr.vince]

Hard to tell with DM.

That's the drawback of RAW. You never know if it's good. Is there a WHDLoad installer for it??