Catweasel Driver / Updates

[Schoenfeld]

Originally Posted by Borg_Number_One

These points: "Catweasel and IPF-support" / "Catweasel and copy protected floppies"
...have been discussed very often (you can use the search in the forum feature),
but no decent solution or no: "The Individual Computers team is working on adding the wanted features." -statement have been reported in the past / in the forums.

I'm happy to have my team implement IPF write support into the Imagetool as soon as we get a documentation of the file format. I have repeatedly made this statement and I'm happy to repeat it one more time. Or is it me who is now missing something? Is there an API that allows us to extract the Paula-datastream including a "beginning" and "end" of track, so we can translate that into Catweasel-write commands? Last time we checked, it was all closed.

Jens

[whiteb]

Originally Posted by Schoenfeld

I'm happy to have my team implement IPF write support into the Imagetool as soon as we get a documentation of the file format. I have repeatedly made this statement and I'm happy to repeat it one more time. Or is it me who is now missing something? Is there an API that allows us to extract the Paula-datastream including a "beginning" and "end" of track, so we can translate that into Catweasel-write commands? Last time we checked, it was all closed.

Jens

I believe that is the contentious issue, unless you use one of the API's from CAPS/SPS then you cannot. (Windows, Mac, Amiga, Linux).

I am not a programmer so I would not know, but given the Floppy Emulator (HxC ?) now starting to implement copy protected floppy support through IPF, I am sure it supports the proper commands that would be present in a REAL disk, like End of Track/Start of Track data.

Winuae Fully supports IPF, HxC supports *SOME* of the protections.

[Jeff_HxC2001]

Originally Posted by whiteb

Winuae Fully supports IPF, HxC supports *SOME* of the protections.

A little correction :

Winuae Fully supports IPF.
USB HxC Floppy Emulator Fully support IPF (since 25/03/2008).
SDCard HxC Floppy Emulator supports *SOME* of the protections (hardware limitation).

[Dic_Ray]

@Schoenfeld

Dear Jens,

at http://www.softpres.org/download there are the developer informations for Windows, Linux, Mac and Amiga.

Here I summarize the links for Windows:
Developer Distribution
http://www.softpres.org/_media/files...ad&cache=cache
http://www.softpres.org/_media/files...ad&cache=cache

Access API
http://www.softpres.org/_media/files...ad&cache=cache

User/Developer Reference Documentation
http://www.softpres.org/_media/files...ad&cache=cache

As I'm absolutely no developer, I can't guess, if these informations include all the needed details. But if something is missed, you could ask for a certain hint.

Could you please explain, what special information or porgram code do you need? and btw with which information perhaps you could support the KryoFlux-project. What's the reason you don't like this USB-solution?

Hope to see you in Essen and have one or two beers with you or something else. I would be glad, to communicate with you as one of the last Amiga geniuses.

Best regards
Dic

Btw I like Catweasel and I like KryoFlux. I love to see a cooperation - to offer one futureproof solution to handle easily all types of disks with modern hardware.

[Schoenfeld]

We're aware of the DLL that softpres has published. However, for writing things back to a disk, it's not an option to re-convert the data for every rotation of the disk. You can only write oncce. In order to use the DLL to create a real disk, I'd have to know which parts are playing around with protection stuff.

I could currently only do that by "reading" multiple rotations and analyzing what I get. So in essence. the information (flux transitions) is there in the file, it'll be encoded through the softpres DLL, then I'd have to decode it, translate it back to physical flux transitions and write it back to disk. This is an error-prone process that could be avoided by adding one function to the softpres DLL: Extract bit stream from a track with a fixed bit rate, fixed start and end (which is also the wrap-around), and we'll do the rest.

The HxC floppy emulator for USB does nothing else than the UAE emulator: It gets the data new from the DLL for every single rotation. That's easy to do if you have the freedom to constantly "ask" the DLL what the track looks like "on this rotation". However, writing a disk means that you have to write once and with that single write, create a track that reads different data on every rotation. We know how to do that (within physical limits), but we don't know how to get the information from the IPF image.

To be honest, I don't even know if the info is really stored in the IPF, as it's a lot easier to brute-force it in by storing multiple images of a single track. If that's the case, we wouldn't get around the analysis step that I described above. Could someone from the softpres team comment on this?

Jens

[IFW]

Yes, that is a legacy API, that was supervised a long time ago.
For example generating multiple-revolutions seemed like a good idea at the time (2003?), but once you see games using weak bits that verify each rotation to be different 50 times it's not so good anymore

IPF has been designed for writing from the start, so unlike say pasti/stx files how to generate the data is stored in the file - not what comes out of it when reading.
In other words: the cause is stored, not the result.
The original IPF files have 1 data stream with a low level scripting desicribing logical areas on a track. This for each track.
Any new images since multi-platform support has been added (any image starting with #3000 StarGlider 2) have 3 streams describing data, and 2 contexts per logical block. This for any number of blocks per track, for any number of tracks.

More details how that works can be read at the WIP reports on our site.

[Jeff_HxC2001]

Originally Posted by Schoenfeld

The HxC floppy emulator for USB does nothing else than the UAE emulator: It gets the data new from the DLL for every single rotation. That's easy to do if you have the freedom to constantly "ask" the DLL what the track looks like "on this rotation". However, writing a disk means that you have to write once and with that single write, create a track that reads different data on every rotation. We know how to do that (within physical limits), but we don't know how to get the information from the IPF image.

No the USB HxC floppy doesn't work like this : Physical datas are exctracted from the IPF at the loading time. During the emulation time the CAPS DLL is not called.

It's funny because i have just explain this in this thread :

http://eab.abime.net/showpost.php?p=683384&postcount=14

The windows app is the same for USB & SD version:
For each file image type supported there are a "plugin" able to load IPF or ADZ file for example.
Tracks data are generated (ADF/ADZ...) or extracted (IPF/SPS) at the loading time.
This generated/extracted information are a physical description of the floppies : Bitstream/bitrate/flakey bits...
Once you get this, you can send/export this data to the USB Floppy Emulator / to a file that the SD HxC Floppy Emulator can understand, or to another file type (*.AFI / *.MFM / *.IMG...)
Depending of the target, this last process may implied some loss of data : For example the SD Card floppy emulator file format doesn't store the bitrate stream. IMG file store only sector data (like ADF file). On the contrary the AFI file format keep all data.

For more details have a look to this:
http://hxc2001.free.fr/floppy_drive_...lator_soft.zip
http://hxc2001.free.fr/floppy_drive_...r_soft_src.zip

So i confirm that you can extract all datas from the IPF file without problem to use data without using the CAPS DLL later.
For example in the HxC Floppy Emulator context you can convert IPF file into AFI (a custom file format) and use AFI file without the need of the caps lib.

You can try this with the hxc software:
http://hxc2001.free.fr/floppy_drive_...lator_soft.zip

[fiath]

Jens, we've now looked into the sampling point you mentioned previously, and we cannot see anything on the read route of a disk drive that would indicate a clock running at 4 MHz. Perhaps this might be the case on the older 8" disk drive you mention, but we don't think that it is true that you are limited to 8 MHz sampling for the majority (if not all) more recent floppy
disk drives. No breaking of Physics required.

We need to know what was originally written by a duplicator exactly, not what a normal FDC can read (which is limited by it's clock speed, unlike a disk drive). So we need the high resolution to determine the duplicator's "finger prints" and work out the scripting used - we're not making simple bit images.

[kipper2k]

Originally Posted by Schoenfeld

I'm happy to have my team implement IPF write support into the Imagetool as soon as we get a documentation of the file format. I have repeatedly made this statement and I'm happy to repeat it one more time. Or is it me who is now missing something? Is there an API that allows us to extract the Paula-datastream including a "beginning" and "end" of track, so we can translate that into Catweasel-write commands? Last time we checked, it was all closed.

Jens

Hi Jens,

I'm curious as to exactly what you want to implement with the ipf format, are you looking to just try to read the ipf images to extract the actual track & sector data or to try to reproduce the non standard data protection too?

Is the catweasel hardware in its present form capable of reproducing protections ?

[Toni Wilen]

Originally Posted by kipper2k

Is the catweasel hardware in its present form capable of reproducing protections ?

I don't see any reason why WRITING any kind of protection isn't possible with Catweasel but READING may not be so simple.. (can't read very long pulses, can't read exact position of index pulse, maybe capability to read multiple revolutions without gaps, anything else?)

[whiteb]

Originally Posted by Jeff_HxC2001

A little correction :

Winuae Fully supports IPF.
USB HxC Floppy Emulator Fully support IPF (since 25/03/2008).
SDCard HxC Floppy Emulator supports *SOME* of the protections (hardware limitation).

Ahhhh so *ALL* protection schemes that IPF support, are supported in the USB model then ?

Okay good to know

[Schoenfeld]

Originally Posted by Toni Wilen

I don't see any reason why WRITING any kind of protection isn't possible with Catweasel but READING may not be so simple.. (can't read very long pulses, can't read exact position of index pulse, maybe capability to read multiple revolutions without gaps, anything else?)

Position of the index pulse is stored within the same data stream, but you have to enable that. This should be obvious from the documentation that you have access to. If you think that it's not precise enough, you should know that from drive to drive, you have variation of up to 8ms in the position of the Index pulse. Anything more precise than "per flux transition" does not carry more information.

As for reading extremely long pulses, even the old cores can be used to find out if any pulse is triggering the overflow bit. I haven't found a single disk that does, but that may be related to me mostly owning the original programs that were made with my protection.
So let's assume you know there's an area of the disk that triggers overflow, then you can re-read it with a higher counter rate and no overflow. Since memory is limited to 128k, very powerful timers have been introduced with the MK4, and they allow precise start and stop points for investigating single windows within a track. Again, these possibilities go beyond the information content of the data stream.

Toni, you should have access to the latest Inside_MK4 textfile (over 35kbytes), dated november 27th, 2006. Timers however were introduced in january 2005, and everyone on the developer's list was excited about their possibilities. Yet, to this day, nobody actually made use of them. It's even worse: The one person in this thread who has access to the Catweasel documentation also believes that the Catweasel has more limitations than it actually has.

I wonder what went wrong. Is the documentation that complicated?

Jens

[Schoenfeld]

Originally Posted by kipper2k

Is the catweasel hardware in its present form capable of reproducing protections ?

Yes, it is.

Jens

[Toni Wilen]

Position of the index pulse is stored within the same data stream, but you have to enable that. This should be obvious from the documentation that you have access to. If you think that it's not precise enough, you should know that from drive to drive, you have variation of up to 8ms in the position of the Index pulse. Anything more precise than "per flux transition" does not carry more information.

As far as I know there are copy protections that have no pulses near index (longer than 8us) and if I understood the documentation correctly, index mark (bit 7) is set when next pulse counter value is written, not when index pulse actually was detected. This is also the situation where longer pulse counts would be nice (can't they be chained like in write mode?)

Originally Posted by Schoenfeld

I wonder what went wrong. Is the documentation that complicated?

Documentation is fine.

IMO no one wants to do that kind of micro-management, joining pieces of track is major pita, requires some advance information about track contents (so that you don't miss anything important).

This simply shouldn't be necessary.

btw, could IDX2IDX write also return last write position? Would be easy to see if buffer under or "overflowed" (whatever the reason) without re-reading the track.

[Schoenfeld]

Originally Posted by Toni Wilen

As far as I know there are copy protections that have no pulses near index (longer than 8us) and if I understood the documentation correctly, index mark (bit 7) is set when next pulse counter value is written, not when index pulse actually was detected. This is also the situation where longer pulse counts would be nice (can't they be chained like in write mode?)

Hmm... you want a continued count with just a single byte. I never followed up on that, as no drive behaves like another, even if it's the same brand, same type, same series and same disk: As soon as there's more than five 0's in a row, random flux transitions are inserted, because the signal is floating around zero. I have to check if the predecode-mode will give you an infinite amount of 0's (and I actually have to check if I ever documented the predecode-mode). One thing I know is that counting is started at Index, no matter if there's a flux transition or not, so you never miss a start.

Originally Posted by Toni Wilen

btw, could IDX2IDX write also return last write position? Would be easy to see if buffer under or "overflowed" (whatever the reason) without re-reading the track.

It does: After writing is stopped on the next IDX pulse, the memory counter stays at it's present position. The position can be read. It's the same registers that set the watermark value, they have a different meaning on read and write. These registers are located in the IO register bank - the one that also has all the timer registers (and obviously the one that is hardly ever touched).

Jens

[Toni Wilen]

Originally Posted by Schoenfeld

Hmm... you want a continued count with just a single byte. I never followed up on that, as no drive behaves like another, even if it's the same brand, same type, same series and same disk: As soon as there's more than five 0's in a row, random flux transitions are inserted, because the signal is floating around zero.

Does drives really try to "analyse" pulses? (except by filtering it so that same pulse isn't accidentally detected twice etc) Floppy controllers surely do above but "normal" disk drives?

In my tests I don't get any ghost pulses (perhaps some drives do, perhaps some older than 3.5" drive types?) but they do filter out pulses that are too closely together.

Anyway, copy protections are doing "illegal" things which means reading them also requires "illegal" things

I have to check if the predecode-mode will give you an infinite amount of 0's (and I actually have to check if I ever documented the predecode-mode).

predecode does not appear to be documented but I don't want any kind if cooked data.

One thing I know is that counting is started at Index, no matter if there's a flux transition or not, so you never miss a start.

ok, this is still confusing..

...P....I......P...
   1    2      3

P = flux transition pulses from drive
I = index

Reading has been started previously and above position is coming: Can I get both distance between 1 to 2 and 2 to 3? And how does the data appear in read stream? (lets say 1 to 2 is 80 cycle units and 2 to 3 is 100 cycle units)

It does: After writing is stopped on the next IDX pulse, the memory counter stays at it's present position. The position can be read. It's the same registers that set the watermark value, they have a different meaning on read and write. These registers are located in the IO register bank - the one that also has all the timer registers (and obviously the one that is hardly ever touched).

Ugh, thats quite ugly way to do it. Read with IOCTL and then poke the hardware to get extra data... (and probably also difficult to support in possible future firmware/hardware versions)

Please add OUT parameter to IDX2IDX IOCTL, thanks

[Schoenfeld]

Originally Posted by Toni Wilen

Please add OUT parameter to IDX2IDX IOCTL, thanks

It would only work with MK4 and up. Until today, we have avoided adding such functions, but it seems at some point we have to break with legacy controllers.

Also, please note that you're talking to the hardware guy. You're requesting software stuff here, and I have to talk to the team before I can give an answer on this. I'll do that on monday. Sounds simple enough, but I would still like to keep it compatible with older controllers (hmm.. so much for my attitude).

Jens

[fiath]

This is exactly the kind of conversation we wanted to have with you years ago. This is a much better and more productive way to progress . Hopefully this new you is permanent.

[Dic_Ray]

I really like the new way of communication. So thanks to all.