Non-standard disk formats`

[DanScott]

Ok, I never got into much regarding Amiga disk formats and my knowledge is fairly limited in this field.... I always used friends standard trackloaders, or trackloader systems available through work etc... so my question is:

What other disk formats other than MFM are usable on Amiga, how much data can one really fit on a standard floppy ???

[mcgeezer]

Probably one for Galahad this.

[Galahad/FLT]

Originally Posted by DanScott

Ok, I never got into much regarding Amiga disk formats and my knowledge is fairly limited in this field.... I always used friends standard trackloaders, or trackloader systems available through work etc... so my question is:

What other disk formats other than MFM are usable on Amiga, how much data can one really fit on a standard floppy ???

If you want a format that everyone can use whether it be real Amiga or WinUAE, you could do a basic MFM system that gives the same capacity as Rob Northens PDOS Protection system but with the ability to be written on a Real Amiga or WinUAE through extended ADF format.

So as you know, a standard Amiga formatted disk using 11 sectors can be 901120 bytes in size which is 160 tracks, 1760 sectors, 1760*512=901120/$dc000

Most typical not very exotic MFM formats can give you 12 sectors which
in total is 983040 bytes which is 160 tracks, 1920 sectors.
1920*512=983040/$f0000

There are bigger capacity formats such as Factor 5's, and the Psygnosis MFM format, but people are unlikely to be able to copy them.

But the good news is some of these MFM formats have been disassembled and are known, Rob Northens PDOS writing code is out in the wild so that can be used, and WinUAE can read from extended ADF files.

Otherwise if you don't want that kind of bollocks, you could stick with usual sector loading AmigaDOS disk but utilise Tracks 81 and 82 which most drives should be fine with.

If you really want to eek every last byte out of a disk, you could utilise something like Gaston / Fairlights loader which allows you to read at any offset into the disk instead of $200 sector offsets.

Pretty sure Ross on this forum has coded his own as well, i've got one but its not very reliable.

But if you want reliable access, quick disk loading, and more importantly no faffing on the user end with MFM formats, Rob Northens standard Sector loader that you used on Heimdall will do exactly what you want, but its 901120 bytes in size and no bigger.

[mr.spiv]

I never did anything but MFM with custom sync words.. although, the format I used in "Smarty and the Nasty Gluttons" had $18a0 bytes per track for user data (access offset alignment was 4 bytes) excluding the first track that had to be in normal Amiga format. That gave 1007968 bytes plus then the whole disk was compressed (Stacker-like loading i.e. each track was decrunched while DMA was loading the next one) so I guess we were +1.5MB in a typical case.

[ross]

The only possible way to break the ~1000000 bytes limit is to overcome the MFM format (without resorting to long tracks).
I've experimented it in late '80 using four consecutive 0 in raw stream (instead of the max three of standard MFM) and somehow worked (two consecutive 1 bits failed miserably).
So something similar to the M2FM format.
It's long time ago but if I don't remember badly more than 25% more data is possible to stock on track.

Problem here is that is no more possible a fast deMFM routine because no more a (integer) 2:1 ratio and properly 'bit interlacing', but you need to resort to something more similar to a GCR format, requiring LUT and unconfortable bits grouping (shift, mask, reconstruction).
Even the read sync start position can be problematic if you use all the quasi-MFM bit combo.
Moreover you need to take account for drive speed drifting that can lead to read error, indeed a good write drive is needed (no problem anyway if the write drive is the same as the write one, and obviously no problem on 'solid drive' or emulated).

I'm pretty sure some Psygnosis games have used something like this.


PS: and yes, I've used a 'free-position' loader for standard format.

[Retro1234]

XFloppy on Aminet can store 1.12mb and DiskSpare 984kb

[ross]

Originally Posted by Retro1234

XFloppy on Aminet can store 1.12mb..

Interesting, I'm not aware of this XFloppy.device

Seems to use 14 sectors/track:

#define XFD_TRACKSECTORS	14

so is fully coherent with my 25% increase and I'm pretty sure is using a similar technique.

This anyway is not encouraging:

This is a development version, it may have bugs
Particularily the reads and writes are pretty unreliable
and r/w errors are quite often.

I wonder if I can find my old sources and tests somewhere, but without a real machine do not make much sense..

[roondar]

Originally Posted by ross

The only possible way to break the ~1000000 bytes limit is to overcome the MFM format (without resorting to long tracks).
I've experimented it in late '80 using four consecutive 0 in raw stream (instead of the max three of standard MFM) and somehow worked (two consecutive 1 bits failed miserably).
So something similar to the M2FM format.
It's long time ago but if I don't remember badly more than 25% more data is possible to stock on track.

Problem here is that is no more possible a fast deMFM routine because no more a (integer) 2:1 ratio and properly 'bit interlacing', but you need to resort to something more similar to a GCR format, requiring LUT and unconfortable bits grouping (shift, mask, reconstruction).
Even the read sync start position can be problematic if you use all the quasi-MFM bit combo.
Moreover you need to take account for drive speed drifting that can lead to read error, indeed a good write drive is needed (no problem anyway if the write drive is the same as the write one, and obviously no problem on 'solid drive' or emulated).

I'm pretty sure some Psygnosis games have used something like this.

PS: and yes, I've used a 'free-position' loader for standard format.

While I think these kind of things are super cool and interesting, I do wonder whether they are an acceptable compromise or not. As in, does the extra space compensate for the lower reliability?

It's an interesting question. Maybe more so now that most Amiga floppy drives (and most/all 3.5" DD floppies) are so old and probably operating closer to their tolerances than normal.

Personally, for now I'll stick with a basic trackloader and a good cruncher. That'll also give you plenty of "extra space"

[NorthWay]

I was working on an answer on my old pc, but it died on me (new parts incoming).

Short version: Can you force GCR to work with 2ms? If not then forget that.
Other option: Don't encode and decode the regular way where you blindly throw away half the bits. If you look at the MFM rules and what are "legal" bit patterns you will find that for some fixed number of bits (say 16) the set of possible bit combos are >256 (depending on what your "previous bits" starting condition is).
This isn't a fixed maximum size but will depend on your data. I guess the way to encode it is to look at it as some kind of optimum uuencode algorithm (that was a thing back in the NetNews days - how many valid characters were there, and how could you pack your data into those most efficiently). Possibly it shares traits with arithmethic compression in using fractional bits?
I never got my head around how to do it in real life, it became a thought experiment.
The size of your encoded data can be bigger than 16 bits, but you will eventually run into either a memory issue if you build tables for bit matching, or a speed issue if you loop through all possible combos for N bits until you match the encoded input. The more bits you look at the more savings you gain.

TLDR: MFM en/de-coding is AFAICT wasteful compared to the rules it lays down.

[ross]

Originally Posted by NorthWay

Short version: Can you force GCR to work with 2ms? If not then forget that.

Yes, but data read/written this way is unreliable.
GCR encoding works only if flux transitions are enough spaced, permitting more changes in the defined time.
Much better stick to 2ns MFM.

Other option: Don't encode and decode the regular way where you blindly throw away half the bits. If you look at the MFM rules and what are "legal" bit patterns you will find that for some fixed number of bits (say 16) the set of possible bit combos are >256 (depending on what your "previous bits" starting condition is).
This isn't a fixed maximum size but will depend on your data. I guess the way to encode it is to look at it as some kind of optimum uuencode algorithm (that was a thing back in the NetNews days - how many valid characters were there, and how could you pack your data into those most efficiently). Possibly it shares traits with arithmethic compression in using fractional bits?
I never got my head around how to do it in real life, it became a thought experiment.
The size of your encoded data can be bigger than 16 bits, but you will eventually run into either a memory issue if you build tables for bit matching, or a speed issue if you loop through all possible combos for N bits until you match the encoded input. The more bits you look at the more savings you gain.

Yes, this is the basis for pushing MFM limits (you can also add four 0 in a row to the encoding breaking the rules, that usually works also).

[Wepl]

I don't see why other encodings than MFM should cause reliability problems as long as not more that one 1 and three 0 are consecutive encoded.
Important is, that a possible sync word remains available.

[NorthWay]

This here was my back-of-an-envelope scribbling about what are the legal encodings for 16 encoded bits of data http://eab.abime.net/attachment.php?...4&d=1539481771

[ross]

Originally Posted by Wepl

I don't see why other encodings than MFM should cause reliability problems as long as not more that one 1 and three 0 are consecutive encoded.
Important is, that a possible sync word remains available.

Sorry Wepl, I don't know if you're referring to my message, but I never said otherwise.
GCR require more than one 1 in a row, so it's practically impossible for it to work with 2ns cells (on Amiga).

Of course if you write an encoding that respect RLL (1,3) codes you do not have any reliability problems.

EDIT:
It's probably only a matter of semantics, when we talk about MFM encoding we usually associate with RLL (1,3), that we know used for encoding on magnetic surfaces, .
Indeed, one should not speak of strict MFM if some change are made (like the SYNC word ); it would be more correct to call it other RLL (1,3) encoding.
The situation is different if the rules are broken, using for example an RLL (1,4) encoding.

[ross]

Originally Posted by NorthWay

This here was my back-of-an-envelope scribbling about what are the legal encodings for 16 encoded bits of data http://eab.abime.net/attachment.php?...4&d=1539481771

No, this cannot work, you need to exclude those who break rules when chained.
Or I'm missing something?

[Wepl]

Originally Posted by ross

Sorry Wepl, I don't know if you're referring to my message, but I never said otherwise.

Sorry, no. Was meant in general.

Originally Posted by ross

It's probably only a matter of semantics, when we talk about MFM encoding we usually associate with RLL (1,3), that we know used for encoding on magnetic surfaces, .
Indeed, one should not speak of strict MFM if some change are made (like the SYNC word ); it would be more correct to call it other RLL (1,3) encoding.

agreed

[ross]

A tentative, first idea that came to my mind.

Use y bits for the pure encoding (14 in my example) and 1 or 2 bits for the chain, this way:
0xxxxxxxxxxxx0, chain bits: 1 if 0X, 10 if 1X
0xxxxxxxxxxxx1, chain bits: 0 if 1X, 01 if 0X
1xxxxxxxxxxxx0, chain bits: 1 if 0X, 10 if 1X
1xxxxxxxxxxxx1, chain bits: 0 if 1X, 01 if 0X

Worst case 16 bits used for A encoding.
Usable value A=$13E versus $100 for strict MFM.

This of course is a fake value because I've to subtract all the values that combined in the raw stream give me a SYNC.
And another problem is how to 'unpack' this A value to something usable

Code:

    lea rll13(pc),a0
    moveq   #0,d0
    moveq   #0,d5

.st moveq   #-2,d2
    moveq   #-4,d3
    move.l  d5,d1
    moveq   #14-1,d4
.ll add.w   d1,d1
    bcc.b   .cc
    moveq   #-4,d3
    addq.w  #1,d2
    beq.b   .ex
    bra.b   .pp
.cc moveq   #-2,d2
    addq.w  #1,d3
    beq.b   .ex
.pp dbf d4,.ll
    move.w  d5,(a0)+
    addq.w  #1,d0
.ex addq.w  #4,d5
    bne.b   .st
    rts
    
rll13   ds.w $13E

Well, this is just a proof of concept, nothing else and not usable.
Correct me if I made some obvious mistakes

EDIT: well, the obvious mistake is that I cannot decode the data because I've no way to know where the 'grouped' bits start after chained bit(s)
But excluding some A values and selecting a proper chain bit (or two?), making it a constant bit(s) quantity, could solve?
I have to think about it

[robinsonb5]

How about having two or maybe even 4 different encodings for each possible byte value, grouped on where the first 1 falls? After encoding each byte you select an encoding for the next byte depending on the position of the last 1 in the byte just encoded.
The decoder would have to do the same, select a decoding for the next byte based on the last 1 of the byte just decoded.

(Finding a suitable sync value could be tricky, though!)

[NorthWay]

Originally Posted by ross

No, this cannot work, you need to exclude those who break rules when chained.

Hm?
That was an example of looking at 16 encoded bits when the three previous ones were all '0'.
Three previous 0 bits is a valid prequel to any random point in a bitstream from the floppy. I think I then enumerated all the valid combinations of 16 encoded bits that follow three 0s.
Which by my account totals up to 256+14(==270). This is 5,4% more than the encoding space utilized in standard mfm. The '000' prefix is also IIRC the worst case as the 4 other will give around 300 combinations AFAIR.
I don't know how common each prefix is, but it would be dynamically varying depending on the source data.

As I said, the more bits you look at the more combinations there are compared to just halving it. I started with looking at 8 instead of 16, but that gave a potential for getting _fewer_ combinations than the normal encoding (because of the pessimal '000' prefix).
Decoding 16 bits from table searches would require something like 5*(16/8)*300 bytes, i.e. 3K, and that is a table you can build on the fly. I haven't done the math on how it would grow with longer bit patterns.

[ross]

Originally Posted by NorthWay

Hm?
That was an example of looking at 16 encoded bits when the three previous ones were all '0'.
Three previous 0 bits is a valid prequel to any random point in a bitstream from the floppy. I think I then enumerated all the valid combinations of 16 encoded bits that follow three 0s.
Which by my account totals up to 256+14(==270). This is 5,4% more than the encoding space utilized in standard mfm. The '000' prefix is also IIRC the worst case as the 4 other will give around 300 combinations AFAIR.
I don't know how common each prefix is, but it would be dynamically varying depending on the source data.

As I said, the more bits you look at the more combinations there are compared to just halving it. I started with looking at 8 instead of 16, but that gave a potential for getting _fewer_ combinations than the normal encoding (because of the pessimal '000' prefix).
Decoding 16 bits from table searches would require something like 5*(16/8)*300 bytes, i.e. 3K, and that is a table you can build on the fly. I haven't done the math on how it would grow with longer bit patterns.

Sorry, I fail to understand.
Yes that 000 is a valid prequel, but how you can chain this? (example taken from your data):
000 1000100010101001
10001000101010011000100010101001 is impossible..

[ross]

In the meantime my second try.
Main idea is to use the MSB as the chain bit and use the LSB as a LUT selector (is this an idea similar to that of robinsonb5? I did not understand properly what he meant.., sorry).

Code is practically the same but splitted for MSB:

    lea rll13_1(pc),a0
    moveq   #0,d0
    moveq   #0,d5

.st moveq   #-2,d2
    moveq   #-4,d3
    move.l  d5,d1
    moveq   #14-1,d4
.ll add.w   d1,d1
    bcc.b   .cc
    moveq   #-4,d3
    addq.w  #1,d2
    beq.b   .ex
    bra.b   .pp
.cc moveq   #-2,d2
    addq.w  #1,d3
    beq.b   .ex
.pp dbf d4,.ll
    move.w  d5,(a0)+
    addq.w  #1,d0
.ex addq.w  #4,d5
    bpl.b   .st
    
    lea rll13_2(pc),a0
    moveq   #0,d0
;    moveq   #0,d5

.s2 moveq   #-2,d2
    moveq   #-4,d3
    move.l  d5,d1
    moveq   #14-1,d4
.l2 add.w   d1,d1
    bcc.b   .c2
    moveq   #-4,d3
    addq.w  #1,d2
    beq.b   .e2
    bra.b   .p2
.c2 moveq   #-2,d2
    addq.w  #1,d3
    beq.b   .e2
.p2 dbf d4,.l2
    move.w  d5,(a0)+
    addq.w  #1,d0
.e2 addq.w  #4,d5
    bne.b   .s2

    rts
    
rll13_1   ds.w $bd
rll13_2   ds.w $81

This guarantee a proper decoding, better than strict MFM but unbalanced, practically unusable..