IDE & Max Transfer

[alexh]

I was replying to someone over at Amiga.org about MAX TRANSFER settings for IDE controllers (such as A1200/A600 Gayle IDE) giving them the standard "try 0x1FE00 and if that doesnt work try 0xFE00".

While writing I accidentally worked out the logical reason behind these "Magic numbers".

• IDE commands (ATA5 or lower) have "sector count" field which is 8-bits.
• SCSI commands (SCSI10) have a "transfer size" field which is 16-bits.

For those who do not know, a sector is the smallest individual amount you can access on a Hard drive and has always been (until recently) 512 bytes. And "sector count / transfer size" is the amount to be transferred in a single command.

I noticed that "0x1FE00" is 255 * 512, and that 255 is the largest value that can be represented by 8-bits. So by using "0x1FE00" it prevents the driver (scsi.device) creating malformed IDE commands with sector counts larger than 8-bit.

I am positive this is the logical answer as to why MAX TRANSFER should be "0x1FE00" for IDE devices.

------------

Now while the IDE (ATA5 and lower) command set supports an 8-bit sector count, very few operating systems use the maximum value. Windows (and most other operating systems) tend to use 64 or 128 sector commands. I know this because I discovered at work several years ago that PATA hard drives have their cache optimised for 64 sectors while SATA were optimised for 128 sector commands.

Back to the plot:

The second magic number touted for MAX transfer "0xFE00" is 127 * 512. I am not sure exactly why this value would be more compatible than 0x1FE00, but I imagine it has to do with the fact poor quality IDE devices do not expect sector counts above 128 sectors.

You pretty much guarantee "sector count" compatibility with any IDE device with 0xFE00 and I cannot believe that it is slower.

Anyhow, hope that has explained things, rather than confused things.

[prowler]

That's a useful display of detective work, alexh!

Thanks for sharing your thoughts on this with us.

[alexh]

I dunno if my explanation is dumb enough.

You still have to be reasonably computer literate (at a low level) to get it.

[Eclipse]

That's very clear and actually explains a lot.

[Zetr0]

I think it should be added to the CF-IDE for the Amiga Wiki...... if we have one....

[Leandro Jardim]

Thanks alexh ! I put the magic number in the RDB of my hardfile, and now it works much better!

[mech]

Nice one alex,
another good reason to use SCSI on amiga also

[Jope]

Added Alexh's post to Amiga Problems.

[Rixa]

Nicely reasoned, and makes sense.

However, when I was setting up my CF card, I went with what looked like the most reliable-looking advice I could find at the time and set the Max Transfers value as high as 0x1FE0000. At 256 times more than reasoned above, it seems quite absurd.

The magic is it has worked just fine, and apparently also in some cases where 0x1FE00 hasn't. I would like to know why it works, whether it is just waiting to cause trouble, or if there is any other reason to change it. I was thinking perhaps it is not taking the entire 32-bit value into account, but the less significant word is all zeroes so wouldn't it likely result in a MaxTransfer of 0 then?

[Loedown]

Originally Posted by Rixa

Nicely reasoned, and makes sense.

However, when I was setting up my CF card, I went with what looked like the most reliable-looking advice I could find at the time and set the Max Transfers value as high as 0x1FE0000. At 256 times more than reasoned above, it seems quite absurd.

The magic is it has worked just fine, and apparently also in some cases where 0x1FE00 hasn't. I would like to know why it works, whether it is just waiting to cause trouble, or if there is any other reason to change it. I was thinking perhaps it is not taking the entire 32-bit value into account, but the less significant word is all zeroes so wouldn't it likely result in a MaxTransfer of 0 then?

If I understand correctly CF cards have a block transfer mode and as long as you stay under the maximum limit for their block transfer and your Amiga has enough ram to accomodate the block before writing then all should be good. 256 bytes and 512 bytes are older block sizes, C64 disks use a 256 byte sector for example, keep in mind when the hardware came out 512 byte sector size was fine and now we have legacy keeping it alive.

[thomas]

Actually there shouldn't be a problem with MaxTransfer. The device driver should know the hardware limitations and automatically split large packets into smaller ones. Just like ping and MTU in TCP/IP world.

But scsi.device does not do this. If you specify a MaxTransfer value larger than 0x1fe00, it will send a mangled command to the device, resulting in corrupted data. So it is important to set MaxTransfer to a reasonable value when using scsi.device.

However, it also depends on the version of scsi.device which is used. For example when using IDEfix, a new scsi.device is used which does automatically split large packets. So with IDEfix MaxTransfer should not be an issue.

Finally MaxTransfer issues only arise when using files larger than the critical limit. Files smaller than 127.5K are not affected at all. Similarly large files might not be affected if a copy program is used which does not attempt to write more than 127.5K at a time.

[Gordon]

Where can I set the max transfers properly? I have tried HD tool box but it doesnt save my changes? im trying to put 0x1fe00 but it kees changeing. Do i need to change the mask as well?

[Retro-Nerd]

You have to press the enter button after typing 1fe00, then save the settings.

[Jope]

No need to adjust the mask.

[Gordon]

Thank you Retro-Nerd for the very quick response, that worked. I cant believe thats what I was missing!

Ok Jope thanks.

Originally Posted by alexh

I was replying to someone over at Amiga.org about MAX TRANSFER settings for IDE controllers (such as A1200/A600 Gayle IDE) giving them the standard "try 0x1FE00 and if that doesnt work try 0xFE00".

While writing I accidentally worked out the logical reason behind these "Magic numbers".

• IDE commands (ATA5 or lower) have "sector count" field which is 8-bits.
• SCSI commands (SCSI10) have a "transfer size" field which is 16-bits.

For those who do not know, a sector is the smallest individual amount you can access on a Hard drive and has always been (until recently) 512 bytes. And "sector count / transfer size" is the amount to be transferred in a single command.

I noticed that "0x1FE00" is 255 * 512, and that 255 is the largest value that can be represented by 8-bits. So by using "0x1FE00" it prevents the driver (scsi.device) creating malformed IDE commands with sector counts larger than 8-bit.

I am positive this is the logical answer as to why MAX TRANSFER should be "0x1FE00" for IDE devices.------------

Actually, max sector count by IDE (ATA) is 256 sectors, because 0 in sector count registers means not 0 (nonsense) but 256 .

With 48-bit LBA (what is necessary with drives over 128GB) max count is 65536 as sector count is given as 16-bit value.

[alexh]

Originally Posted by AmigTart

Actually, max sector count by IDE (ATA) is 256 sectors, because 0 in sector count registers means not 0 (nonsense) but 256 .

Agreed, but I didn't want to confuse people. The largest number of sectors possible in a non zero 8-bit number is still 255 which corresponds to MAX TRANSFER of 0x1FE00 bytes.

Originally Posted by AmigTart

With 48-bit LBA (what is necessary with drives over 128GB) max count is 65536 as sector count is given as 16-bit value.

The recent patches to the Amiga IDE scsi.device which adds ATA-6 command set (48-bit LBA) may have increased the sector count to 16-bit but I don't know. Can anyone confirm?

You would need this version of scsi.device in your Kickstart ROM or eFlash/Kickflash/Deneb to be able to boot from a device with a the MAX TRANSFER set to larger than 0x1FE00. This is going to be a VERY small set of people.

I thought that I had used the scsi.device (45.45) with 48-bit LBA support using a MAX TRANSFER larger than 0x1FE00 with a SanDisk CF card and had data corruption. I will be able to confirm soon as I'm making my own KS3.9 EPROMs for my A4000

[rkauer]

The real problem is not the sector count, but the fact compact flash & microdrives doesn't have a built-in buffer .

Notice that real HD doesn't suffer from this issue, from the old IDE to SCSI drives.

So anything over 255 will choke in the Amiga IDE bus (read "Gayle"), who will still try to put more data than the slow IDE channel can really cope.

[alexh]

???? How did you work that out? I was sure the problem is about malformed commands. The Amiga tries to transfer more data on the bus than the drive is expecting.

AmigaOS tries to send a command for say 131584 bytes (257 sectors). It made sense that scsi.device can only send ATA commands with an 8-bit sector count (and it doesn't check the incoming request size) which gets converted into an ATA command of 1 sector.

Originally Posted by rkauer

Notice that real HD doesn't suffer from this issue

With a MAX TRANSFER above 0x1FE00 I get data corruption on Gayle IDE using 2.5" PATA IDE disks just as I do with CF cards.

I imagined it was not a problem for SCSI drives because SCSI commands have a bigger (32-bit?) sector count field.

[Retro-Nerd]

Really? I never had any issues with standard MaxTransfer settings and PC hard drives. Pure luck?

[alexh]

Using the onboard IDE?

[Retro-Nerd]

Yes.

[alexh]

Well maybe rkauer is right that it is not a sector count problem.

I cannot imagine his description of what could be going wrong is the truth though. Pretty sure IDE is 100% flow controlled. CF cards even without a buffer are many, many times faster than the old 2.5" PATA hard drives of the time.

[Jope]

I can verify that 1fe00 is needed with many real hard disks too.

[Toni Wilen]

I may have (finally) found the problem..

When AOS IDE driver gets transfer that is longer than 256 bytes, it splits it in two or more pieces (because max ATA transfer is 256 blocks). After first piece has finished, it reads IDE CHS/LBA registers, adds one to address and then writes it back as start address of second piece. (ATA spec says they contain last read/written address) So far so good but:

ATA-1 spec says:

At command completion, the Command Block Registers contain the cylinder, head, and sector number of the last sector read.

ATA-2 spec says:

This register shall be updated to reflect the media address of the error when a media access command is unsuccessfully completed.

NOTE 9 - Prior to the development of this standard, this register was updated at the end of every media access command to reflect the current media address

In other words ATA-2 or newer device does not need to update CHS/LBA address after command finishes normally, only in error situations. This will break AOS ATA driver logic, start address of second piece becomes invalid (I assume drive keeps old values unmodified, second piece of data overwrites beginning of first piece which also explains why filesystem structure corruption has never been reported, only data corruption)

NOTE: This theory is not tested! (EDIT: it is now)

[alexh]

I tried it today with every 2.5" PATA drive I have and if I set the MAX TRANSFER above 0x1FE00 I get data corruption on a READ on all of them.

The file has to be larger than 255 sectors obviously!

[Hewitson]

Originally Posted by Retro-Nerd

Really? I never had any issues with standard MaxTransfer settings and PC hard drives. Pure luck?

Me either..

[Retro-Nerd]

I've used PC drives in the 90s on my A1200 and now since 2007 with the standard MaxTransfer settings. No problems at all. To be honest: never heard/read of this 1fe00 setting before the IDE-CF HD solution became popular, and there it's obviously needed. X-file?

[alexh]

All my hard drives are very modern PC laptop hard drives. Dunno if that makes any difference?

[Gordon]

I bumped this thread becuase when i installed classic wb on my 10.1gb 2.5 IDE hard drive(over 10 yeas old), some programs didnt open and some games came back with illegal expression errors. I reinstaled everything again after setting the drive to 0x1fe00 and it works fine.

[Retro-Nerd]

I've never used 2.5 HDs, only 3.5. Could this really make a difference?

[alexh]

Not if it is malformed ATA commands as I suggested.

If I could be arsed I could easily prove it with our ATA analyser at work.

I wonder if IDE emulation on WinUAE is closely tied closely enough to the hardware to show this problem?

I know if you use the high-level UAE hard disk driver then it does not show up. (You can prep disks in WinUAE using the high level driver with MAX TRANSFER above 0x1FE00 which work fine in WinUAE but fail on Amiga)

[rkauer]

@Alexh: I don't doubt your research, I just see the problem occurring mostly on CF & micro Drives (which doesn't have a cache memory between the inner "disk" controller and the outside IDE bus).

SCSI drives is another can of worms and they have their own behaviour.

The problem is related to the Amiga IDE and the way it talks to the devices hooked in. But I never had a validation problem before the CF era.

Now I had so much of those errors that I simply not use CF for anything but file transfers.

I don't no much how Amiga OS works. But usually hard disk driver is responsible to take care about max sector counts possible with certain hard drives. So, if it exceeds it driver splits into several requests with allowed counts.

Considering buffering by CF cards: some may have it, some not. All it is pretty irrelevant on slow machines like Amiga. Modern CF cards are much faster. Buffer is needed mostly by slow mechanical hard disks. But by CF cards sector access times are much-much better, so buffer is not a must.

I see other problems with CF cards and old micros without some advanced ATA compatible IDE IF : signal timing problems. It is worst in case of writing. Problem is that usual IDE logics activate and deactivate IDE SELP (CS0) signal together with RD and WR signals. By ATA spec it is not so (must activate RD and WR later and deactivate earlier) . Was OK with old hard drives, but after 20 years things are different.
With Sandisk CF cards usually no timing problems, because they have better behaviour and different timing than usual CF cards (as Kingston). Of course, best is to design new IDE (ATA) adapters with better signal timing, according to new ATA specs.

[alexh]

Originally Posted by AmigTart

I don't no much how Amiga OS works. But usually hard disk driver is responsible to take care about max sector counts possible with certain hard drives. So, if it exceeds it driver splits into several requests with allowed counts.

I thought that was what MAX TRANSFER was for. It tells some part of the OS/driver the limit at which a file system request of N bytes is split into multiple transfers.

[Toni Wilen]

Originally Posted by alexh

I thought that was what MAX TRANSFER was for. It tells some part of the OS/driver the limit at which a file system request of N bytes is split into multiple transfers.

Max transfer is filesystem parameter, filesystem (if it cares about the parameter) splits the request in less than max transfer pieces before sending it to device driver.

Device driver does not know nothing about max transfer parameter.

(and check also post 25)

[thomas]

Originally Posted by Toni Wilen

Device driver does not know nothing about max transfer parameter.

Yes, but the device driver should know the limitations of the hardware it drives. In this case, the ATA driver should know about the 256 sector limit of ATA commands and if a larger amount of sectors is to be transferred, it should split it automatically into multiple ATA commands or use another ATA command which can transfer more than 256 sectors.

BTW, WinUAE emulates the MaxTransfer problem on the internal IDE port, too. So if this wasn't intentional, then it's really scsi.device which sends malformed ATA commands to the hardware and not some missing cache on the harddrive or something.

[thomas]

Originally Posted by Toni Wilen

In other words ATA-2 or newer device does not need to update CHS/LBA address after command finishes normally, only in error situations. This will break AOS ATA driver logic, start address of second piece becomes invalid (I assume drive keeps old values unmodified, second piece of data overwrites beginning of first piece which also explains why filesystem structure corruption has never been reported, only data corruption)

NOTE: This theory is not tested!

If you examine the data written by WinUAE to a HDF connected to IDE0, you'll see exactly this: if you transfer 257 sectors (0 - 256), sector 1 to 255 are written correctly, sector 0 contains the data of written sector 256 and sector 256 remains unchanged.

[alexh]

Originally Posted by Toni Wilen

(and check also post 25)

I missed that post. That certainly explains why only some devices are affected. Seems dumb to read the tailgate status to form the next command but hey.

I know that our modern ATA8 hard drive controllers always update the tailgate status for all commands, not just errors.

[Toni Wilen]

Originally Posted by thomas

Yes, but the device driver should know the limitations of the hardware it drives. In this case, the ATA driver should know about the 256 sector limit of ATA commands and if a larger amount of sectors is to be transferred, it should split it automatically into multiple ATA commands or use another ATA command which can transfer more than 256 sectors.

It does split it properly but unfortunately splitting uses ATA feature that was removed in later ATA specs.

Originally Posted by thomas

If you examine the data written by WinUAE to a HDF connected to IDE0, you'll see exactly this: if you transfer 257 sectors (0 - 256), sector 1 to 255 are written correctly, sector 0 contains the data of written sector 256 and sector 256 remains unchanged.

Thanks, this should confirm my theory.

Should WinUAE detect this situation and warn about it? It is easy to emulate this exactly as ATA-1 spec says which is technically right thing to do but may not the best choice from non-technical point of view