CF / SD and large drives FAQ

[thomas]

You should check the card first. Like I said there are fake cards which identify themselves as a larger capacity than they actually are. Format it under windows and fill it with files. There is also Windows software which does this check automatically. I just can't name one, never needed it.

[Reido]

I'll try that out so, thanks for the tip.

[Reido]

So I followed your advice and it is indeed an 8gb card, thanks, I know its not fake now at least. I'll try the installation process again and hope for a different outcome!!

[Malakie]

I have two CF cards working in my A2000 under both 3.2 and 3.9, master slave BUT I have seen two issues.. the first, for whatever reason, it will not allow a partition over 8 GB. Second, when you attempt to format the second CF with verify, it produces an out of memory error?! Plenty of ram, 1 Meg Chip, 8 Meg Fast, and it does not do it with the first CF card.

However, the partition still formats and shows on Workbench even with the out of ram error from verify... Question is though, is it a valid partition for the full 8GB?

Anyone know about this issue?

[ItsTheSmell]

You can't dual boot cf cards as they done have buffers like hdd's.

[Jope]

Originally Posted by ItsTheSmell

You can't dual boot cf cards as they done have buffers like hdd's.

Seems you are mixing up jargon, as this statement can't be really validated with a true/untrue response, the best I can do is a confused single eyebrow raise.

Dual boot? Do you mean master/slave?

What kinds of buffers do you mean? How are they relevant?

It is true that some CF cards do not support master/slave, but obviously Malakie managed to find a pair that does since they are all visible to the OS.

[Jope]

Originally Posted by Malakie

I have two CF cards working in my A2000 under both 3.2 and 3.9, master slave BUT I have seen two issues.. the first, for whatever reason, it will not allow a partition over 8 GB. Second, when you attempt to format the second CF with verify, it produces an out of memory error?! Plenty of ram, 1 Meg Chip, 8 Meg Fast, and it does not do it with the first CF card.

Barely enough RAM for the normal 512 byte block size at least.

Please always use quick format with hard disks/memory cards to avoid hitting any bugs with older versions of the OS and also to give your memory card's controller a chance to do wear leveling.

However, the partition still formats and shows on Workbench even with the out of ram error from verify... Question is though, is it a valid partition for the full 8GB?

You can use Thomas' check4gb to see. http://thomas-rapp.homepage.t-online.de/download.html

[ItsTheSmell]

Originally Posted by Jope

Seems you are mixing up jargon, as this statement can't be really validated with a true/untrue response, the best I can do is a confused single eyebrow raise.

Dual boot? Do you mean master/slave?

What kinds of buffers do you mean? How are they relevant?

It is true that some CF cards do not support master/slave, but obviously Malakie managed to find a pair that does since they are all visible to the OS.

I write it because according to Doug Compton its true and he knows a thing or two about amigas.

[Malakie]

Originally Posted by ItsTheSmell

You can't dual boot cf cards as they done have buffers like hdd's.

I did not say dual boot.. I said one card I setup as 3.2 the other 3.9, and as I just mentioned in my larger post, it is possible to dual boot but takes some work to do so. Too much work than is worth the effort.

[Malakie]

Originally Posted by ItsTheSmell

I write it because according to Doug Compton its true and he knows a thing or two about amigas.

I actually was with Commodore. It can be done, just takes a bit of work using assigns, env and some other things.

Right now, I have dual CF cards working fine, master/slave.

I found the problem, part me, part the OS. Will explain in another post.

[Malakie]

I originally stopped in here as I was having one hell of a problem getting two CF cards to work on the same IDE chain. Errors or not even seeing the second card was a huge issue until I figured out what was going on.

Keep in mind, I am back into Amiga after a 30 year absence serving in the US Military etc. And this was after I not only worked for Commodore and Commodore Amiga as a technical engineer, but after I also started and ran an Amiga software and hardware development company. Some of those products being WMS (Workbench Management System), Floptical 120 drivers, SCSI tape and DAT interface and drivers, Sapphire 68020 card, and other software and hardware as well. Our VP was the man that did the Mechforce game on Amiga and we help design and build the cockpits for FASA's Battletech and supported the hardware in each, which were base A500 boards. We are the ones that came up with most of those drivers used for all that hardware to make it Amiga functional (Floptical 120, SCSI and DAT Tape drives).

While I am slowly getting back into things and the memory is waking up, once in a while I find myself forgetting something and then realizing I was wrong or screwed up when the past memory suddenly says DUH.

It has taken a bit for me to get back to remembering a lot of which had been left to decay in the brain. So once in a while I stop here to ask being I can't remember something until someone says something that clicks in a memory.. And with my previous question and reading some of the replies, that is exactly what happened.. my brain went DUH wait a second, this is how it actually works...

CF cards is the new one for me but simply using the IDE capability of the A2091 or a 3rd part IDE card (yes the A2091 has an IDE port) IS all you need to do, even on older machines. They are identified and used nearly identical to a standard HD.

How to make them work.

NOTE this is on an A2000 with a Terrible Fire 534 030 Card using its IDE port. Other cards also work fine, including the A2091 and its IDE port.

After playing a lot of catch up, my first mistake was listening to a lot of different things said.. that turned out to be false. BUT some of that is how my memory came back to what was and was not correct. My first mistake though, was assuming. I was under the impression that 3.2 and 3.9 OS's with the latest kickstarts, would allow for drives larger than 4GB across the board. Nope, not true.

It will now allow you to have HD's larger than 4gb but NOT fully verifiable partitions larger than 4gb. In fact, drives larger than 8gb will also still be problematic as I have found. You can get lucky with larger partitions and never see a read write issue but in most cases, you are just biding time until it happens and when it does, it will NOT be recoverable.

Even if you cut them into 4gb partitions, I am finding a lot of errors still occur on drives larger than 8gb size even using the 4gb limit partition. Note this is with 7.0 or greater ROMS on even the A2091 cards as well as the super rare, 8.0 roms, which I do have a set of.

This same holds true for CF cards. I have also found this to be true on a few different 3rd party IDE controllers that I have now tried, which leads me to believe this is still either a kickstart or OS limitation as of now but I have not dove into the kernel level stuff at all and currently have no plan to do so. I am still learning more about 3.2 and 3.9. Official 3.1 was the last OS I dealt with at Commodore until I left, so for me, 3.2 and 3.9 are a whole new beast under the hood.

If you want to use dual CF cards, what you need is something like this for the best results:

NOTE this is a DUAL SIDED CF card interface.. master card on one side, slave on the opposite side.

Next, buy two TYPE I 4gb or 8gb CF cards. You can try larger size but as I said, I continue to run into problems with anything larger than 8gb and I have purposely decided to stick with 4gb cards myself. I use 1gb for system partition, 3gb for tools and work, and the second card at 4gb entirely for games or whatever else I want.

HD Toolbox WILL read, partition and format both cards just fine. You don't need anything else. Just plug the card into your IDE port, and run HD Toolbox. This works great on 3.2 or 3.9 OS's version but note I have NOT tried this on anything lower to date.

I have one system running base 2.04 OS but I have SCSI in that for mainly for a boot drive and for my Jaz, Zip and Orb drive backup archives.

When you format the CF's, make sure you turn on verify. You should increase the buffers to about 300. You can change the Boot option, partition size etc. You can also play with block size but it does alter the speed through put unless you stay with the default of 512 blocks. You should ALWAYS let HD Toolbox READ the specs from the card you use and use those numbers. Do not alter anything that will most likely change those primary specifications. Finally, do not change the max transfer as it is NOT needed unlike for slower mechanical HD's. It seems to cause more read write issues on CF cards probably due to the speed they move data at.

I have had zero issues at all using 4gb and 8gb CF cards. I might give a 16gb card a try but have not yet done so. I HAVE tested multiple 32gb CF card brands and every single one of them causes either errors in verifying, read write errors at some point or just plain can't be seen by the system. Even using all 4gb partitions on those it does see, are a problem for whatever reason.

NOTE I HAVE NOT done my A1200 yet so some of this may change as I move to it for the next setup I do. This is for the A2000 series only at this time.

[Malakie]

Originally Posted by Jope

Barely enough RAM for the normal 512 byte block size at least.

Please always use quick format with hard disks/memory cards to avoid hitting any bugs with older versions of the OS and also to give your memory card's controller a chance to do wear leveling.



You can use Thomas' check4gb to see. http://thomas-rapp.homepage.t-online.de/download.html

Actually they are not bugs you are running into, rather the limits of the OS and data verification ability for larger drives.

[Jope]

Originally Posted by Malakie

Actually they are not bugs you are running into, rather the limits of the OS and data verification ability for larger drives.

Unfortunately with some versions of the format command you can run into these limits, even though your hard disk driver and filesystem handler are recent enough to not have a 2GB or 4GB limit.

As a result, it's a good idea to just always quick format hard drives so that you won't run into these limitations and have a bad time. The limits involve not only verification, but the actual formatting too.

If I were less lazy, I would compile a list of buggy / working versions of format commands for you right here, but unfortunately I don't need the info myself since format quick is always safe and I don't have time to wait for 8GB or more to verify.

3.1.4 might be the first release (in terms of release date) with a fully 4GB safe format command, but perhaps someone else can chime in with the real truth, I hate to post speculation.

I respect that you were intimately familiar with the amiga and have a lot of old habits from back in the day, so it must be annoying to read stuff that does not explain the full rationale behind it. And of course it is a good idea to challenge the status quo if it seems like people are just cargo cult repeating something without knowing why.

[Malakie]

Originally Posted by Jope

Unfortunately with some versions of the format command you can run into these limits, even though your hard disk driver and filesystem handler are recent enough to not have a 2GB or 4GB limit.

As a result, it's a good idea to just always quick format hard drives so that you won't run into these limitations and have a bad time. The limits involve not only verification, but the actual formatting too.

If I were less lazy, I would compile a list of buggy / working versions of format commands for you right here, but unfortunately I don't need the info myself since format quick is always safe and I don't have time to wait for 8GB or more to verify.

3.1.4 might be the first release (in terms of release date) with a fully 4GB safe format command, but perhaps someone else can chime in with the real truth, I hate to post speculation.

I respect that you were intimately familiar with the amiga and have a lot of old habits from back in the day, so it must be annoying to read stuff that does not explain the full rationale behind it. And of course it is a good idea to challenge the status quo if it seems like people are just cargo cult repeating something without knowing why.

Interesting since quick format is actually causing the problems. Full format usually, but not always, works. but quick format fails for the second CF card or hard drive from out of memory errors, every time.

[Jope]

Seems you are running out of memory in this case, and it's not the format command that runs out of memory, but the filesystem handler. With a block size of 512, FFS wants about 1MB of memory per 1GB of partition when formatting/validating and you were right on the limit.

Since you have 3.2, you should use its FFS and a higher block size than 512.

Or use pfs3aio and ditch FFS and its constant validation. :-)

[Daedalus]

What's important to note though, is that there are some things here that are beyond the scope of the newer OS. For large hard drives to work properly, you need both a filesystem and a device driver to support the large sizes. The various OS updates include the necessary filesystem update, but only update the drivers for built-in SCSI/IDE ports. Using an interface on any other hardware, like an A2091, Buddha, Blizzard SCSI etc., uses its own driver, so the OS updates can't fix that for you. If the drivers for that interface don't support large drives, you'll still have problems regardless of OS version.

My understanding, too, is that 3.1.4 is the first version with a fully 4GB-safe format command, but I can't remember any time in the past 25 years that I've used full format for a hard drive. It's pointless on anything likely to fall foul of the 4GB limit anyway.

Running out of memory can be alleviated by increasing the block size - a TF534 gives you only 4MB of RAM, and that's not enough to validate larger partitions with the default block size.

Edit: Beaten to it by Jope...

[Malakie]

Originally Posted by Jope

Seems you are running out of memory in this case. With a block size of 512, FFS wants about 1MB of memory per 1GB of partition when formatting/validating and you were right on the limit.

Since you have 3.2, you should use its FFS and a higher block size than 512.

Or use pfs3aio and ditch FFS and its constant validation. :-)

Yea been considering moving to some other file system for this. But first wanted to see what limits still remain in the standard FFS system libraries.

The CF cards want 512 and seem to work much better and faster sticking with the 512 actually. I tried 1024 and 4096 and they read and wrote much slower so I went back to 512.

[Malakie]

Originally Posted by Daedalus

What's important to note though, is that there are some things here that are beyond the scope of the newer OS. For large hard drives to work properly, you need both a filesystem and a device driver to support the large sizes. The various OS updates include the necessary filesystem update, but only updates the drivers for built-in SCSI/IDE ports. Using an interface on any other hardware, like an A2091, Buddha, Blizzard SCSI etc., uses its own driver, so the OS updates can't fix that for you. If the drivers for that interface don't support large drives, you'll still have problems regardless of OS version.

My understanding, too, is that 3.1.4 is the first version with a fully 4GB-safe format command, but I can't remember any time in the past 25 years that I've used full format for a hard drive. It's pointless on anything likely to fall foul of the 4GB limit anyway.

Running out of memory can be alleviated by increasing the block size - a TF534 gives you only 4MB of RAM, and that's not enough to validate larger partitions with the default block size.

Edit: Beaten to it by Jope...

Both of you mentioned increasing block size so am going to try this again at a higher setting and see if the result improves this time...

[Daedalus]

Increasing the block size improves performance for some operations - at least it made a significant difference for me when using mechanical hard drives when I went from 512 to 8192. Bear in mind that if you increase the block size, it will increase the size of each buffer (and hence RAM usage) accordingly - 300 buffers is 150kB at 512 blocksize but close to 2.5MB at 8192 blocksize. You'll likely need far less than 300 though, and the buffers won't affect reliability.

[Malakie]

Originally Posted by Daedalus

Increasing the block size improves performance for some operations - at least it made a significant difference for me when using mechanical hard drives when I went from 512 to 8192. Bear in mind that if you increase the block size, it will increase the size of each buffer (and hence RAM usage) accordingly - 300 buffers is 150kB at 512 blocksize but close to 2.5MB at 8192 blocksize. You'll likely need far less than 300 though, and the buffers won't affect reliability.

In one of my posts, I did say specifically that using block size made a much bigger different on mechanical HD's vs CF cards etc. I just did another test and other than memory use, I don't see any speed advantage using anything other than 512 which is the CF's read default number.