SCSI2SD v6 and large drives

[Futaura]

Have checked DOpus now and can't seem to find any settings relating to copy buffering...

Devmon tells a story though. DOpus Magellan appears to have dynamic buffering in that on a single large file it starts off with an 8K write, followed by 16K, 32K, and so on, doubling up to a maximum of 512K writes. However, looking at the source code this may vary as it is timing how long writes take and adjusting the write buffer accordingly.

DOpus 4 is very different and simpler - looking at the source code, it picks a static buffer size, based on the size of the file to be copied, maxing out at 128K writes. Things may be complicated a little in that it uses AsyncIO to read the file (some versions might be using it for writing - disabled in the source code I'm looking at).

Nothing wrong with either of those approaches, of course, but might help with tracking the issue in SCSI2SD. The shell Copy command is unsurprisingly much more basic - the version I use has a static 64K buffer size, but this can be changed in more recent versions with the BUF command line parameter.

[AC/DC HACKER!]

Originally Posted by Futaura

FYI, my standard boot is DOpus Magellan in WBR mode. However, I've not tried copying with it yet - only Copy command from shell or BackItAllUp. The Copy command that I'm using defaults to a 64K buffer - I think I mentioned earlier that raising this to 128K or 256K caused major SCSI bus fatal errors (errors messages from blizzppc.device required a power off/on). This was consistently reproducible with 6.2.9, but no such errors with 6.2.5 or 6.2.10. As it happens SCSI2SD uses a 64K write buffer internally.

I don't have it in WBR mode. From Shell, I've been using standard copy (not any changing of buffer). DOpus 4.12 doesn't have a way to change the buffer. I considered that. I rechecked to verify. DOpus 5.81 and.91 do, sort of..but I haven't changed them either. I also went on ahead and renamed my default config for them, loaded them as default and..same thing. DOpus 5.xx and 4 all function fine with normal Amiga stuff..and though I understand they were intended for original hardware, and < = 3.1, I don't see any reason for DOpus x.xx to fail. Unless they're doing something very unique. I didn't get into debuggers of that sort.

I'll check later, but does DOpus have a copy buffer setting? If so, I'd bet this is making a difference. I'll check with devmon later also, which will show the write sizes being sent.

Parity checking should usually be enabled on all devices. The only time I've ever seen parity errors was years ago when my Epson scanner was hooked up. I know I keep banging on about termination, but it is even more important on the CSPPC, seeing as it is 68pin and the controller itself doesn't act as a terminator (unlike the Blizzards). Ideally, you need to place an active terminator at both ends of the cable and the SCSI2SD in the middle with termination disabled. I have been told that using SCSI2SD's built-in termination is probably a bad idea on a 68pin bus: "Just looking at this scsi2sd HW it seems to auto terminate simple 50pin scsi but with a 68pin bus that probably causes a real mess." - from RS himself (I have a contact on the MorphOS team).

Okay, okay, okay....stop being a nag about it (I'm just messing with you!). I'll add the terminator back to the end of the line. It's the only eternal one I have so..it'll have to do. I'll disable it in SCSI2SD V6 and give it another go... I'll also enable parity again...and just leave it on. Good to have some feedback going on, Futaura.

[Ferry]

Summary of my findings with the new firmware v6.2.10:

-I can partition volumes and edit them afterwards, no corruption occurs in the RDB, and I can format disks with no problems at all, they are there after a reboot.

-Data integrity still compromised: two partitions copied with DOpus4 and checked with CompareDirs ( http://aminet.net/package/util/cli/CompareDirs ). It lists, if specified, only files that are different even if their sizes are identical. In my WB partition, 124 files corrupted. In my Work partition, 1990 files corrupted. Curiously, corrupted files are all above 64KB, there are no small files in the list.

-SysInfo crashes when trying to test disk speed, while it didn't in previous tests.

Info and samples of corrupted files, together with their respective originals, already sent to Michael.

Saluditos,

Ferrán.

[AC/DC HACKER!]

Originally Posted by Ferry

Summary of my findings with the new firmware v6.2.10:

-I can partition volumes and edit them afterwards, no corruption occurs in the RDB, and I can format disks with no problems at all, they are there after a reboot.

-Data integrity still compromised: two partitions copied with DOpus4 and checked with CompareDirs ( http://aminet.net/package/util/cli/CompareDirs ). It lists, if specified, only files that are different even if their sizes are identical. In my WB partition, 124 files corrupted. In my Work partition, 1990 files corrupted. Curiously, corrupted files are all above 64KB, there are no small files in the list.

-SysInfo crashes when trying to test disk speed, while it didn't in previous tests.

Info and samples of corrupted files, together with their respective originals, already sent to Michael.

Saluditos,

Ferrán.

You message seems a lot like mine with CSPPC's SCSI. I'm using the same firmware now. Using Workbench to Copy or shell to Copy produces valid results. Using DOpus of any version, does not.

Michael McMasters is checking into it. I sent him some quality/poor examples. Soon, we'll know.

I haven't read all your previous posts so far completely. I'll do so soon.

Are you using SysInfo 4.xx? It has been updated and available through Aminet. https://aminet.net/package/util/moni/SysInfo

Functions well for me...and current Not Public Firmware.

[Ferry]

Originally Posted by AC/DC HACKER!

Are you using SysInfo 4.xx? It has been updated and available through Aminet. https://aminet.net/package/util/moni/SysInfo

Functions well for me...and current Not Public Firmware.

Yes, firmware is the one Michael sent me by mail, v6.2.10. I tried both version of SysInfo, the one I had installed in my system and I dl'ed the one in Aminet. Both crash when testing disk speed. Curious thing is that does not crash always: on empty partitions seems to work, but as soon as I try a one with data on it, it crashes. I don't know if having data on it its related to the crash or just a coincidence.

Anyway, Michael just sent me a new fw, installing and testing right now.

Saluditos,

Ferrán.

[Futaura]

I too get corruption with large files when using DOpus or Copy with a 128K buffer, using 6.2.10. Michael has sent me another version to test, but the update util unfortunately crashed just before it was going to upload the firmware! Now the PC won't see the card and I don't seem to be having much luck with the BOOTLDR jumper (and lack of).

[Futaura]

Never mind, a tiny bit of aluminium foil pressed onto the BOOTLDR jumper solder pads did the trick .

Unfortunately, I'm still getting corruption with 6.2.11 when copying a single large file with a 128K buffer in Copy (as per DOpus 4).

[AMike]

Hey guys - sorry, I've didn't read the whole thread. I'm using the V6 since 3 years and quite happy with the solution. For german speakers you can find here my findings.
https://www.a1k.org/forum/index.php?threads/58280/
I haven't tested the latest updates of the firmware but I got the best results with 6.2.1 - so there was no need for me to go further. (sync 8.5MB/s) In my experience are the following points crucial:

• The right SD Card - not every Card delivers reliable results - I use this card: https://www.amazon.de/SanDisk-Ultra-...puters&sr=1-5#
• It's important that the setup, initialise and partitioning is done in the V6. Normally I setup my harddisks/CF Cards on the A4000 with the Deneb, but that was not helpful with the V6.
• To increase the speed I use myunitcrontrol from the Aminet and set the V6 to 10MB/s: http://aminet.net/package/util/misc/MyUnitControl

[Futaura]

I've been testing 6.2.12 this morning and this is better, in terms of less corruption, but data corruption is still there - there is a definite pattern to the corrupted byte offsets in the files. With 128K writes I am also getting corruption in async mode, same in sync 5Mb mode, but worse in sync 10Mb mode.

@AMike My card shipped with 6.2.8, so I've never tried 6.2.1 - if that works for you then great. However, it is quite clear that newer versions (6.2.5, and newer, at least) do have issues, possibly caused by performance improvements, and the best way forward is for Michael to fix it. If nobody ever reports issues, all future updates will probably inherit the same data corruption issues. So, going back to 6.2.1 might be a solution, but fixing the latest firmware is the future proof solution. For there to be 3 of us here reporting the same issue, on different hardware setups, it is highly likely that the firmware is the problem.

[AMike]

That was just an offer of help - what you do with it is up to you. I am happy with my V6 and don't really need more. If needed here are the settings and a speed test.

[Ferry]

Originally Posted by Futaura

I've been testing 6.2.12 this morning and this is better, in terms of less corruption, but data corruption is still there - there is a definite pattern to the corrupted byte offsets in the files. With 128K writes I am also getting corruption in async mode, same in sync 5Mb mode, but worse in sync 10Mb mode.

I also tested 6.1.12, and I'm still getting corruption too, but with a big difference: while before I got hundreds or thousands of differences between the original and its copy, now I get only from 2 to 8 bytes:



and the size of files affected is bigger now: before it was from 64KB upwards, now seems to be from 128KB up.

BTW, are you really getting data corruption with "Sync, 5MB/s" mode? With 6.2.5 it worked well, and my "reliable" booting SD card was made with that FW and speed, but if you say that you are getting corruption with 5MB/s with newer FWs…… If I don't know if I can trust newer FWs, I'll better keep it apart and safe until this issue is solved.

Saluditos,

Ferrán.

[Ferry]

Originally Posted by AMike

That was just an offer of help - what you do with it is up to you. I am happy with my V6 and don't really need more. If needed here are the settings and a speed test.

Thanks for the info.

The problem here is data corruption: filles have the same size when copied but the copy does not match the original. I mean: data corruption is not apparent, you wont see a requester saying "Data could not be copied" or similar, it's simply that copied files (some of them, at least), even having the same size than their originals, got their data corrupted, and you only notice it when you try to open them: executables, libraries, devices won't load or will crash, and guides, text, pics and other data files will load with errors or won't load at all.

Oh, and this don't happen to all the files in a partition. F.ex., in my WB partition only 124 files are affected, from executables (AmiPDF, Installer or PGP) to Kickstart files (all 6 I have get corruption when copied).

What I mean is: if you are a bit lucky, you will be able to copy and load your WB at that speed without even noticing there's a problem until you load a corrupted file/dev/lib/etc., and even then you'll will probably blame another thing, since almost nobody ever think it can be a corrupted file.

What I used to compare originals against copies is CompareDirs:

http://aminet.net/package/util/cli/CompareDirs

very easy to use:

comparedirs dir1: dir2: ram:results.txt noident

The "noident" keyword instructs the program to list only files with differences and ignore those that match.

Would it be possible for you to try with your config is and post your results? Thanks.

Saluditos,

Ferrán.

[AC/DC HACKER!]

Originally Posted by Futaura

I too get corruption with large files when using DOpus or Copy with a 128K buffer, using 6.2.10. Michael has sent me another version to test, but the update util unfortunately crashed just before it was going to upload the firmware! Now the PC won't see the card and I don't seem to be having much luck with the BOOTLDR jumper (and lack of).

Originally Posted by Futaura

Never mind, a tiny bit of aluminium foil pressed onto the BOOTLDR jumper solder pads did the trick .

Unfortunately, I'm still getting corruption with 6.2.11 when copying a single large file with a 128K buffer in Copy (as per DOpus 4).

Mine didn't include a "shunt" either, the ability wasn't acceptable unless you're creative enough. I used and have a Paperclip ready. Which I had to use soon after mine arrived...because update failed.

I'm behind you by 1 digit. Ha! But glad to see you've noticed it as well. Wooohoo!

[AMike]

Originally Posted by Ferry

Would it be possible for you to try with your config is and post your results? Thanks.

Saluditos,

Ferrán.

Hello Ferrán, will do the test. If I understand you correctly it make only sense with partitions which I copied directly from internal IDE to the V6? I copied the boot partiton directly from IDE to V6. (700MB) The "Work" Partition is a backup of my Amiga4000 partition which I copied on the A4000 and the deneb directly on the SD Card. (90GB) In the first step I compare the bootpartitions.

Originally Posted by Ferry

Thanks for the info.

What I mean is: if you are a bit lucky, you will be able to copy and load your WB at that speed without even noticing there's a problem until you load a corrupted file/dev/lib/etc., and even then you'll will probably blame another thing, since almost nobody ever think it can be a corrupted file.
.

From time to time (very seldom) I've got read/write errors, but I'm not sure if that is a problem of the V6 or the 60Mhz speed of my Blizzard1260. The SCSI Kit don't like speeds above 50Mhz.

[Futaura]

Originally Posted by Ferry

I also tested 6.1.12, and I'm still getting corruption too, but with a big difference: while before I got hundreds or thousands of differences between the original and its copy, now I get only from 2 to 8 bytes:



and the size of files affected is bigger now: before it was from 64KB upwards, now seems to be from 128KB up.

BTW, are you really getting data corruption with "Sync, 5MB/s" mode? With 6.2.5 it worked well, and my "reliable" booting SD card was made with that FW and speed, but if you say that you are getting corruption with 5MB/s with newer FWs…… If I don't know if I can trust newer FWs, I'll better keep it apart and safe until this issue is solved.

Yes, you should go back to 6.2.5 (or 6.2.1) for general usage. However, corruption appears to normally only happen with larger than 64K writes, and most software, especially during booting, is never going to even write that much, or it will be using smaller buffers. DOpus 4 copying uses 128K writes, which will show the problem, and DOpus 5 also.

I get the same corruption in async and sync 5mb modes with 6.2.12. And it is worse in sync 10mb mode. However, it is not as bad as before as there is less corruption. I think what was happening originally was that it would drop/miss a few bytes out completely, which then meant the following bytes were out of sync with the original until it got into sync again. With 6.2.12, incorrect/corrupt bytes take the place of the missed bytes, so the following bytes are in the correct place and match the original. At least corruption is consistent with 6.2.12 and not as random as it was in 6.2.9 and earlier. That hopefully means a solution can be found.

At the moment, I'm just testing copying with a 512K file using Copy command with buffer set to 128K (BUF=256), as I only have DOpus Magellan installed. I created the file with some C code so that it has a set pattern - that way hopefully all the affected corrupt bytes get caught, when comparing, and it is also easy to see a definite pattern. Like you, byte 66048 (hex 10200) is always the offset in a file to get corrupted. I'm sure a fix will come.

[AMike]

Originally Posted by Ferry

T
What I used to compare originals against copies is CompareDirs:

http://aminet.net/package/util/cli/CompareDirs

very easy to use:

comparedirs dir1: dir2: ram:results.txt noident

On my IDE Partition are 3 backups - so I compared SD1: (V6) with a directory on my Backup IDE partition - but I think that's ok for the test!?
The sysinfo differences are probably due to the fact that I haven't made a backup for a while and changed Sysinfo manually on SD1, but I can't explain the differences on the info files.

I will do another test and copy some data (300MB are enough?) from IDE to V6 and compare the result again. I think this delivers a more accurate result.

[Ferry]

Originally Posted by AMike

On my IDE Partition are 3 backups - so I compared SD1: (V6) with a directory on my Backup IDE partition - but I think that's ok for the test!?
The sysinfo differences are probably due to the fact that I haven't made a backup for a while and changed Sysinfo manually on SD1, but I can't explain the differences on the info files.

I will do another test and copy some data (300MB are enough?) from IDE to V6 and compare the result again. I think this delivers a more accurate result.

In your results file there are only small files, and none appeared in mine, only files above 64KB, so I'd say it's OK. The difference in those files in your results must be that you snapshoted icons or saved preferences, so nothing to worry about.

Anyway, the correct way to do the test would be: first check that your v6 partition is set to Sync and 8 or 10MB/s. Then copy a big chunk of data (those 300MB should be ok) to that v6 partition (in a drawer should be OK) from another source (IDE will do), so origin and destination should be identical, and then compare them with CompareDirs.

Edit: BTW, my setup is also a B1260+SCSI Kit, but at 50MHz, not 60, but this should not make any difference regarding data corruption, at least not only with the v6.

Saluditos,

Ferrán.

[Ferry]

Originally Posted by Futaura

However, corruption appears to normally only happen with larger than 64K writes, and most software, especially during booting, is never going to even write that much, or it will be using smaller buffers. DOpus 4 copying uses 128K writes, which will show the problem, and DOpus 5 also.

I get the same corruption in async and sync 5mb modes with 6.2.12. And it is worse in sync 10mb mode.

Hmmm… Do you think there will be any difference setting the Sync+speed mode from the Amiga side compared to setting it in the v6? I know you have to set the Sync flag also in the Amiga RDB anyway, but you don't set the speed. According to @AMike, he is setting flag&speed in the Amiga side, letting the v6 side as "No limit (safe)" and he's getting no corruption as it seems (waiting for his new test results, anyway).

Saluditos,

Ferrán.

[Futaura]

Originally Posted by Ferry

Hmmm… Do you think there will be any difference setting the Sync+speed mode from the Amiga side compared to setting it in the v6? I know you have to set the Sync flag also in the Amiga RDB anyway, but you don't set the speed. According to @AMike, he is setting flag&speed in the Amiga side, letting the v6 side as "No limit (safe)" and he's getting no corruption as it seems (waiting for his new test results, anyway)

I wouldn't have thought so - the setting on the SCSI2SD merely sets the upper limit available to the host. Unitcontrol queries it, for example - that's why if you set the sync in unitcontrol higher than the limit on SCSI2SD, the UI automatically changes to that limit.

When you set the Sync flag in the RDB, the .device driver will automatically choose the maximum available speed (as set in SCSI2SD or the limit supported by the driver on the Amiga side).

Basically, as I understand it, it is the Amiga side which tells SCSI2SD whether async or sync is in use and what speed. SCSI2SD will tell the Amiga side what speed settings are available.

[Ferry]

A bit offtopic, but since we all here are using SCSI2SD v6, maybe someone is interested: I made a box for my SCSI2SD v6 card + IDC50-to-DB25 adapter in a 3D printer and uploaded the model to Thingiverse, so anyone can print it:

https://www.thingiverse.com/thing:4072201





Saluditos,

Ferrán.