English Amiga Board


Go Back   English Amiga Board > Support > support.Hardware

 
 
Thread Tools
Old 24 December 2019, 12:40   #101
Futaura
Registered User

Futaura's Avatar
 
Join Date: Aug 2018
Location: United Kingdom
Posts: 62
Have checked DOpus now and can't seem to find any settings relating to copy buffering...

Devmon tells a story though. DOpus Magellan appears to have dynamic buffering in that on a single large file it starts off with an 8K write, followed by 16K, 32K, and so on, doubling up to a maximum of 512K writes. However, looking at the source code this may vary as it is timing how long writes take and adjusting the write buffer accordingly.

DOpus 4 is very different and simpler - looking at the source code, it picks a static buffer size, based on the size of the file to be copied, maxing out at 128K writes. Things may be complicated a little in that it uses AsyncIO to read the file (some versions might be using it for writing - disabled in the source code I'm looking at).

Nothing wrong with either of those approaches, of course, but might help with tracking the issue in SCSI2SD. The shell Copy command is unsurprisingly much more basic - the version I use has a static 64K buffer size, but this can be changed in more recent versions with the BUF command line parameter.
Futaura is offline  
Old 24 December 2019, 19:57   #102
AC/DC HACKER!
Registered User

AC/DC HACKER!'s Avatar
 
Join Date: Aug 2016
Location: SCC-NAC-dC-iNT
Posts: 474
Quote:
Originally Posted by Futaura View Post
FYI, my standard boot is DOpus Magellan in WBR mode. However, I've not tried copying with it yet - only Copy command from shell or BackItAllUp. The Copy command that I'm using defaults to a 64K buffer - I think I mentioned earlier that raising this to 128K or 256K caused major SCSI bus fatal errors (errors messages from blizzppc.device required a power off/on). This was consistently reproducible with 6.2.9, but no such errors with 6.2.5 or 6.2.10. As it happens SCSI2SD uses a 64K write buffer internally.
I don't have it in WBR mode. From Shell, I've been using standard copy (not any changing of buffer). DOpus 4.12 doesn't have a way to change the buffer. I considered that. I rechecked to verify. DOpus 5.81 and.91 do, sort of..but I haven't changed them either. I also went on ahead and renamed my default config for them, loaded them as default and..same thing. DOpus 5.xx and 4 all function fine with normal Amiga stuff..and though I understand they were intended for original hardware, and < = 3.1, I don't see any reason for DOpus x.xx to fail. Unless they're doing something very unique. I didn't get into debuggers of that sort.

Quote:
I'll check later, but does DOpus have a copy buffer setting? If so, I'd bet this is making a difference. I'll check with devmon later also, which will show the write sizes being sent.

Parity checking should usually be enabled on all devices. The only time I've ever seen parity errors was years ago when my Epson scanner was hooked up. I know I keep banging on about termination, but it is even more important on the CSPPC, seeing as it is 68pin and the controller itself doesn't act as a terminator (unlike the Blizzards). Ideally, you need to place an active terminator at both ends of the cable and the SCSI2SD in the middle with termination disabled. I have been told that using SCSI2SD's built-in termination is probably a bad idea on a 68pin bus: "Just looking at this scsi2sd HW it seems to auto terminate simple 50pin scsi but with a 68pin bus that probably causes a real mess." - from RS himself (I have a contact on the MorphOS team).
Okay, okay, okay....stop being a nag about it (I'm just messing with you!). I'll add the terminator back to the end of the line. It's the only eternal one I have so..it'll have to do. I'll disable it in SCSI2SD V6 and give it another go... I'll also enable parity again...and just leave it on. Good to have some feedback going on, Futaura.
AC/DC HACKER! is offline  
Old 25 December 2019, 03:49   #103
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
Summary of my findings with the new firmware v6.2.10:

-I can partition volumes and edit them afterwards, no corruption occurs in the RDB, and I can format disks with no problems at all, they are there after a reboot.

-Data integrity still compromised: two partitions copied with DOpus4 and checked with CompareDirs ( http://aminet.net/package/util/cli/CompareDirs ). It lists, if specified, only files that are different even if their sizes are identical. In my WB partition, 124 files corrupted. In my Work partition, 1990 files corrupted. Curiously, corrupted files are all above 64KB, there are no small files in the list.

-SysInfo crashes when trying to test disk speed, while it didn't in previous tests.

Info and samples of corrupted files, together with their respective originals, already sent to Michael.

Saluditos,

Ferrán.
Ferry is offline  
Old 25 December 2019, 07:20   #104
AC/DC HACKER!
Registered User

AC/DC HACKER!'s Avatar
 
Join Date: Aug 2016
Location: SCC-NAC-dC-iNT
Posts: 474
Quote:
Originally Posted by Ferry View Post
Summary of my findings with the new firmware v6.2.10:

-I can partition volumes and edit them afterwards, no corruption occurs in the RDB, and I can format disks with no problems at all, they are there after a reboot.

-Data integrity still compromised: two partitions copied with DOpus4 and checked with CompareDirs ( http://aminet.net/package/util/cli/CompareDirs ). It lists, if specified, only files that are different even if their sizes are identical. In my WB partition, 124 files corrupted. In my Work partition, 1990 files corrupted. Curiously, corrupted files are all above 64KB, there are no small files in the list.

-SysInfo crashes when trying to test disk speed, while it didn't in previous tests.

Info and samples of corrupted files, together with their respective originals, already sent to Michael.

Saluditos,

Ferrán.
You message seems a lot like mine with CSPPC's SCSI. I'm using the same firmware now. Using Workbench to Copy or shell to Copy produces valid results. Using DOpus of any version, does not.

Michael McMasters is checking into it. I sent him some quality/poor examples. Soon, we'll know.

I haven't read all your previous posts so far completely. I'll do so soon.

Are you using SysInfo 4.xx? It has been updated and available through Aminet. https://aminet.net/package/util/moni/SysInfo

Functions well for me...and current Not Public Firmware.
AC/DC HACKER! is offline  
Old 25 December 2019, 21:01   #105
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
Quote:
Originally Posted by AC/DC HACKER! View Post
Are you using SysInfo 4.xx? It has been updated and available through Aminet. https://aminet.net/package/util/moni/SysInfo

Functions well for me...and current Not Public Firmware.
Yes, firmware is the one Michael sent me by mail, v6.2.10. I tried both version of SysInfo, the one I had installed in my system and I dl'ed the one in Aminet. Both crash when testing disk speed. Curious thing is that does not crash always: on empty partitions seems to work, but as soon as I try a one with data on it, it crashes. I don't know if having data on it its related to the crash or just a coincidence.

Anyway, Michael just sent me a new fw, installing and testing right now.

Saluditos,

Ferrán.
Ferry is offline  
Old 26 December 2019, 14:00   #106
Futaura
Registered User

Futaura's Avatar
 
Join Date: Aug 2018
Location: United Kingdom
Posts: 62
I too get corruption with large files when using DOpus or Copy with a 128K buffer, using 6.2.10. Michael has sent me another version to test, but the update util unfortunately crashed just before it was going to upload the firmware! Now the PC won't see the card and I don't seem to be having much luck with the BOOTLDR jumper (and lack of).
Futaura is offline  
Old 26 December 2019, 14:33   #107
Futaura
Registered User

Futaura's Avatar
 
Join Date: Aug 2018
Location: United Kingdom
Posts: 62
Never mind, a tiny bit of aluminium foil pressed onto the BOOTLDR jumper solder pads did the trick .

Unfortunately, I'm still getting corruption with 6.2.11 when copying a single large file with a 128K buffer in Copy (as per DOpus 4).
Futaura is offline  
Old 26 December 2019, 19:31   #108
AMike
Registered User
AMike's Avatar
 
Join Date: Jan 2007
Location: near Vienna/Austria
Posts: 192
Hey guys - sorry, I've didn't read the whole thread. I'm using the V6 since 3 years and quite happy with the solution. For german speakers you can find here my findings.
https://www.a1k.org/forum/index.php?threads/58280/
I haven't tested the latest updates of the firmware but I got the best results with 6.2.1 - so there was no need for me to go further. (sync 8.5MB/s) In my experience are the following points crucial:
AMike is offline  
Old 27 December 2019, 13:06   #109
Futaura
Registered User

Futaura's Avatar
 
Join Date: Aug 2018
Location: United Kingdom
Posts: 62
I've been testing 6.2.12 this morning and this is better, in terms of less corruption, but data corruption is still there - there is a definite pattern to the corrupted byte offsets in the files. With 128K writes I am also getting corruption in async mode, same in sync 5Mb mode, but worse in sync 10Mb mode.

@AMike My card shipped with 6.2.8, so I've never tried 6.2.1 - if that works for you then great. However, it is quite clear that newer versions (6.2.5, and newer, at least) do have issues, possibly caused by performance improvements, and the best way forward is for Michael to fix it. If nobody ever reports issues, all future updates will probably inherit the same data corruption issues. So, going back to 6.2.1 might be a solution, but fixing the latest firmware is the future proof solution. For there to be 3 of us here reporting the same issue, on different hardware setups, it is highly likely that the firmware is the problem.
Futaura is offline  
Old 27 December 2019, 13:14   #110
AMike
Registered User
AMike's Avatar
 
Join Date: Jan 2007
Location: near Vienna/Austria
Posts: 192
That was just an offer of help - what you do with it is up to you. I am happy with my V6 and don't really need more. If needed here are the settings and a speed test.
Attached Thumbnails
Click image for larger version

Name:	IMG_20191224_192320__01.jpg
Views:	29
Size:	1,001.0 KB
ID:	65688   Click image for larger version

Name:	IMG_20191226_185620.jpg
Views:	29
Size:	852.8 KB
ID:	65689   Click image for larger version

Name:	IMG_20191226_185629.jpg
Views:	25
Size:	876.2 KB
ID:	65690  
AMike is offline  
Old 28 December 2019, 04:10   #111
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
Quote:
Originally Posted by Futaura View Post
I've been testing 6.2.12 this morning and this is better, in terms of less corruption, but data corruption is still there - there is a definite pattern to the corrupted byte offsets in the files. With 128K writes I am also getting corruption in async mode, same in sync 5Mb mode, but worse in sync 10Mb mode.
I also tested 6.1.12, and I'm still getting corruption too, but with a big difference: while before I got hundreds or thousands of differences between the original and its copy, now I get only from 2 to 8 bytes:



and the size of files affected is bigger now: before it was from 64KB upwards, now seems to be from 128KB up.

BTW, are you really getting data corruption with "Sync, 5MB/s" mode? With 6.2.5 it worked well, and my "reliable" booting SD card was made with that FW and speed, but if you say that you are getting corruption with 5MB/s with newer FWs…… If I don't know if I can trust newer FWs, I'll better keep it apart and safe until this issue is solved.

Saluditos,

Ferrán.
Ferry is offline  
Old 28 December 2019, 04:46   #112
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
Quote:
Originally Posted by AMike View Post
That was just an offer of help - what you do with it is up to you. I am happy with my V6 and don't really need more. If needed here are the settings and a speed test.
Thanks for the info.

The problem here is data corruption: filles have the same size when copied but the copy does not match the original. I mean: data corruption is not apparent, you wont see a requester saying "Data could not be copied" or similar, it's simply that copied files (some of them, at least), even having the same size than their originals, got their data corrupted, and you only notice it when you try to open them: executables, libraries, devices won't load or will crash, and guides, text, pics and other data files will load with errors or won't load at all.

Oh, and this don't happen to all the files in a partition. F.ex., in my WB partition only 124 files are affected, from executables (AmiPDF, Installer or PGP) to Kickstart files (all 6 I have get corruption when copied).

What I mean is: if you are a bit lucky, you will be able to copy and load your WB at that speed without even noticing there's a problem until you load a corrupted file/dev/lib/etc., and even then you'll will probably blame another thing, since almost nobody ever think it can be a corrupted file.

What I used to compare originals against copies is CompareDirs:

http://aminet.net/package/util/cli/CompareDirs

very easy to use:

comparedirs dir1: dir2: ram:results.txt noident

The "noident" keyword instructs the program to list only files with differences and ignore those that match.

Would it be possible for you to try with your config is and post your results? Thanks.

Saluditos,

Ferrán.
Ferry is offline  
Old 28 December 2019, 06:05   #113
AC/DC HACKER!
Registered User

AC/DC HACKER!'s Avatar
 
Join Date: Aug 2016
Location: SCC-NAC-dC-iNT
Posts: 474
Quote:
Originally Posted by Futaura View Post
I too get corruption with large files when using DOpus or Copy with a 128K buffer, using 6.2.10. Michael has sent me another version to test, but the update util unfortunately crashed just before it was going to upload the firmware! Now the PC won't see the card and I don't seem to be having much luck with the BOOTLDR jumper (and lack of).
Quote:
Originally Posted by Futaura View Post
Never mind, a tiny bit of aluminium foil pressed onto the BOOTLDR jumper solder pads did the trick .

Unfortunately, I'm still getting corruption with 6.2.11 when copying a single large file with a 128K buffer in Copy (as per DOpus 4).
Mine didn't include a "shunt" either, the ability wasn't acceptable unless you're creative enough. I used and have a Paperclip ready. Which I had to use soon after mine arrived...because update failed.

I'm behind you by 1 digit. Ha! But glad to see you've noticed it as well. Wooohoo!
AC/DC HACKER! is offline  
Old 28 December 2019, 12:18   #114
AMike
Registered User
AMike's Avatar
 
Join Date: Jan 2007
Location: near Vienna/Austria
Posts: 192
Quote:
Originally Posted by Ferry View Post

Would it be possible for you to try with your config is and post your results? Thanks.

Saluditos,

Ferrán.
Hello Ferrán, will do the test. If I understand you correctly it make only sense with partitions which I copied directly from internal IDE to the V6? I copied the boot partiton directly from IDE to V6. (700MB) The "Work" Partition is a backup of my Amiga4000 partition which I copied on the A4000 and the deneb directly on the SD Card. (90GB) In the first step I compare the bootpartitions.

Quote:
Originally Posted by Ferry View Post
Thanks for the info.

What I mean is: if you are a bit lucky, you will be able to copy and load your WB at that speed without even noticing there's a problem until you load a corrupted file/dev/lib/etc., and even then you'll will probably blame another thing, since almost nobody ever think it can be a corrupted file.
.
From time to time (very seldom) I've got read/write errors, but I'm not sure if that is a problem of the V6 or the 60Mhz speed of my Blizzard1260. The SCSI Kit don't like speeds above 50Mhz.
AMike is offline  
Old 28 December 2019, 13:47   #115
Futaura
Registered User

Futaura's Avatar
 
Join Date: Aug 2018
Location: United Kingdom
Posts: 62
Quote:
Originally Posted by Ferry View Post
I also tested 6.1.12, and I'm still getting corruption too, but with a big difference: while before I got hundreds or thousands of differences between the original and its copy, now I get only from 2 to 8 bytes:



and the size of files affected is bigger now: before it was from 64KB upwards, now seems to be from 128KB up.

BTW, are you really getting data corruption with "Sync, 5MB/s" mode? With 6.2.5 it worked well, and my "reliable" booting SD card was made with that FW and speed, but if you say that you are getting corruption with 5MB/s with newer FWs…… If I don't know if I can trust newer FWs, I'll better keep it apart and safe until this issue is solved.
Yes, you should go back to 6.2.5 (or 6.2.1) for general usage. However, corruption appears to normally only happen with larger than 64K writes, and most software, especially during booting, is never going to even write that much, or it will be using smaller buffers. DOpus 4 copying uses 128K writes, which will show the problem, and DOpus 5 also.

I get the same corruption in async and sync 5mb modes with 6.2.12. And it is worse in sync 10mb mode. However, it is not as bad as before as there is less corruption. I think what was happening originally was that it would drop/miss a few bytes out completely, which then meant the following bytes were out of sync with the original until it got into sync again. With 6.2.12, incorrect/corrupt bytes take the place of the missed bytes, so the following bytes are in the correct place and match the original. At least corruption is consistent with 6.2.12 and not as random as it was in 6.2.9 and earlier. That hopefully means a solution can be found.

At the moment, I'm just testing copying with a 512K file using Copy command with buffer set to 128K (BUF=256), as I only have DOpus Magellan installed. I created the file with some C code so that it has a set pattern - that way hopefully all the affected corrupt bytes get caught, when comparing, and it is also easy to see a definite pattern. Like you, byte 66048 (hex 10200) is always the offset in a file to get corrupted. I'm sure a fix will come.
Futaura is offline  
Old 28 December 2019, 17:50   #116
AMike
Registered User
AMike's Avatar
 
Join Date: Jan 2007
Location: near Vienna/Austria
Posts: 192
Quote:
Originally Posted by Ferry View Post
T
What I used to compare originals against copies is CompareDirs:

http://aminet.net/package/util/cli/CompareDirs

very easy to use:

comparedirs dir1: dir2: ram:results.txt noident
On my IDE Partition are 3 backups - so I compared SD1: (V6) with a directory on my Backup IDE partition - but I think that's ok for the test!?
The sysinfo differences are probably due to the fact that I haven't made a backup for a while and changed Sysinfo manually on SD1, but I can't explain the differences on the info files.

I will do another test and copy some data (300MB are enough?) from IDE to V6 and compare the result again. I think this delivers a more accurate result.
Attached Files
File Type: txt results.txt (1.1 KB, 8 views)
AMike is offline  
Old 28 December 2019, 18:38   #117
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
Quote:
Originally Posted by AMike View Post
On my IDE Partition are 3 backups - so I compared SD1: (V6) with a directory on my Backup IDE partition - but I think that's ok for the test!?
The sysinfo differences are probably due to the fact that I haven't made a backup for a while and changed Sysinfo manually on SD1, but I can't explain the differences on the info files.

I will do another test and copy some data (300MB are enough?) from IDE to V6 and compare the result again. I think this delivers a more accurate result.
In your results file there are only small files, and none appeared in mine, only files above 64KB, so I'd say it's OK. The difference in those files in your results must be that you snapshoted icons or saved preferences, so nothing to worry about.

Anyway, the correct way to do the test would be: first check that your v6 partition is set to Sync and 8 or 10MB/s. Then copy a big chunk of data (those 300MB should be ok) to that v6 partition (in a drawer should be OK) from another source (IDE will do), so origin and destination should be identical, and then compare them with CompareDirs.

Edit: BTW, my setup is also a B1260+SCSI Kit, but at 50MHz, not 60, but this should not make any difference regarding data corruption, at least not only with the v6.

Saluditos,

Ferrán.

Last edited by Ferry; 28 December 2019 at 18:41. Reason: Adding data
Ferry is offline  
Old 28 December 2019, 19:20   #118
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
Quote:
Originally Posted by Futaura View Post
However, corruption appears to normally only happen with larger than 64K writes, and most software, especially during booting, is never going to even write that much, or it will be using smaller buffers. DOpus 4 copying uses 128K writes, which will show the problem, and DOpus 5 also.

I get the same corruption in async and sync 5mb modes with 6.2.12. And it is worse in sync 10mb mode.
Hmmm… Do you think there will be any difference setting the Sync+speed mode from the Amiga side compared to setting it in the v6? I know you have to set the Sync flag also in the Amiga RDB anyway, but you don't set the speed. According to @AMike, he is setting flag&speed in the Amiga side, letting the v6 side as "No limit (safe)" and he's getting no corruption as it seems (waiting for his new test results, anyway).

Saluditos,

Ferrán.
Ferry is offline  
Old 28 December 2019, 19:38   #119
Futaura
Registered User

Futaura's Avatar
 
Join Date: Aug 2018
Location: United Kingdom
Posts: 62
Quote:
Originally Posted by Ferry View Post
Hmmm… Do you think there will be any difference setting the Sync+speed mode from the Amiga side compared to setting it in the v6? I know you have to set the Sync flag also in the Amiga RDB anyway, but you don't set the speed. According to @AMike, he is setting flag&speed in the Amiga side, letting the v6 side as "No limit (safe)" and he's getting no corruption as it seems (waiting for his new test results, anyway)
I wouldn't have thought so - the setting on the SCSI2SD merely sets the upper limit available to the host. Unitcontrol queries it, for example - that's why if you set the sync in unitcontrol higher than the limit on SCSI2SD, the UI automatically changes to that limit.

When you set the Sync flag in the RDB, the .device driver will automatically choose the maximum available speed (as set in SCSI2SD or the limit supported by the driver on the Amiga side).

Basically, as I understand it, it is the Amiga side which tells SCSI2SD whether async or sync is in use and what speed. SCSI2SD will tell the Amiga side what speed settings are available.
Futaura is offline  
Old 28 December 2019, 22:47   #120
Ferry
Registered User

Ferry's Avatar
 
Join Date: Jul 2018
Location: Spain
Posts: 54
A bit offtopic, but since we all here are using SCSI2SD v6, maybe someone is interested: I made a box for my SCSI2SD v6 card + IDC50-to-DB25 adapter in a 3D printer and uploaded the model to Thingiverse, so anyone can print it:

https://www.thingiverse.com/thing:4072201





Saluditos,

Ferrán.
Ferry is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
CF / SD and large drives FAQ fgh support.Hardware 407 13 February 2020 22:28
Problem with SUPER LARGE hard drives Karpow support.Hardware 32 08 June 2015 00:13
Large Hard-Drives (over 4gb) keitha1200 support.Hardware 4 20 April 2012 09:09
What sort of Filemaster to use with large drives? Ebster support.Apps 4 08 February 2009 18:53
Large hard drives and WB3.0... darkwave support.Hardware 3 05 July 2004 04:19

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 14:27.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.
Page generated in 0.09508 seconds with 16 queries