English Amiga Board


Go Back   English Amiga Board > Support > support.Apps

 
 
Thread Tools
Old 20 March 2019, 17:06   #1
Toni Wilen
WinUAE developer
 
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 44
Posts: 23,014
Testing PFS3AIO

Explanation is still the same: it won't become official until someone makes good enough test program (that automatically fills the partition and does all kinds of file accesses and also checksums each file). Only if it survives multi-hour testing of this kind: it can be considered safe

Max partition limit is already about 1.6T. There are 4 "limits": original ~5G (1k "smalldisk"), original ~104G (1k), new 411G (2k) and 1.6T (4k). Internal index block size doubles each time max size increases.
Toni Wilen is offline  
Old 20 March 2019, 17:57   #2
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by Toni Wilen View Post
Explanation is still the same: it won't become official until someone makes good enough test program (that automatically fills the partition and does all kinds of file accesses and also checksums each file). Only if it survives multi-hour testing of this kind: it can be considered safe

Max partition limit is already about 1.6T. There are 4 "limits": original ~5G (1k "smalldisk"), original ~104G (1k), new 411G (2k) and 1.6T (4k). Internal index block size doubles each time max size increases.
I would do it as soon as I resolve my power supply issue. Apparently my power supply have killed two hard drives and a dvd reader before I realized it killed them. Yesterday when my dvd reader started to smell burning I decided..okay...a new power supply solution is imminent. Right now...I am...<goes on into endless details of what he plans to do that no one on Earth really cares about>......

....although....I could write such apps in amiBlitz if you want me to? I can right an app that rights a large sum of files to fill the partition...what the program will do is fill the hard drive up..then ones the file hard drive is filled up..it will open each file and modify them by updating their contents, changing their sizes...and doing that for all the files...then it would delete 50% of the files and rewrite them again with different sizes. That I can do without asking someone for help in how to code such function.

The only problem that makes me go...uh help?....is the checksums each file. What do you mean by checksumming and how does one code a checksum function?

---------------------------------------------

What do you think of this method of testing the partition?

A) Checks the available space of the partition and use that to divide it by 4096 (4 MB) to determine how many of 4 MB files it will produce to fill it up.
B) Then it will produce that many 4 MB files of gibberish contents.
C) Ones the hard drive space is full it would open each 4 MB and modify it's contents to become 2 MB until 50% of that are 2 MB.
D) Then the remaining 25% will make the files 1 MB.
E) Then the remaining 25% will make the files to be 512KB
F) Then it will calculate how much space left available and then take the available space and divide it by 512 KB to determine the actual size of the file it need to grow big again to fill up the space again.
G) Then it calculates how many of those files it need to fill up the space and then expand those files to that size until the partition is full again.

The file names will be 0.DUM, 1.DUM, 2.DUM, etc easy when used in programming for looping.

H) (After you explain checksumming and how it works and code it) I will check checksum each file until it is completed.
I) Then it deletes the files one by one until all the test files are terminated.

How does that work? Is that good? I will use AmiBlitz to develop the application...it will use MUI and it will be verbose in terms of it's actions until completed.

Last edited by xboxown; 20 March 2019 at 18:12.
xboxown is offline  
Old 20 March 2019, 18:17   #3
ShK
Registered User
ShK's Avatar
 
Join Date: Mar 2013
Location: Lahti / Finland
Age: 48
Posts: 350
Could you quickly calculate how long this will take with 1.6T and multi-hour testing including checksums each of file, if your IDE read/write speed is say ~3MB/s?
ShK is offline  
Old 20 March 2019, 18:19   #4
cyberhead97
Registered User

 
Join Date: Feb 2016
Location: Denmark
Posts: 210
I would be tempted to write such a filesystem test program to assist with stress testing large 100-200GB+ partitions. I would write it in C and have it do the following steps:
  1. Write file with random filename and configurable size, calculate md5 checksum while writing.
  2. Read file, calculate md5 and compare.
  3. Log result.
  4. Repeat until stopped with eg. Ctrl+C or configurable number of repeats.

Would such a test program be sufficient for testing pfs3aio on large partitions?
cyberhead97 is offline  
Old 20 March 2019, 18:20   #5
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by ShK View Post
Could you quickly calculate how long this will take with 1.6T and multi-hour testing including checksums each of file, if your IDE read/write speed is say ~3MB/s?
Shk: You have a 1.6 TB??
xboxown is offline  
Old 20 March 2019, 18:21   #6
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by cyberhead97 View Post
I would be tempted to write such a filesystem test program to assist with stress testing large 100-200GB+ partitions. I would write it in C and have it do the following steps:
  1. Write file with random filename and configurable size, calculate md5 checksum while writing.
  2. Read file, calculate md5 and compare.
  3. Log result.
  4. Repeat until stopped with eg. Ctrl+C or configurable number of repeats.

Would such a test program be sufficient for testing pfs3aio on large partitions?
heheheh cyberhead97 do it! But can you make it where it stops by itself and deletes the files and not wait for human interaction? Such approach you will be doing will end up destroying the hard drive's magnetic by constantly saving/deleting/making non-stop. Please make your program merciful to the user.
xboxown is offline  
Old 20 March 2019, 18:25   #7
cyberhead97
Registered User

 
Join Date: Feb 2016
Location: Denmark
Posts: 210
Quote:
Originally Posted by xboxown View Post
heheheh cyberhead97 do it! But can you make it where it stops by itself and deletes the files and not wait for human interaction? Such approach you will be doing will end up destroying the hard drive's magnetic by constantly saving/deleting/making non-stop. Please make your program merciful to the user.
Yes, it could regularly check free diskspace, where it’s writing files and delete all written test files, when no space is left. Then it could run continuously without any user interaction. Maybe even have it switch file size after e.g. 1000 writes to make it even more automatic.

Last edited by cyberhead97; 20 March 2019 at 18:29. Reason: Corrected them all to written files
cyberhead97 is offline  
Old 20 March 2019, 18:26   #8
ShK
Registered User
ShK's Avatar
 
Join Date: Mar 2013
Location: Lahti / Finland
Age: 48
Posts: 350
Quote:
Originally Posted by xboxown View Post
Shk: You have a 1.6 TB??
Not with Amiga. I made a simple checksum tests back then with DirDiff program, but I quess it's not enough on this case.

ShK is offline  
Old 20 March 2019, 18:29   #9
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by cyberhead97 View Post
Yes, it could regularly check free diskspace, where it’s writing files and delete all written test files, when no space it left. Then it could run continuously without any user interaction. Maybe even have it switch file size after e.g. 1000 writes to make it even more automatic.
And then it stops by itself?......................................
xboxown is offline  
Old 20 March 2019, 18:29   #10
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by ShK View Post
Not with Amiga. I made a simple checksum tests back then with DirDiff program, but I quess it's not enough on this case.

Ah!
xboxown is offline  
Old 20 March 2019, 18:35   #11
cyberhead97
Registered User

 
Join Date: Feb 2016
Location: Denmark
Posts: 210
Quote:
Originally Posted by xboxown View Post
And then it stops by itself?......................................
It could have a configurable number of writes, then it would stop after having completed all those writes. During writing it will delete files, if needed when no diskspace is left. Eg. write 1.000.000 times and stop.

It could also be possible to use configurable time span for writing files. Eg. stop after 1 hour or 30 minutes.

Last option would be run until stopped manually.
cyberhead97 is offline  
Old 20 March 2019, 18:38   #12
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by cyberhead97 View Post
It could have a configurable number of writes, then it would stop after having completed all those writes. During writing it will delete files, if needed when no diskspace is left. Eg. write 1.000.000 times and stop.

It could also be possible to use configurable time span for writing files. Eg. stop after 1 hour or 30 minutes.

Last option would be run until stopped manually.
Wait...isn't 1 million times will be kill the hard drive??!
xboxown is offline  
Old 20 March 2019, 19:31   #13
Toni Wilen
WinUAE developer
 
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 44
Posts: 23,014
- Fill the partition (optional parameter: how much to fill) with random size files (optional parameter: min and max size) with random contents (so that if some data block overwrites other file's block, checksum must be guaranteed changed). Checksum must be calculated from original generated data. Must be much better algorithm than CRC-32! Must stop immediately if error detected.
- After partition has been filled, each file must be read and checksum calculated. All checksums must match 100%.
- Then randomly chosen files should be deleted and new random size files generated (with same checksum generation as in first step)
- Repeat (stop with Control-C)

File size/name random generator should use user given seed value parameter so that exact same test can be replicated.

It does not matter if disk isn't 100% filled, it isn't important. Important test case is to guarantee nothing gets corrupted (or filesystem crash) when creating/reading/deleting files.

EDIT: Also some way to only rewrite existing file partially would be nice to have but it can make checksumming more complex.

Program must work even if more than one instance is running simultaneously (to simulate multiple programs doing disk accesses at the same time). Must work even if both are using same path = file names should not have collisions.

Also very important: must runnable from CLI, (no GUI!), all messages must be written to standard output.

Obviously I run it in emulation for best (disk) performance.

(Don't discuss pointless stuff like "writing too much". It does not happen, even with SSD. Unless you do continuous writing at max speed for a year or more.)
Toni Wilen is offline  
Old 20 March 2019, 20:03   #14
cyberhead97
Registered User

 
Join Date: Feb 2016
Location: Denmark
Posts: 210
Quote:
Originally Posted by Toni Wilen View Post
- Fill the partition (optional parameter: how much to fill) with random size files (optional parameter: min and max size) with random contents (so that if some data block overwrites other file's block, checksum must be guaranteed changed). Checksum must be calculated from original generated data. Must be much better algorithm than CRC-32! Must stop immediately if error detected.
- After partition has been filled, each file must be read and checksum calculated. All checksums must match 100%.
- Then randomly chosen files should be deleted and new random size files generated (with same checksum generation as in first step)
- Repeat (stop with Control-C)

File size/name random generator should use user given seed value parameter so that exact same test can be replicated.

It does not matter if disk isn't 100% filled, it isn't important. Important test case is to guarantee nothing gets corrupted (or filesystem crash) when creating/reading/deleting files.

EDIT: Also some way to only rewrite existing file partially would be nice to have but it can make checksumming more complex.

Program must work even if more than one instance is running simultaneously (to simulate multiple programs doing disk accesses at the same time). Must work even if both are using same path = file names should not have collisions.

Also very important: must runnable from CLI, (no GUI!), all messages must be written to standard output.

Obviously I run it in emulation for best (disk) performance.

(Don't discuss pointless stuff like "writing too much". It does not happen, even with SSD. Unless you do continuous writing at max speed for a year or more.)
Thanks for input Toni. I like the pseudo main loop you described as it's more refined than my immediate thoughts.

For tools like this I agree with NO GUI. CLI is best choice for script-able and reproducible testing with standard output messages so it can be shown or redirected to a file, e.g >PC:log.txt.

I would also run it using emulation for best performance and being able to stress it further than I would be able to with real hardware. It would still be interesting to test with real hardware.

I'm very tempted to get started writing such a filesystem test tool and put it on a github for any one interested.

This might also be the time where I start investigating and use modern Amiga cross compilers for Windows instead of using eg. SAS/C in AmigaOS.
cyberhead97 is offline  
Old 20 March 2019, 20:09   #15
xboxown
Registered User
 
Join Date: Dec 2012
Location: Vancouver/Canada
Posts: 486
Quote:
Originally Posted by cyberhead97 View Post
Thanks for input Toni. I like the pseudo main loop you described as it's more refined than my immediate thoughts.

For tools like this I agree with NO GUI. CLI is best choice for script-able and reproducible testing with standard output messages so it can be shown or redirected to a file, e.g >PC:log.txt.

I would also run it using emulation for best performance and being able to stress it further than I would be able to with real hardware. It would still be interesting to test with real hardware.

I'm very tempted to get started writing such a filesystem test tool and put it on a github for any one interested.

This might also be the time where I start investigating and use modern Amiga cross compilers for Windows instead of using eg. SAS/C in AmigaOS.
All I have to say is thank you for taking up the challenge! PFS3 is literally my favorite file system for the Amiga and it have opened the door for Amiga 68k to see beyond the partition limit like any modern system. I will only use PFS3 as long as I am breathing and using Amiga.
xboxown is offline  
Old 20 March 2019, 22:56   #16
Locutus
Registered User

 
Join Date: Jul 2014
Location: Finland
Posts: 954
Maybe its worth it to take a step back and not reinvent the wheel here as this is a fairly typical monkey testing approach.

Most filesystems have test suites, so there's tests for this out there already.

for example:

https://git.kernel.org/pub/scm/fs/xf...e/src/fstest.c

and the classical FSX:

http://codemonkey.org.uk/projects/fsx/
https://github.com/linux-test-projec...ux/fsx-linux.c
Locutus is offline  
Old 08 May 2019, 20:29   #17
AC/DC HACKER!
Registered User

 
Join Date: Aug 2016
Location: DigitalCorruption/iNT
Posts: 395
Greetings all,

I bought a new SanDisk 32GB MicroSD A1 and formatted it with 3 small partitions at the moment. I'm not running any of the partitions as SYS. On my SYS which is a CompactFlash 32GB I used SysSpeed to check SanDisk speed in my SCSI2SD V5.zero.

"While doing so, PS3-III reports: "Write 2 Error 173 on block 8184 + 2
F0.00.04.00.00.1F.F8.0A.00..00.00.00.08.00.00.00.00.00

RETRY CANCEL"

My first time using PS3AIO with a MicroSD.

A4000, CSPPC, Picasso4, BigRamPlus... I have WHDLoad installed games on another partition on the same MicroSD..and so far, not any other prompts. I noticed this last night. I ran PFS3Doctor and Beta Doctor on the partition and not a problem reported. I have to eject the MicroSD, reset the machine, and put it back in for it to be accessible again...after I exit the other prompts after the one I typed...then the A4000 will reset from CTRL+A+A. I'm not sure this is a bug, so I'm placing it here in testing.
AC/DC HACKER! is offline  
Old 09 May 2019, 18:05   #18
Toni Wilen
WinUAE developer
 
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 44
Posts: 23,014
Those SCSI direct mode errors are almost always hardware or driver related problems. PFS3 is simply reporting driver returned error information.

04 = HARDWARE ERROR
08/00 LOGICAL UNIT COMMUNICATION FAILURE (Perhaps this means SCSI2SD had trouble writing to the SD card?)
Toni Wilen is offline  
Old 09 May 2019, 19:43   #19
AC/DC HACKER!
Registered User

 
Join Date: Aug 2016
Location: DigitalCorruption/iNT
Posts: 395
Hmm...yeah, very possible. I'll check making a dump and actual SD..use it with WinUAE & see if I can some of those kind of results. I'm thinking, though, the older SCSI2SD's might not function so well with newer MacroSD's? Nice to know PFS3 displays errors or "something" rather than just being silent. That helps a lot.
AC/DC HACKER! is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
PFS3AIO v3.1 Final Toni Wilen News 389 16 September 2019 09:06
PFS3aio vs SFS/02 White support.WinUAE 10 13 April 2018 21:44
PFS3AIO Probs ? Foul support.FS-UAE 0 21 February 2018 22:49
PFS3AIO Questions manic23 support.Hardware 26 22 April 2014 13:49
Pfs3aio ???? stingray76 support.Hardware 2 19 February 2014 09:15

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 11:32.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Page generated in 0.08685 seconds with 12 queries