Testing PFS3AIO

[Toni Wilen]

Explanation is still the same: it won't become official until someone makes good enough test program (that automatically fills the partition and does all kinds of file accesses and also checksums each file). Only if it survives multi-hour testing of this kind: it can be considered safe

Max partition limit is already about 1.6T. There are 4 "limits": original ~5G (1k "smalldisk"), original ~104G (1k), new 411G (2k) and 1.6T (4k). Internal index block size doubles each time max size increases.

[xboxown]

Originally Posted by Toni Wilen

Explanation is still the same: it won't become official until someone makes good enough test program (that automatically fills the partition and does all kinds of file accesses and also checksums each file). Only if it survives multi-hour testing of this kind: it can be considered safe

Max partition limit is already about 1.6T. There are 4 "limits": original ~5G (1k "smalldisk"), original ~104G (1k), new 411G (2k) and 1.6T (4k). Internal index block size doubles each time max size increases.

I would do it as soon as I resolve my power supply issue. Apparently my power supply have killed two hard drives and a dvd reader before I realized it killed them. Yesterday when my dvd reader started to smell burning I decided..okay...a new power supply solution is imminent. Right now...I am...<goes on into endless details of what he plans to do that no one on Earth really cares about>......

....although....I could write such apps in amiBlitz if you want me to? I can right an app that rights a large sum of files to fill the partition...what the program will do is fill the hard drive up..then ones the file hard drive is filled up..it will open each file and modify them by updating their contents, changing their sizes...and doing that for all the files...then it would delete 50% of the files and rewrite them again with different sizes. That I can do without asking someone for help in how to code such function.

The only problem that makes me go...uh help?....is the checksums each file. What do you mean by checksumming and how does one code a checksum function?

---------------------------------------------

What do you think of this method of testing the partition?

A) Checks the available space of the partition and use that to divide it by 4096 (4 MB) to determine how many of 4 MB files it will produce to fill it up.
B) Then it will produce that many 4 MB files of gibberish contents.
C) Ones the hard drive space is full it would open each 4 MB and modify it's contents to become 2 MB until 50% of that are 2 MB.
D) Then the remaining 25% will make the files 1 MB.
E) Then the remaining 25% will make the files to be 512KB
F) Then it will calculate how much space left available and then take the available space and divide it by 512 KB to determine the actual size of the file it need to grow big again to fill up the space again.
G) Then it calculates how many of those files it need to fill up the space and then expand those files to that size until the partition is full again.

The file names will be 0.DUM, 1.DUM, 2.DUM, etc easy when used in programming for looping.

H) (After you explain checksumming and how it works and code it) I will check checksum each file until it is completed.
I) Then it deletes the files one by one until all the test files are terminated.

How does that work? Is that good? I will use AmiBlitz to develop the application...it will use MUI and it will be verbose in terms of it's actions until completed.

[ShK]

Could you quickly calculate how long this will take with 1.6T and multi-hour testing including checksums each of file, if your IDE read/write speed is say ~3MB/s?

[cyberhead97]

I would be tempted to write such a filesystem test program to assist with stress testing large 100-200GB+ partitions. I would write it in C and have it do the following steps:

1. Write file with random filename and configurable size, calculate md5 checksum while writing.
2. Read file, calculate md5 and compare.
3. Log result.
4. Repeat until stopped with eg. Ctrl+C or configurable number of repeats.

Would such a test program be sufficient for testing pfs3aio on large partitions?

[xboxown]

Originally Posted by ShK

Could you quickly calculate how long this will take with 1.6T and multi-hour testing including checksums each of file, if your IDE read/write speed is say ~3MB/s?

Shk: You have a 1.6 TB??

[xboxown]

Originally Posted by cyberhead97

I would be tempted to write such a filesystem test program to assist with stress testing large 100-200GB+ partitions. I would write it in C and have it do the following steps:

1. Write file with random filename and configurable size, calculate md5 checksum while writing.
2. Read file, calculate md5 and compare.
3. Log result.
4. Repeat until stopped with eg. Ctrl+C or configurable number of repeats.

Would such a test program be sufficient for testing pfs3aio on large partitions?

heheheh cyberhead97 do it! But can you make it where it stops by itself and deletes the files and not wait for human interaction? Such approach you will be doing will end up destroying the hard drive's magnetic by constantly saving/deleting/making non-stop. Please make your program merciful to the user.

[cyberhead97]

Originally Posted by xboxown

heheheh cyberhead97 do it! But can you make it where it stops by itself and deletes the files and not wait for human interaction? Such approach you will be doing will end up destroying the hard drive's magnetic by constantly saving/deleting/making non-stop. Please make your program merciful to the user.

Yes, it could regularly check free diskspace, where it’s writing files and delete all written test files, when no space is left. Then it could run continuously without any user interaction. Maybe even have it switch file size after e.g. 1000 writes to make it even more automatic.

[ShK]

Originally Posted by xboxown

Shk: You have a 1.6 TB??

Not with Amiga. I made a simple checksum tests back then with DirDiff program, but I quess it's not enough on this case.

[xboxown]

Originally Posted by cyberhead97

Yes, it could regularly check free diskspace, where it’s writing files and delete all written test files, when no space it left. Then it could run continuously without any user interaction. Maybe even have it switch file size after e.g. 1000 writes to make it even more automatic.

And then it stops by itself?......................................

[xboxown]

Originally Posted by ShK

Not with Amiga. I made a simple checksum tests back then with DirDiff program, but I quess it's not enough on this case.

Ah!

[cyberhead97]

Originally Posted by xboxown

And then it stops by itself?......................................

It could have a configurable number of writes, then it would stop after having completed all those writes. During writing it will delete files, if needed when no diskspace is left. Eg. write 1.000.000 times and stop.

It could also be possible to use configurable time span for writing files. Eg. stop after 1 hour or 30 minutes.

Last option would be run until stopped manually.

[xboxown]

Originally Posted by cyberhead97

It could have a configurable number of writes, then it would stop after having completed all those writes. During writing it will delete files, if needed when no diskspace is left. Eg. write 1.000.000 times and stop.

It could also be possible to use configurable time span for writing files. Eg. stop after 1 hour or 30 minutes.

Last option would be run until stopped manually.

Wait...isn't 1 million times will be kill the hard drive??!

[Toni Wilen]

- Fill the partition (optional parameter: how much to fill) with random size files (optional parameter: min and max size) with random contents (so that if some data block overwrites other file's block, checksum must be guaranteed changed). Checksum must be calculated from original generated data. Must be much better algorithm than CRC-32! Must stop immediately if error detected.
- After partition has been filled, each file must be read and checksum calculated. All checksums must match 100%.
- Then randomly chosen files should be deleted and new random size files generated (with same checksum generation as in first step)
- Repeat (stop with Control-C)

File size/name random generator should use user given seed value parameter so that exact same test can be replicated.

It does not matter if disk isn't 100% filled, it isn't important. Important test case is to guarantee nothing gets corrupted (or filesystem crash) when creating/reading/deleting files.

EDIT: Also some way to only rewrite existing file partially would be nice to have but it can make checksumming more complex.

Program must work even if more than one instance is running simultaneously (to simulate multiple programs doing disk accesses at the same time). Must work even if both are using same path = file names should not have collisions.

Also very important: must runnable from CLI, (no GUI!), all messages must be written to standard output.

Obviously I run it in emulation for best (disk) performance.

(Don't discuss pointless stuff like "writing too much". It does not happen, even with SSD. Unless you do continuous writing at max speed for a year or more.)

[cyberhead97]

Originally Posted by Toni Wilen

- Fill the partition (optional parameter: how much to fill) with random size files (optional parameter: min and max size) with random contents (so that if some data block overwrites other file's block, checksum must be guaranteed changed). Checksum must be calculated from original generated data. Must be much better algorithm than CRC-32! Must stop immediately if error detected.
- After partition has been filled, each file must be read and checksum calculated. All checksums must match 100%.
- Then randomly chosen files should be deleted and new random size files generated (with same checksum generation as in first step)
- Repeat (stop with Control-C)

File size/name random generator should use user given seed value parameter so that exact same test can be replicated.

It does not matter if disk isn't 100% filled, it isn't important. Important test case is to guarantee nothing gets corrupted (or filesystem crash) when creating/reading/deleting files.

EDIT: Also some way to only rewrite existing file partially would be nice to have but it can make checksumming more complex.

Program must work even if more than one instance is running simultaneously (to simulate multiple programs doing disk accesses at the same time). Must work even if both are using same path = file names should not have collisions.

Also very important: must runnable from CLI, (no GUI!), all messages must be written to standard output.

Obviously I run it in emulation for best (disk) performance.

(Don't discuss pointless stuff like "writing too much". It does not happen, even with SSD. Unless you do continuous writing at max speed for a year or more.)

Thanks for input Toni. I like the pseudo main loop you described as it's more refined than my immediate thoughts.

For tools like this I agree with NO GUI. CLI is best choice for script-able and reproducible testing with standard output messages so it can be shown or redirected to a file, e.g >PC:log.txt.

I would also run it using emulation for best performance and being able to stress it further than I would be able to with real hardware. It would still be interesting to test with real hardware.

I'm very tempted to get started writing such a filesystem test tool and put it on a github for any one interested.

This might also be the time where I start investigating and use modern Amiga cross compilers for Windows instead of using eg. SAS/C in AmigaOS.

[xboxown]

Originally Posted by cyberhead97

Thanks for input Toni. I like the pseudo main loop you described as it's more refined than my immediate thoughts.

For tools like this I agree with NO GUI. CLI is best choice for script-able and reproducible testing with standard output messages so it can be shown or redirected to a file, e.g >PC:log.txt.

I would also run it using emulation for best performance and being able to stress it further than I would be able to with real hardware. It would still be interesting to test with real hardware.

I'm very tempted to get started writing such a filesystem test tool and put it on a github for any one interested.

This might also be the time where I start investigating and use modern Amiga cross compilers for Windows instead of using eg. SAS/C in AmigaOS.

All I have to say is thank you for taking up the challenge! PFS3 is literally my favorite file system for the Amiga and it have opened the door for Amiga 68k to see beyond the partition limit like any modern system. I will only use PFS3 as long as I am breathing and using Amiga.

[Locutus]

Maybe its worth it to take a step back and not reinvent the wheel here as this is a fairly typical monkey testing approach.

Most filesystems have test suites, so there's tests for this out there already.

for example:

https://git.kernel.org/pub/scm/fs/xf...e/src/fstest.c

and the classical FSX:

http://codemonkey.org.uk/projects/fsx/
https://github.com/linux-test-projec...ux/fsx-linux.c

[AC/DC HACKER!]

Greetings all,

I bought a new SanDisk 32GB MicroSD A1 and formatted it with 3 small partitions at the moment. I'm not running any of the partitions as SYS. On my SYS which is a CompactFlash 32GB I used SysSpeed to check SanDisk speed in my SCSI2SD V5.zero.

"While doing so, PS3-III reports: "Write 2 Error 173 on block 8184 + 2
F0.00.04.00.00.1F.F8.0A.00..00.00.00.08.00.00.00.00.00

RETRY CANCEL"

My first time using PS3AIO with a MicroSD.

A4000, CSPPC, Picasso4, BigRamPlus... I have WHDLoad installed games on another partition on the same MicroSD..and so far, not any other prompts. I noticed this last night. I ran PFS3Doctor and Beta Doctor on the partition and not a problem reported. I have to eject the MicroSD, reset the machine, and put it back in for it to be accessible again...after I exit the other prompts after the one I typed...then the A4000 will reset from CTRL+A+A. I'm not sure this is a bug, so I'm placing it here in testing.

[Toni Wilen]

Those SCSI direct mode errors are almost always hardware or driver related problems. PFS3 is simply reporting driver returned error information.

04 = HARDWARE ERROR
08/00 LOGICAL UNIT COMMUNICATION FAILURE (Perhaps this means SCSI2SD had trouble writing to the SD card?)

[AC/DC HACKER!]

Hmm...yeah, very possible. I'll check making a dump and actual SD..use it with WinUAE & see if I can some of those kind of results. I'm thinking, though, the older SCSI2SD's might not function so well with newer MacroSD's? Nice to know PFS3 displays errors or "something" rather than just being silent. That helps a lot.

[apex]

Any progress here? :-)