Testing PFS3AIO

[Docent]

Originally Posted by Toni Wilen

- Fill the partition (optional parameter: how much to fill) with random size files (optional parameter: min and max size) with random contents (so that if some data block overwrites other file's block, checksum must be guaranteed changed). Checksum must be calculated from original generated data. Must be much better algorithm than CRC-32! Must stop immediately if error detected.
- After partition has been filled, each file must be read and checksum calculated. All checksums must match 100%.
- Then randomly chosen files should be deleted and new random size files generated (with same checksum generation as in first step)
- Repeat (stop with Control-C)

File size/name random generator should use user given seed value parameter so that exact same test can be replicated.

It does not matter if disk isn't 100% filled, it isn't important. Important test case is to guarantee nothing gets corrupted (or filesystem crash) when creating/reading/deleting files.

EDIT: Also some way to only rewrite existing file partially would be nice to have but it can make checksumming more complex.

Program must work even if more than one instance is running simultaneously (to simulate multiple programs doing disk accesses at the same time). Must work even if both are using same path = file names should not have collisions.

Also very important: must runnable from CLI, (no GUI!), all messages must be written to standard output.

Hi Toni,
To help progressing with release of pfsaio, I made an utility that does most of the requirements you listed above.
FileSystemStressTest has the following features:
- configurable destination path, where files will be generated
- configurable filename prefix so two or more instances can generate and validate files in the same directory
- configurable total volume of files to generate in MB
- configurable minimum and maximum size of a single file - the actual file size will be randomly generated to be between these two values
- configurable initial random generator seed for file contents
- configurable workbuffer size, used to generate/validate files
- option to keep generated files in the destination path
- configurable verbose level
- configurable number of iterations
- shell only

The random number generator used is xoshiro128++ 1.0, random seed for it is generated using splitmix32 seeded by the provided seed or time(NULL).
The hash used to verify file integrity is xxhash.
fsst uses stdio to perform read/writes.

Currently it does not support one point from your list (delete randomly selected files and generate new replacement files), I can add it but need to think a bit how to parametrize it from shell. Perhaps the number of files to delete/regenerate will do, but then the new file size should not be bigger than the removed one...

Interestingly, native windows version with options e:fstest -s 300 -l 0 -h 1000000 -r 0xBABA -b 0x7fffff -v 2 -p test_ runs single test on my laptop in 11 secs. The amiga version under winuae (configured with volumes as directiories) on the same laptop with the same options runs in 275 secs. 75 secs is spent on deleting files, while native version deletes in 1 sec.. There might be some room for performance improvement in volume as dir emulation's file removal.

[Toni Wilen]

Hi, sorry for delay. That sounds great, maybe I'll check pfs3 after DraCo emulation is done

[Docent]

Originally Posted by Toni Wilen

Hi, sorry for delay. That sounds great, maybe I'll check pfs3 after DraCo emulation is done

No problem, looking at the progress with the Draco emulation, you'll finish it over the weekend
In the meantime I implemented the last requirement - now you can specify the percent of newly created files to be deleted and recreated with new random content and size. I'll post the new version tomorrow - need to do some more testing first.

[Docent]

Here's attached the latest version 1.1 with the following features added:

- faster random number generator (sfc) used
- configurable percent of generated files to be removed, regenerated and validated in each iteration
- option to disable generation of random file sizes
- option to configure fill value to be used instead of random values
- option to force filling up the whole volume


More details in ReadMe.txt inside the archive.


Have fun