PFS3aio v3.2 test

[Toni Wilen]

As I mentioned in pfs3aio v3.1 news thread, I decided to add >512 logical block size support to pfs3aio.

READ BELOW FULLY BEFORE DOWNLOADING. YOU HAVE BEEN WARNED!

I don't have time or interest to do any deep testing so here it is.

Only read and download if you want to test if 4096 block size improves performance and/or reduce memory usage. Don't download only because you want to use latest version because it is latest version.


---

pfs3aio v3.2 (v20.0) Development test version. 29.07.2023.

WARNING: >512 block size requires use of still experimental large partition (>106G) mode, even if partition is smaller than 106G! You MUST have backup of whole partition! Data loss is always possible! Tests done so far are simple "copy random files until partition is full, then delete half of it and copy again and repeat few times + pfsdoctor check after each step"

Don't use this if you are not sure what are you doing or don't want anything possibly unexpected to happen!

Changes from v3.1 to v3.2:

Test 5:

- Fixed DirectSCSI mode. DirectSCSI didn't work correctly and caused corruption if >512 block size. Pfsdoctor DirectSCSI mode was also broken.
- If for some reason max transfer is smaller than block size: return error instead of hanging in infinite loop.
- Reserved block allocation increased again.

Test 4:

- Larger partition size reserved block increased yet again. (500 000+ files/directories again possible with larger partitions now)
- Implemented (forgotten because it was v39 introduced?) ACTION_EXAMINE_ALL_END. Almost no-operation packet handler function, no need to free anything. Probably does not make much difference because dos.library emulates this if it is not implemented.
- Allow file sizes up to 4G-1 (previously was 2G-1), allows smoother on the fly switch to >=4G mode if needed in the future. >=4G support mode is currently disabled.

Test 3:

- Adjusted reserved block calculation again, low partition sizes were lower than expected. Increase larger partition reserved size again.
- Added workaround for pre-v20 pfs3(aio) partitions that have >512 block size, now old partitions with >512 block mounts automatically, without need to manually adjust block size to 512. Pre-v20 versions ignored block size and always used default 512 block size.

Test 2:

- Adjusted reserved block calculation, test1 reduced amount of reserved blocks too much

Test 1:

- Logical block sizes >512 (1024, 2048, 4096) supported. Previously pfs3 ignored set block size and always used 512 byte block size for data blocks and 1024 for metadata blocks.
- Disables v3.1 introduced free space hack and forced max transfer limit if OS 3.2+ or newer is detected.
- Geometry used when formatted is stored in root extension block. Used when mounting pfs3 partition in superfloppy mode (LOWCYL=0), overriding current, possibly incorrectly guessed, geometry.
- Amount of reserved blocks reduced in large partition sizes (>5G+ partition size allocated far too many reserved blocks). Reserved blocks are blocks that are allocated when partition is formatted, all metadata is stored in reserved blocks (bitmaps, indexes, file/dir header blocks etc). This will reduce amount of RAM allocated when partition is mounted.

- pfsdoctor updated to support >512 logical block sizes. Added backwards compability check for old pfs3 partitions that have (previously always ignored) >512 block size.
- pfsdoctor INFO option added, shows internal details of pfs3 partition, does not do any checks.

Details:

>512 block size support requires formatting. Use HDToolBox to update pfs3aio. Version field should get automatically filled with 20 (Remember to set correct dostype, if you don't know what dostype is: STOP IMMEDIATELY!) and set larger block size (it might already been set to value higher than 512 but it made no difference previously). >4096 is not supported and causes format to fail. I recommend using 4096, especially if you use any kind of solid state storage device.

During formatting (Use Quick format!) warning message will appear. (PBS=physical block size, almost always 512,LBS=logical block size, this is the one you selected in hdtoolbox, 512/1024/2048/4096. RBS=reserved block size, same as logical block size except if LBS is 512, RBS minimum is 1024). Check that values match. pfsdoctor INFO option also lists these values.

At least in theory this should improve performance on solid state devices (HDF files might not get any noticeably faster because host OS usually does write back caching and combines multiple separate writes) and it can also reduce memory usage because pfs3 needs to keep reserved block bitmap in RAM and RAM usage is based on number of blocks, the larger the block size, the less blocks are needed. I am waiting for speed test results :)

Do this before continuing normal use (and hope no sudden random corruption): Change max file size (if needed) using setfnsize and/or configure deldir (setdeldir), both come with official aminet pfs3_53.lha package. Copy data to newly formatted partition ("copy xyz: zyx: all clone quiet". Make sure shell stack is large enough if you have deep directory structure or use long file names), then run pfsdoctor CHECK. It should not report anything wrong. If it does report errors or warnings: DO NOT USE PARTITION, DO NOT ATTEMPT TO FIX IT. Include whole pfsdoctor output ("pfsdoctor check xyz >ram:output.txt") and report the problem. Reboot and rerun it again to confirm.

---

(Above text is also included with the archive).

https://download.abime.net/winuae/fi...v3.2_test5.zip (Test 5 update)

[mfilos]

Thanks for the update Toni.
Will give it a test for sure (probably at weekend when I'll have more free time).
I guess it's not needed to repartition the CF and just update and change block size (from 1024 that I have it now to 4096) and reformat. Although I haven't really understood the last edit which might affect me as I'm using 1024?

[Toni Wilen]

Originally Posted by mfilos

Thanks for the update Toni.
Will give it a test for sure (probably at weekend when I'll have more free time).
I guess it's not needed to repartition the CF and just update and change block size (from 1024 that I have it now to 4096) and reformat. Although I haven't really understood the last edit which might affect me as I'm using 1024?

Repartitioning is not needed, only pfs3 update + block size change + reboot.

Extra note only explains this situation: if you have pre-v20 pfs3 formatted with 1024 or larger block size (which was technically wrong), it fails to mount when using above version because block size configuration does not match. I probably add workaround for this situation in future versions. Block size can be safely set to correct 512 (Note that hdtoolbox will complain that data will be lost) to fix it.

[pcotter]

I have no idea what is going on! :-P
After copying ~15.3 GB, the system claims that the partition is full.
dh1: Previous pfs version
dh1_0: pfs Beta
WinUAE 5.0.0

[Toni Wilen]

Originally Posted by pcotter

I have no idea what is going on! :-P
After copying ~15.3 GB, the system claims that the partition is full.
dh1: Previous pfs version
dh1_0: pfs Beta
WinUAE 5.0.0

How many files/directories do you have? (I assume a lot?) It looks like reserved block count is now too small.

EDIT: pfsdoctor check didn't find anything wrong?

[mfilos]

Is it safe to use different PFS versions per partition?
I thought that putting same version is the correct way.

[Toni Wilen]

Originally Posted by mfilos

Is it safe to use different PFS versions per partition?
I thought that putting same version is the correct way.

You can only safely do this if different versions have different dostypes (for example PFS\1 and PFS\2 etc. PFS does not care about dostype)

[pcotter]

I do have some files :-)
The doctor didn't find anything strange...

Validating reserved 0...
.
Validating main 217...
..........................................................................................................................................................................................................................
Validating anodes 10848...
.......................................................................
finishing up 2...
..

[Toni Wilen]

Uploaded test2. Re-adjusted reserved block allocation logic, larger now than in previous version but still less than in previous official version.

[pcotter]

Success! :-)

I will now try to fill a 128GB SD to check if it also fails.

[TuKo]

With test2, I quickly get Disk full error once trying to unlzx my system partition (60MB) to a newly partitioned and formatted 250MB partition.

Formatted with pfsformat NOICONS FNSIZE=107 QUICK

[Toni Wilen]

Originally Posted by TuKo

With test2, I quickly get Disk full error once trying to unlzx my system partition (60MB) to a newly partitioned and formatted 250MB partition.

Formatted with pfsformat NOICONS FNSIZE=107 QUICK

"Small" partitions (less than 1G) should have same reserved block size but 4k block size might affect its behavior.

pfsdoctor info output needed.

Are you sure it isn't really full because if you have huge amount of tiny files, 4096 blocksize vs 512 can waste lots of space and reserved blocks might take many megabytes of free space. (Every single file will take at least 1 block, even if it is 1 byte in size)

[TuKo]

Autodetected disk access mode: Standard
Initializing
rootblock 100...
Checking rootblock
Checking disk System2
Option mask: 000017ff
Reserved blocksize: 4096
Reserved block cluster: 1
First reserved block: 2
Last reserved block: 1249
Reserved blocks: 1248
Free reserved blocks: 10
Free blocks: 10262
Always free blocks: 3777
Total blocks: 614400
..
Checking rext
Max file name length: 107
Disk formatted with PFS-III 0.20

[pcotter]

111GB partition.

[Toni Wilen]

Originally Posted by TuKo

With test2, I quickly get Disk full error once trying to unlzx my system partition (60MB) to a newly partitioned and formatted 250MB partition.

Formatted with pfsformat NOICONS FNSIZE=107 QUICK

Could you do same with 512 block sizes? (and include pfsdoctor info output) As a comparison because only difference should be block size. I guess it is possible >1024 reserved block size wastes some more space which could explain the difference.

EDIT: No need, I rechecked old algorithm results and small sizes were affected. Another update coming soon..

[Toni Wilen]

Test 3:

- Adjusted reserved block calculation again. Low partition sizes were lower than expected. Increased larger partition reserved block size again.
- Added workaround for pre-v20 pfs3(aio) partitions that have >512 block size, now old partitions with >512 block mounts automatically, without need to manually adjust block size to 512. Pre-v20 versions ignored block size and always used default 512 block size.

This really should fix remaining out of disk space errors without actually being out of disk space.

[pcotter]

Not yet boss.
Copied maybe the double amount of files and failed.

[Toni Wilen]

Getting quite extreme

Yeah, it seems block allocation still is not large enough.

[Solskogen]

Do you want any test done on emulator or FPGA?

[Toni Wilen]

Originally Posted by Solskogen

Do you want any test done on emulator or FPGA?

I'd like to see some speed results using memory cards/SSD drives (512 vs 4096 block size differences), FPGAs or real HWs. Manually copying and timing lots of small or few large files. DiskSpeed also should work. Raw read results (Sysinfo) are useless.

Emulators can't be used to get useful disk speed test results because host OS handles the disk access.

(I am too lazy to do any real HW tests..)