18 July 2023, 16:47 | #1 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
PFS3aio v3.2 test
As I mentioned in pfs3aio v3.1 news thread, I decided to add >512 logical block size support to pfs3aio.
READ BELOW FULLY BEFORE DOWNLOADING. YOU HAVE BEEN WARNED! I don't have time or interest to do any deep testing so here it is. Only read and download if you want to test if 4096 block size improves performance and/or reduce memory usage. Don't download only because you want to use latest version because it is latest version. --- pfs3aio v3.2 (v20.0) Development test version. 29.07.2023. WARNING: >512 block size requires use of still experimental large partition (>106G) mode, even if partition is smaller than 106G! You MUST have backup of whole partition! Data loss is always possible! Tests done so far are simple "copy random files until partition is full, then delete half of it and copy again and repeat few times + pfsdoctor check after each step" Don't use this if you are not sure what are you doing or don't want anything possibly unexpected to happen! Changes from v3.1 to v3.2: Test 5: - Fixed DirectSCSI mode. DirectSCSI didn't work correctly and caused corruption if >512 block size. Pfsdoctor DirectSCSI mode was also broken. - If for some reason max transfer is smaller than block size: return error instead of hanging in infinite loop. - Reserved block allocation increased again. Test 4: - Larger partition size reserved block increased yet again. (500 000+ files/directories again possible with larger partitions now) - Implemented (forgotten because it was v39 introduced?) ACTION_EXAMINE_ALL_END. Almost no-operation packet handler function, no need to free anything. Probably does not make much difference because dos.library emulates this if it is not implemented. - Allow file sizes up to 4G-1 (previously was 2G-1), allows smoother on the fly switch to >=4G mode if needed in the future. >=4G support mode is currently disabled. Test 3: - Adjusted reserved block calculation again, low partition sizes were lower than expected. Increase larger partition reserved size again. - Added workaround for pre-v20 pfs3(aio) partitions that have >512 block size, now old partitions with >512 block mounts automatically, without need to manually adjust block size to 512. Pre-v20 versions ignored block size and always used default 512 block size. Test 2: - Adjusted reserved block calculation, test1 reduced amount of reserved blocks too much Test 1: - Logical block sizes >512 (1024, 2048, 4096) supported. Previously pfs3 ignored set block size and always used 512 byte block size for data blocks and 1024 for metadata blocks. - Disables v3.1 introduced free space hack and forced max transfer limit if OS 3.2+ or newer is detected. - Geometry used when formatted is stored in root extension block. Used when mounting pfs3 partition in superfloppy mode (LOWCYL=0), overriding current, possibly incorrectly guessed, geometry. - Amount of reserved blocks reduced in large partition sizes (>5G+ partition size allocated far too many reserved blocks). Reserved blocks are blocks that are allocated when partition is formatted, all metadata is stored in reserved blocks (bitmaps, indexes, file/dir header blocks etc). This will reduce amount of RAM allocated when partition is mounted. - pfsdoctor updated to support >512 logical block sizes. Added backwards compability check for old pfs3 partitions that have (previously always ignored) >512 block size. - pfsdoctor INFO option added, shows internal details of pfs3 partition, does not do any checks. Details: >512 block size support requires formatting. Use HDToolBox to update pfs3aio. Version field should get automatically filled with 20 (Remember to set correct dostype, if you don't know what dostype is: STOP IMMEDIATELY!) and set larger block size (it might already been set to value higher than 512 but it made no difference previously). >4096 is not supported and causes format to fail. I recommend using 4096, especially if you use any kind of solid state storage device. During formatting (Use Quick format!) warning message will appear. (PBS=physical block size, almost always 512,LBS=logical block size, this is the one you selected in hdtoolbox, 512/1024/2048/4096. RBS=reserved block size, same as logical block size except if LBS is 512, RBS minimum is 1024). Check that values match. pfsdoctor INFO option also lists these values. At least in theory this should improve performance on solid state devices (HDF files might not get any noticeably faster because host OS usually does write back caching and combines multiple separate writes) and it can also reduce memory usage because pfs3 needs to keep reserved block bitmap in RAM and RAM usage is based on number of blocks, the larger the block size, the less blocks are needed. I am waiting for speed test results :) Do this before continuing normal use (and hope no sudden random corruption): Change max file size (if needed) using setfnsize and/or configure deldir (setdeldir), both come with official aminet pfs3_53.lha package. Copy data to newly formatted partition ("copy xyz: zyx: all clone quiet". Make sure shell stack is large enough if you have deep directory structure or use long file names), then run pfsdoctor CHECK. It should not report anything wrong. If it does report errors or warnings: DO NOT USE PARTITION, DO NOT ATTEMPT TO FIX IT. Include whole pfsdoctor output ("pfsdoctor check xyz >ram:output.txt") and report the problem. Reboot and rerun it again to confirm. --- (Above text is also included with the archive). https://download.abime.net/winuae/fi...v3.2_test5.zip (Test 5 update) Last edited by Toni Wilen; 29 July 2023 at 08:52. |
18 July 2023, 22:27 | #2 |
Paranoid Amigoid
Join Date: Mar 2008
Location: Athens/Greece
Age: 45
Posts: 1,978
|
Thanks for the update Toni.
Will give it a test for sure (probably at weekend when I'll have more free time). I guess it's not needed to repartition the CF and just update and change block size (from 1024 that I have it now to 4096) and reformat. Although I haven't really understood the last edit which might affect me as I'm using 1024? |
19 July 2023, 09:18 | #3 | |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Quote:
Extra note only explains this situation: if you have pre-v20 pfs3 formatted with 1024 or larger block size (which was technically wrong), it fails to mount when using above version because block size configuration does not match. I probably add workaround for this situation in future versions. Block size can be safely set to correct 512 (Note that hdtoolbox will complain that data will be lost) to fix it. |
|
19 July 2023, 13:11 | #4 |
Registered User
Join Date: Aug 2016
Location: Germany
Posts: 123
|
I have no idea what is going on! :-P
After copying ~15.3 GB, the system claims that the partition is full. dh1: Previous pfs version dh1_0: pfs Beta WinUAE 5.0.0 Last edited by pcotter; 19 July 2023 at 13:15. Reason: gram |
19 July 2023, 13:42 | #5 | |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Quote:
EDIT: pfsdoctor check didn't find anything wrong? |
|
19 July 2023, 14:20 | #6 |
Paranoid Amigoid
Join Date: Mar 2008
Location: Athens/Greece
Age: 45
Posts: 1,978
|
Is it safe to use different PFS versions per partition?
I thought that putting same version is the correct way. |
19 July 2023, 14:24 | #7 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
|
19 July 2023, 14:38 | #8 |
Registered User
Join Date: Aug 2016
Location: Germany
Posts: 123
|
I do have some files :-)
The doctor didn't find anything strange... Validating reserved 0... . Validating main 217... .......................................................................................................................................................................................................................... Validating anodes 10848... ....................................................................... finishing up 2... .. Last edited by pcotter; 19 July 2023 at 14:39. Reason: gram |
19 July 2023, 17:16 | #9 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Uploaded test2. Re-adjusted reserved block allocation logic, larger now than in previous version but still less than in previous official version.
|
19 July 2023, 18:55 | #10 |
Registered User
Join Date: Aug 2016
Location: Germany
Posts: 123
|
Success! :-)
I will now try to fill a 128GB SD to check if it also fails. |
19 July 2023, 20:52 | #11 |
Apollo Team
Join Date: May 2014
Location: not far
Posts: 381
|
With test2, I quickly get Disk full error once trying to unlzx my system partition (60MB) to a newly partitioned and formatted 250MB partition.
Formatted with pfsformat NOICONS FNSIZE=107 QUICK |
19 July 2023, 21:25 | #12 | |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Quote:
pfsdoctor info output needed. Are you sure it isn't really full because if you have huge amount of tiny files, 4096 blocksize vs 512 can waste lots of space and reserved blocks might take many megabytes of free space. (Every single file will take at least 1 block, even if it is 1 byte in size) |
|
19 July 2023, 22:14 | #13 |
Apollo Team
Join Date: May 2014
Location: not far
Posts: 381
|
Code:
Autodetected disk access mode: Standard Initializing rootblock 100... Checking rootblock Checking disk System2 Option mask: 000017ff Reserved blocksize: 4096 Reserved block cluster: 1 First reserved block: 2 Last reserved block: 1249 Reserved blocks: 1248 Free reserved blocks: 10 Free blocks: 10262 Always free blocks: 3777 Total blocks: 614400 .. Checking rext Max file name length: 107 Disk formatted with PFS-III 0.20 |
20 July 2023, 06:46 | #14 |
Registered User
Join Date: Aug 2016
Location: Germany
Posts: 123
|
111GB partition.
|
20 July 2023, 11:13 | #15 | |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Quote:
EDIT: No need, I rechecked old algorithm results and small sizes were affected. Another update coming soon.. Last edited by Toni Wilen; 20 July 2023 at 12:35. |
|
20 July 2023, 18:55 | #16 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Test 3:
- Adjusted reserved block calculation again. Low partition sizes were lower than expected. Increased larger partition reserved block size again. - Added workaround for pre-v20 pfs3(aio) partitions that have >512 block size, now old partitions with >512 block mounts automatically, without need to manually adjust block size to 512. Pre-v20 versions ignored block size and always used default 512 block size. This really should fix remaining out of disk space errors without actually being out of disk space. |
20 July 2023, 21:37 | #17 |
Registered User
Join Date: Aug 2016
Location: Germany
Posts: 123
|
Not yet boss.
Copied maybe the double amount of files and failed. |
21 July 2023, 11:58 | #18 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
Getting quite extreme
Yeah, it seems block allocation still is not large enough. |
22 July 2023, 09:45 | #19 |
Registered User
Join Date: Nov 2016
Location: Skien / Norway
Posts: 112
|
Do you want any test done on emulator or FPGA?
|
22 July 2023, 10:09 | #20 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,611
|
I'd like to see some speed results using memory cards/SSD drives (512 vs 4096 block size differences), FPGAs or real HWs. Manually copying and timing lots of small or few large files. DiskSpeed also should work. Raw read results (Sysinfo) are useless.
Emulators can't be used to get useful disk speed test results because host OS handles the disk access. (I am too lazy to do any real HW tests..) |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
PFS3AIO v3.1 Final | Toni Wilen | News | 501 | 22 September 2023 21:55 |
Pfs3aio | caver99 | support.Apps | 10 | 15 February 2022 19:03 |
Req: '240p Test Suite' similar test software for Amiga? | c0dehunter | support.Hardware | 10 | 22 February 2021 21:38 |
Pfs3aio ???? | stingray76 | support.Hardware | 2 | 19 February 2014 09:15 |
CD32 Frog Feast test available. Test out the final! | cdoty | News | 42 | 01 April 2008 16:20 |
|
|