PFS3 19.0 issue - > ALERT: Wrong index block ID when copying folder with many files

[Yulquen74]

This is happening in WinUAE since I prefer it over my genuine machine when copying large amount of files into the CF card I use in my real A1200.

The last days I have unpacked thousands of LHA archives onto my CF card
in my emulation environment in WinUAE 4.0.0 (set up to resemble my real A1200, emulating Blizzard1260/132MB RAM, full 060 in compatible mode) without any issues, in fact I have had no (real) PFS issues since I started building up this CF card partition.

I downloaded an archive with close to 20000 game cover scans, intended for tinylauncher, where all files are contained within a single folder. As the archive was zipped, I unpacked the cover folder to my C:Temp drive (available as a "drive" in WinUAE), and when I try and copy this folder to CF-card from inside WinUAE, after a while it fails with the PFS error requester.

If I restart the emulated machine, everything is fine (pfsdoctor reports no errors), and it turns out 3500 - 4500 files was copied (I renamed the cover folder each time to see if the problem was bad sectors).

The problem is consistent for each attempt, and I have tried to remove all commodities (have also retired mcp for a while now), increase stack to huge value, increase partition buffers to 500, copy in workbench instead of dopus5, but it still fails.

The CF is a 16GB Kingston ultimate x266 card set up with one 7.8GB partition (just below the CHS limit), and now has about 1GB left. Total amount of files of the entire disk is approaching 250.000.

Is there a limit of how many files one can/should have in one single folder, or partition for that matter with PFS3, or does my problem lie elsewhere?

Thanks in advance for suggestions.

[talybont]

http://eab.abime.net/showthread.php?t=90778


- Bug fixes, including finally fixed nasty bug causing Wrong Index Block ID
(and others) if partition was >=5G and lots of small files were created
very quickly, for example unpacking archive with thousands of small files.
This bug has most likely existed since PFS3 introduced >=5G partition
support.

[Yulquen74]

I'm using the newest version, but to verify I re-downloaded the aminet version, updated the filesystem although the existing one was the same number of bytes, rebooted, and then did the same copy process over, but still failing, now at >9000 files.

Another observation, when rebooting and inspecting the destination folder, if I copy one of the images to RAM: and double-click it in dopus (lauches multiview) the picture displays fine. But if I double-click it in the >9000 file directory, sometimes I get a PFS3 error requester, and if it does load, it takes several seconds before the picture is shown on the screen (we are talking small <50KB iff images here).

Also, during copy process, it starts out fast, and then gets slower and slower to a point where it looks like it takes 1 second or more to copy each small picture before it ultimately fails.

The files I try and copy, is this archive:
ftp://ftp:amiga@grandis.nu/~Uploads/...rt-PAL-v0.3.7z

[Toni Wilen]

Use hardfile (with same size as CF) and repeat the test.

Removes lots of unknown variables (bad CF card, bad adapter, unreliable USB and so on..)

[Yulquen74]

I did, and it was able to copy all the files without errors.

But I had the same observation with regards to copy speed: At some point in the copy process, it went from many files per second to 1-2 files, but still faster than with CF card though. Doing copy tests from big folder on Amiga partition to RAM: showed that I was able to copy any file from the top of the list down to and including the file Indianapolis500_SCR4.iff fast, which is file no.9104 from the top.

All files from here down to the bottom copy very very slow, both with HDF file and CF card (Using HDF, 1-2 files per second, from CF card, less than 1 file per second).

Either something is very wrong in my setup, or PFS3 is just slow with that many files in the same folder.

With that said, if I copy files from the PC mounted folder inside WinUAE to RAM, no such speed problems exist.

I have tried with another CF card, and have the same result there.


Using the card in my real A1200 (060@85/fast-ata pio4), copying the big folder to somewhere else, the copy process starts to really slow down after a few thousand files, and looking at the HD LED it is lit almost continously during the copy process, even when copying only 1-2 files per second.

Does the PFS housekeeping increase that much when the number of files increase to a certain number?

Also, when copying files from the big folder to RAM I also experience the big slowdown in copying from file 9105 from the top, as in WinUAE (same image). Copy test done with dopus 5.91 and cli copy.
During copy, HD LED also implies heavy activity even when copying less than 1 small file per second.

With regards to PFS errors experienced in WinUAE, I have tried several CF cards, and something tells me it is not block errors on the cards themselves, at least not initially (unless the copy process wears it down in the process), but there may be something with the CF adapter or USB as you said.

All in all it seems having that many files in a single folder is a bad idea for now, but it would be nice to find out the cause of the problem.

[Toni Wilen]

I don't know (or want to know) too much about PFS3 internals but I haven't heard any other similar issues but Amiga directory containing 10000+ files is very rare.

Does increasing buffers help? (What is current size? Check what addbuffers return, not that hdtoolbox shows because pfs3 does some adjustments)

WinUAE direct HD access is relatively slow because there is no caching. Increasing buffers to something like 1000-2000 should increase performance noticeably.

[Yulquen74]

I had it set to 500 in RDB, but using addbuffers it maxes out at 600 if one tries to set it higher.