Fully KS 1.3 compatible large drive/partition FastFileSystem v44.5

[Amiga1992]

Originally Posted by Toni Wilen

But quick answer: for example if you already use FFS and don't bother changing it or just want to use FFS.

Thanks. That's great. It could be useful, for example, when using CD32Load. If it works with it (should!)

[Olaf Barthel]

Originally Posted by Akira

I thought Fast File System was garbage to begin with.

Don't be so harsh on the old file system, and the FFS which followed it.

In 1985 it was a perfectly adequate hierarchical file system for use on microcomputers. It did not suffer from artificial limitations on the number of files per directory or the directory nesting level.

It kept checksums on all metadata and data blocks, and (almost) managed to both detect and recover from damage. File names would use the ISO-8859-1 character set (not just plain 7 bit ASCII), and the maximum file name length (31 characters) wasn't so bad, considering what everybody else had to put up with.

Just imagine Commodore settling for something like GEMDOS and its FAT file system, like Atari did, or Apple's original Macintosh file system, followed by the various HFS incarnations. All of that would have been so much worse. Microsoft never managed to really kick FAT to the curb until NTFS was ready for the mass market in 2001.

The FFS was, technically, quite the impressive implementation which mixed packet and device I/O to great effect. That was almost what you could have called an event-driven architecture. Most of the time, the FFS would just sit there idle, waiting for I/O operations to complete, and if you had a disk controller capable of DMA, then the FFS didn't even have to copy any data around. If the stars aligned, you'd have a very low latency, low CPU load file system layer.

I'd say that the original file system and the FFS did better than average, but the downsides were hard to ignore. Back in 1985 nobody really expected a floppy disk file system to cache data yet to be written to the medium, which made this a rather error prone solution (it really was a file system intended for larger mass storage devices, shoehorned for use with floppy disks). Ejecting the disk at the wrong time would require the validator to repair the damage, and when that failed (when rather than if), the DiskDoctor to finish it off. Good intentions, no doubt, but poor execution.

All the technology which eventually displaced the 1970'ies file system designs, of which the original file system and FFS were typical examples, wasn't ready until the early 1990'ies. That's when OFS and FFS had started to smell really rank.

By this time, also owing to disks with much more than 20 Megabytes of storage (practical limit for OFS on-disk data structures) becoming cheaper and cheaper by the month, the limitations of the OFS/FFS designs really hit home. The file system data structures, both on-disk and in-memory, were all built around lists rather than trees. The scalability just wasn't there, by design. The joke was that if you allowed the file system to use more memory buffers for caching, the slower it became because it had to search the buffer list for the matching block.

Write-ahead journaling could have been retrofitted into the FFS, at least, but it didn't happen because it was not an obvious extension to the existing design. It would have done away with the validation process and made all write operations (including on-disk data structure changes) both faster and resistant to crash damage.

[Amiga1992]

Fair enough, I was being too harsh. But from the get-go I suffered those stupid Validation issues whenever a shitty program crashed while the disk was being accessed somehow, and they were really painful to stomach, and that is when there were no other errors after validation. I really grew to hate it, haha.

[kolla]

There are ways to avoid that problem, if you have some spare RAM

[Olaf Barthel]

Originally Posted by Akira

Fair enough, I was being too harsh. But from the get-go I suffered those stupid Validation issues whenever a shitty program crashed while the disk was being accessed somehow, and they were really painful to stomach, and that is when there were no other errors after validation. I really grew to hate it, haha.

The file system turned out to be a poor fit for this operating system.

Both Kickstart and Workbench software suffered greatly from stability problems until the entire code was reviewed and reworked for Kickstart/Workbench 2.0. Some of the big problems resulted from not testing for memory allocation failure, from failing to propagate error conditions through the callchain, and (who knew) operating system code relying on the contents of scratch registers.

Commodore could address these issues, but 3rd party developers often struggled to produce robust software: the Amiga operating system design is very unforgiving when it comes to accidental memory corruption. When it was new, the Amiga operating system was much, much more complex than what came before it, if you were familiar only with the hobbyist and home computers that shipped in the previous 10 years.

Finding the right spot in which the floppy disks were not a nuisance and the file system didn't cause trouble was possible, but it was damn hard to get there

I recall that for me the turning point came when I bought an external disk drive. Finally, I was able to use the external drive to make my first steps into becoming a better programmer, by keeping the project data on it rather than on the boot floppy. Making backups also became much easier. Containing the damage, mitating the risk, was possible at last.

Still, you could not rely upon the file system, and just trying to mitigate the risks made it harder what was already hard to begin with: Amiga software development. You probably know the many anecdotal stories of programmers losing all their work just because one floppy disk failed, no usable backups were available and repairs did not succeed.

[mark_k]

Originally Posted by Olaf Barthel

Sorry, the recorded development history (RCS files, etc.) of the FFS, as implemented by Steve Beats, does not go any further back in time than version 34.3 (no version string, and no proper date stamp on the files, so I guess this might have been a 1987 or 1988 release). That version assumes a minimum block size of 512 bytes, there's even the telltale 'moveq #9,d1' for translating byte into block offsets (like trackdisk.device, hddisk.device, etc. prefer it).

It's possible, but I somewhat doubt that a 256 byte option escaped from the lab

It's possible that I mis-remembered of course. But I seem to remember reading a changelog for FFS (would that be on one of the developer CDs?) mentioning removal of support for 256-byte blocks.

If it was ever supported, it was probably first possible with some post-1.3 FFS. Looking at the code here, v36.35 doesn't have the MOVEQ #9,D1 and LSR sequence, whereas
34.85 (WB 1.3) and 36.03 (WB 1.3.3) both do.

I'll try testing various FFS versions in WinUAE to see whether I'm wrong about that. Update: At least FFS 36.104 does support 256-byte filesystem block size! I'll upload WinUAE config and sample HDF in a separate thread.

[Toni Wilen]

r3 update:

- Force Direct SCSI if device driver is scsi.device, all scsi.device variants support it and at least CDTV scsi.device (and probably older A590/A2091 ROMs too) don't return error when calling unsupported functions, causing TD64 misdetection.
- Fixed bug in SCSI direct code, it never set io_Length in SCSI direct mode. TD64 check sets it to larger than required value (block size) but nothing sets it if TD64 is disabled.
- Always disable TD64 capability test under KS 1.3.

Filesystem is now fully compatible with CDTV SCSI adapter and A590/A2091 pre-7.0 ROMs (if some of them have same bug as CDTV scsi.device), also bypasses 1G drive size limit.

[Olaf Barthel]

Originally Posted by mark_k

It's possible that I mis-remembered of course. But I seem to remember reading a changelog for FFS (would that be on one of the developer CDs?) mentioning removal of support for 256-byte blocks.

If it was ever supported, it was probably first possible with some post-1.3 FFS. Looking at the code here, v36.35 doesn't have the MOVEQ #9,D1 and LSR sequence, whereas
34.85 (WB 1.3) and 36.03 (WB 1.3.3) both do.

I'll try testing various FFS versions in WinUAE to see whether I'm wrong about that. Update: At least FFS 36.104 does support 256-byte filesystem block size! I'll upload WinUAE config and sample HDF in a separate thread.

I just found out the awful truth: the original FFS does not seem to care about the sector size at all. Sector sizes which are not a power of two are fine. So are sector sizes of odd numbers, for example. Just don't use a number smaller than 1, because then either a) memory allocations might fail or b) the log2(sector size) calculation might slip up. Sector sizes smaller than some 240 bytes will be problematic, because then the number of hash table entries in a directory will lead to instant corruption of said data structures.

Seems that the only guard against utter hilarity was in that storage device drivers would not necessary allow for sector sizes smaller than 512 bytes, or for that matter, sector sizes not divisible by 512 bytes. However, one probably shouldn't be so sure about that either, considering how rough-and-tumble many Amiga device drivers used to be

Needless to say, if you want to go ahead and experiment with the software, make sure you have adequate backups of your data at hand, make sure, too, that you can restore them if you need to, and please take pictures of the enfolding desaster and share them with the eager readers of this forum

[Toni Wilen]

Change of plans. "FFSAIO" will be released soon after all

Current changelog (1.x requesters to do and testing):

Based on v45.13 (OS3.9BB2 version) which is KS2.0+ and NSD only (if partition is outside of first 4G).

Patches:

- Full Kickstart 1.3 compatibility.
- TD64 and DirectSCSI support and autodetection.
- TD_GETGEOMETRY/Direct SCSI/TD64 blacklist from v44.5
- V45.14 validator FreeMem() fix. (I am not sure what v45.16 bug fix does)

Autodetection is different than in PFS3AIO because dostype can't be used to configure detection mode.

- NSD and TD64 detection is skipped if partition is inside first 4G.
- NSD detection is first, NSCMD_DEVICEQUERY must return sane results.
- TD64 detection is next if NSD was not detected and device driver was not in blacklist.

- Direct SCSI is last and detection is only executed if no NSD and no TD64 was detected and device driver was not in blacklist. Partition location (inside 4G or not) is not checked. (To support old A590/A2091 and CDTV SCSI with 1G limit)
- Direct SCSI detection executes HD_SCSICMD READ CAPACITY which must return valid data. Direct SCSI is not selected if returned drive size is less than 1G.

[thebajaguy]

I recall Bernoulli SCSI removable drives having 256-byte sectors. I remember the original GVP scsidev.device 'Advanced Autoboot v2.2' ROMs being the ones needed to deal with it until later gvpscsi.device 3.x and higher handled things much better. I also recall forcing Quantum disk drives into 2048-block mode for testing. Although not a FFS/OFS issue, I somehow recall there being some CD-ROM devices that in some modes could return 256-byte sectoring. <br />
<br />
That's nearly a 30-yr memory dusted off...

[mark_k]

SyQuest 44MB disks also have 256-byte sectors, though it may be that some drives' firmware emulated 512 bytes.

If you still have those Advanced Autoboot v2.2 ROMs, please dump them, I'd like to look at the code.

[Toni Wilen]

Originally Posted by Toni Wilen

Change of plans. "FFSAIO" will be released soon after all

Here is first test version. Do not upload to Aminet! Do not add to any patch packages!

"FFSAIO"

Based on official v45.13 (OS3.9BB2 version) which is KS 2+ and NSD only (if partition is not located inside first 4G).

Patches:

- Full Kickstart 1.3 compatibility (see below).
- TD64 and DirectSCSI support and autodetection.
- TD_GETGEOMETRY/Direct SCSI/TD64 blacklist from v44.5.
- v45.14 validator FreeMem() fix and v45.16 bitmap size calculation off by one bitmap bit fix.
- ACTION_FH_FROM_LOCK freed the lock if call failed. (Out of memory or lock pointed to a directory)
- Fixed wrong offset (which was supposed to point at the end of block buffer) used in some DirCache-only routine.
- Test read last block of partition, if read fails, partition won't be mounted.

- WARNING! ULTRA-UNSUPPORTED!: Enabled and fixed hidden partially implemented long file name support. If partition dos type is DOS\8, max file/directory name length gets increased to 54 characters (from 30) by using previously unused bytes of file/dir header block. WARNING^2: No recovery tools.

Information:

Autodetection is different than in PFS3AIO because dostype can't be used to configure detection mode:

- NSD and TD64 detection is skipped if partition is fully inside first 4G.
- NSD detection is first, NSCMD_DEVICEQUERY must return sane results (No changes in NSD detection, uses original code).
- TD64 detection is next if NSD was not detected and device driver was not in blacklist.

- Direct SCSI is last and detection is only executed if no NSD and no TD64 was detected and device driver was not in blacklist. Partition location (inside 4G or not) is ignored.
- Direct SCSI detection executes HD_SCSICMD READ CAPACITY which must return valid data. Direct SCSI is not selected if returned drive size is less than 1G.

KS 1.3 patches:

- ACTION_STARTUP dp_Arg1 and dp_Arg3 are not valid under KS 1.3. (Same patch as in PFS3AIO)
- utility.library UMult32 and UDivMod32 calls replaced.
- Requesters (read/write error, you must insert, validation error) replaced with calls to BCPL makesysreq().
- WB 1.3 "ghost" icon fix. ACTION_INFO/ACTION_DISK_INFO must always return DOS\0, if dos type is DOS\1 or higher.

Instructions:

Download patch file http://www.winuae.net/files/b/fastfi...m_45.20_r1.zip
Unpack patch
Find OS39BB2 L:FastFilesystem (45.13)
CD to directory where patch file was unpacked
Copy 45.13 FastFileSystem to current directory
Create 45.20: spatch -ofastfilesystem_45.20 -pfastfilesystem_45.20_r1.pch fastfilesystem
Copy/Rename fastfilesystem_45.20 to L: (or somewhere else)
Use HDToolBox to install it.

NOTE: This thread is not for generic filesystem/hdtoolbox instructions.

FINAL WARNING: You should know what are you doing! Take backups. There is no guarantees. There is no support.

[mark_k]

For anyone brave enough to test the long filename support...

DOS\8 is identical to DOS\3 (international FFS) when there are no files with long names. So you can in-place convert an existing DOS\3 partition to DOS\8 just by changing the DosType in the first longword (and edit your mount file to reflect the new DosType).

Similarly, you could convert back to DOS\3, providing you're certain that there are no files or directories with long names in the partition.

Programs like DiskSalv and ReOrg won't of course work with DOS\8 partitions.

[Aladin]

ffs 45.20 compatible too with os4 classic and morphos classic?

[E-Penguin]

Was there ever a final release of FFSAIO? I'm in the mood to fiddle with my A590 / scsi2sd and thought I'd give this a go.

[Toni Wilen]

I decided to wait for OS 3.1.4 update FFS. (when/if it gets released)

[E-Penguin]

Fair enough, thanks for the update

[Jope]

Originally Posted by Toni Wilen

I decided to wait for OS 3.1.4 update FFS. (when/if it gets released)

Woo, it's out. :-)

[Toni Wilen]

Originally Posted by jdryyz

Hope I'm not hijacking this thread.

I think you are. This is quite specific thread about KS 1.3 patched FFS version, not generic help thread.

Back to topic: I now have KS 1.3 patch for FFS 46.13.

Unfortunately it uses even more dos/Fault() calls than any older versions to generate error requester contents before displaying it which is pain to patch because BCPL only has error requester function that takes single error code as a parameter. (No, I am not going to reimplement those using intuition.library calls..)

So either I release it in this form or it is not going to get released

It means all filesystem generated error requesters are disabled and error is returned to dos immediately (Same as pressing Cancel). Includes errors like read/write error, validation error, and I guess disk full during writing. Dos generated errors like write protected or no disk in drive are not affected.

[schlika]

Hello !

I'd like testing that patch but download links are now dead and archive.org has no trace of it. Could you upload it somewhere else ?

Thanks a lot !

Originally Posted by Toni Wilen

Here is first test version. Do not upload to Aminet! Do not add to any patch packages!

"FFSAIO"

Based on official v45.13 (OS3.9BB2 version) which is KS 2+ and NSD only (if partition is not located inside first 4G).

Patches:

- Full Kickstart 1.3 compatibility (see below).
- TD64 and DirectSCSI support and autodetection.
- TD_GETGEOMETRY/Direct SCSI/TD64 blacklist from v44.5.
- v45.14 validator FreeMem() fix and v45.16 bitmap size calculation off by one bitmap bit fix.
- ACTION_FH_FROM_LOCK freed the lock if call failed. (Out of memory or lock pointed to a directory)
- Fixed wrong offset (which was supposed to point at the end of block buffer) used in some DirCache-only routine.
- Test read last block of partition, if read fails, partition won't be mounted.

- WARNING! ULTRA-UNSUPPORTED!: Enabled and fixed hidden partially implemented long file name support. If partition dos type is DOS\8, max file/directory name length gets increased to 54 characters (from 30) by using previously unused bytes of file/dir header block. WARNING^2: No recovery tools.

Information:

Autodetection is different than in PFS3AIO because dostype can't be used to configure detection mode:

- NSD and TD64 detection is skipped if partition is fully inside first 4G.
- NSD detection is first, NSCMD_DEVICEQUERY must return sane results (No changes in NSD detection, uses original code).
- TD64 detection is next if NSD was not detected and device driver was not in blacklist.

- Direct SCSI is last and detection is only executed if no NSD and no TD64 was detected and device driver was not in blacklist. Partition location (inside 4G or not) is ignored.
- Direct SCSI detection executes HD_SCSICMD READ CAPACITY which must return valid data. Direct SCSI is not selected if returned drive size is less than 1G.

KS 1.3 patches:

- ACTION_STARTUP dp_Arg1 and dp_Arg3 are not valid under KS 1.3. (Same patch as in PFS3AIO)
- utility.library UMult32 and UDivMod32 calls replaced.
- Requesters (read/write error, you must insert, validation error) replaced with calls to BCPL makesysreq().
- WB 1.3 "ghost" icon fix. ACTION_INFO/ACTION_DISK_INFO must always return DOS\0, if dos type is DOS\1 or higher.

Instructions:

Download patch file http://www.winuae.net/files/b/fastfi...m_45.20_r1.zip
Unpack patch
Find OS39BB2 L:FastFilesystem (45.13)
CD to directory where patch file was unpacked
Copy 45.13 FastFileSystem to current directory
Create 45.20: spatch -ofastfilesystem_45.20 -pfastfilesystem_45.20_r1.pch fastfilesystem
Copy/Rename fastfilesystem_45.20 to L: (or somewhere else)
Use HDToolBox to install it.

NOTE: This thread is not for generic filesystem/hdtoolbox instructions.

FINAL WARNING: You should know what are you doing! Take backups. There is no guarantees. There is no support.