Directory hard drives with full support for file attributes

[FrodeSolheim]

The implementation of file attribute support with directory hard drives is complete and will be released with the next development version. It just remains to rewrite part of the code because a well-known operating system from Redmond lacks useful functions common on other systems...

I have also written a new documentation page about hard drive support in FS-UAE: http://fengestad.no/fs-uae/using-hard-drives, which includes this information (and also describes support for other hard drive types). Please let me know if you find any inaccuracies.

Here's how it works:

When FS-UAE looks up existing files in a directory hard drives, it initializes metadata about the file either from the host file system, or if it exists, from an accompanying metadata file.

When a file has an accompanying metadata file, date/time, file permissions and comment are read from the metadata file. If a metadata file does not exist, then:

• The file will get the default permissions: —-rwed.
• The file date/time will be read from the last modified timestamp of the host file.
• The file will have no file comment.

When files are created/modified from within the Amiga, the host file mtime (last modified time) is updated based on the Amiga file date/time.

FS-UAE then decides if it needs to store more information about the file in a metadata file:

• If the file has a file comment, FS-UAE will create a metadata file.
• If the file has non-default file permissions, FS-UAE will create a metadata file.
• FS-UAE reads back the last modified time from the host file, and checks that the time was stored with high enough precision (Amiga files are stored with 1/50s precision). If the time could not be stored accurately, FS-UAE will create a metadata file.
• If the file already has a metadata file, FS-UAE will always update the metadata file.

Additional metadata is stored in files with .uaem extension. If the original file is called Image.jpg, the additional metadata is stored in Image.jpg.uaem in the same directory. The metadata is stored in a text format, and you can even edit this file manually from the host side to alter the information about the file. Here’s an example of the content of a metadata file (where "hello" is a file comment):

----rwed 2013-02-12 21:20:52.02 hello

I will probably later add support for:

• Forcing metadata files to always be written (useful if one want to archive / unarchive directories, since archives may not store time with enough precision).
• Forcing metadata files to never be written (useful if you don't need them, and don't want to clutter the hard drive directories with metadata files).

[FrodeSolheim]

This feature is now implemented (and tested) on Linux, Windows and Mac OS X. While testing, I discovered that the default file system on Mac OS X (HFS+) does not support time stamps with higher precision that 1s, so Amiga file date/time cannot be recorded exactly on this file system (without .uaem files). The system as described and implemented will detect this, and the consequence is that .uaem files are always* written on OS X even with default permissions (* always = 49/50 = 98% of the time )

To ensure long-term accuracy of the file attributes, the best way is to always write metadata to the .uaem files. Perhaps the default should be to always store .uaem files for all written/modified files (simply reading an existing file will never cause a metadata file to be written).

Alternative strategies, which can be implemented as options include:
- Write .uaem files when necessary (as described in the initial post). On HFS+ (and FAT) file systems, this means almost always. On NTFS and most Linux file systems, this means when comments/non-default attributes are used.
- Same as above, but relax the requirements for time resolution to 1s.
- Use .uaem files in even fewer cases, for example by ignoring the execute attribute (always enabled).
- Never write .uaem files. Files will always get default permissions, comments are not stored, and date/time is stored as accurately as possible on the host system.

Any thoughts?

[lem79]

About the only thing I can think of is perhaps naming the metadata file .<original filename>.uaem (note the leading period), to perhaps make it hidden by default in regular file browsers? Obviously that's a feature common to Linux, not sure about OSX or Windows..

The other way I could think of is to have a single metadata file like the old __UAEFS.DB__ or whatever it was called back in the E-UAE days, but to use your new format, perhaps just having the first part of the line being the filename, then its attributes, like:

image.jpg ----rwed 2013-02-12 21:20:52.02 hello

If you're relying on spaces to separate the data, that wont work when filenames contain spaces, so perhaps using one of the common CSV formats would be good? i.e. comma separated, quote enclosed, and escaped (e.g. \" for a literal quote), like this:

"image with space.jpg","----rwed","2013-02-12","21:20:52.02","hello"

and

"image with \"quotes\".jpg","----rwed","2013-02-12","21:20:52.02","hello"

Maybe? At least with CSV it's still human readable and editable. It could then also have a header line, and be a bit more robust/extensible. People could also load the file into Calc or other spreadsheet program, if they had a need for that.

edit: If there are changes made to the filesystem by the host, FS-UAE would need to check if the file still exists. Renaming files could also pose a problem. There would be no real way to protect against this, and users would have to be discouraged from making major changes to the host filesystem outside of FS-UAE.

[FrodeSolheim]

Hi, yes it would be possible to hide the files (start with a dot, for Linux/Unix, and also use hidden attribute for Windows). Another simple way would be to put the meta files inside a hidden .uaemeta directory.

But it is by design that the files are visible (and put alongside the corresponding files). This is to make the user aware of them. Otherwise, the user would for example move a file to another directory, unaware that the metadata got lost (because he didn't move the meta file, and didn't even know about it) -this would also be a problem with the older __UAEFS.DB__.

So I prefer to let the user know that there are other files to consider when doing file operations from the host side. Making it hidden takes away some of the point!

Another advantage of using one meta file per file (instead of a per-directory file) is that it is possible (and easy) to split and merge directories from the host side. This could not be done without special tools (or manual editing) if all metadata is assembled in one meta file.

[cybernoid]

Forcing metadata files to never be written (useful if you don't need them, and don't want to clutter the hard drive directories with metadata files).

I'm still waiting.
Even better was to make the old E-UAE mechanism work (I only have two ___UAE files, and they are the only needed).

FS-UAE is superb, but I don't agree at all with that paranoic way of registering file changes. It's not Amiga I think Workbench raises 40% with all those crazy-log-files.

I still use FS-UAE 2.0.1 because of that.



Many tks, and keep your excellent work.