.uaem and info.uaem files?

[edd_jedi]

I've just noticed that while backing up my real Amiga files to my computer via FS-UAE, it has created the above 2 files for every single file! So for example I now have:

AlienBreed3D (folder)
AlienBreed3D.info (icon)
AlienBreed3D.info.uaem
AlienBreed3D.uaem

so this has obviously doubled the amount of files, which is a considerable amount on a 4GB hard disk!

What are these files, and how do I get rid of them? When I have to copy these back to my real Amiga HD, I don't want to have to exclude all of these files manually, it will take forever.

[thevoice]

FS-UAE uses these files to store file attributes and such iirc.
if you play whdloadgames in FS-UAE you might need these files on certain titles. if you play on a real amiga these files are not needed at all.

also each .ueam file is just a few bytes usually whopping 33 bytes.
so even with the entire whdload collection (games/demos/mags) the "wasted" space shouldn't be a big problem at all even on a 4gb hd .

also getting rid of them on the amiga shouldnt be a huge problem. just look for some kind of a script/program which is able to mass delete files with pre defined filtes e.g. *.uaem or something.

[mr_a500]

I thought when these were implemented that there would be an option to turn them off.

[FrodeSolheim]

What thevoice said, and you probably want to keep them (especially if you want the directory hard drive to function as a backup) in order to keep all metadata intact.

When you copy content back to an Amiga drive (from within FS-UAE), the .uaem files will not be copied (they'll only be read in order to figure out the correct permissions/dates). So you won't have to exclude then manually

@mr_a500 Yes, that was and is still a possibiliy, I haven't implemented that yet (one might want to be able to turn it on/off per volume), and there hasn't been any demand for it... (= no-one has bugged me about it )

[amigafreak68k]

Maybe the files could be hidden on host OS? In Linux/Mac OS X and - maybe - BSD, a simple "." in front of the file name will hide the file. In Windows there could files be hidden by set it in files properties - maybe this could be a file flag in Windows.

[FrodeSolheim]

@amigafreak68k Yes, I have considered (and rejected) that, see http://eab.abime.net/showpost.php?p=909394&postcount=32.

[mark_k]

Originally Posted by thevoice

also each .ueam file is just a few bytes usually whopping 33 bytes.
so even with the entire whdload collection (games/demos/mags) the "wasted" space shouldn't be a big problem at all even on a 4gb hd .

It depends on the filesystem, but if you have your files on a FAT32 partition (on an SD card or USB flash drive say), even the smallest file typically occupies 32KB.

I was wondering, is there any reason why FS-UAE doesn't use the same _UAEFSDB.___ method as WinUAE? That just adds one file per directory, and would be great for interoperability too, e.g. if the user has shared folders he created using WinUAE.

[Der Wanderer]

Another idea is instead of brute force creating those files, create them only if an attribute is different from the default (most common) case. Should keep FS UAE from creating these files in 99.9% of the cases, and guess those file are a performance loss.
And ehm, I bother you now to implement a switch to turn this off. ;-) Because most programs don't require it.

[jbl007]

I use this to delete the unneeded .uaem-files in Linux:

find /path/to/directory/ -name "*.uaem" -print0 | xargs -0 grep -lZe "----rwed" | xargs -0 rm

Originally Posted by Der Wanderer

create them only if an attribute is different from the default (most common) case.

Perhaps someone who "speaks" C can give a little help here and change 'need_metadata_file=1' into something like 'if metadata = "----rwed" -> need_metadata_file=0' in fsdb_host.cpp. Or isn't it that easy?

Edit: In fact there seem to be some people here with some good knowledge. So guys, what about some practical solutions instead of only requesting features?

[FrodeSolheim]

Originally Posted by mark_k

I was wondering, is there any reason why FS-UAE doesn't use the same _UAEFSDB.___ method as WinUAE? That just adds one file per directory, and would be great for interoperability too, e.g. if the user has shared folders he created using WinUAE.

Very short version: When I implemented file system support, there was a problem with UAEFSDB not working in FS-UAE, and I decided to not try to fix it for several reasons:
- I personally liked the .uaem approach better, more info here http://eab.abime.net/showpost.php?p=909698&postcount=34 and here http://eab.abime.net/showthread.php?t=67882.
- I think there were some concerns with the portability of the UAEFSDB files / file name escaping across multiple OSse because the UAE versions for different platforms didn't agree on what file names had to be rewritten, etc.

I don't even remember all the details now, but I concluded that writing a new meta system would be both safer and nicer (the "nicer" argument being somewhat subjective of course )

Originally Posted by Der Wanderer

Another idea is instead of brute force creating those files, create them only if an attribute is different from the default (most common) case. Should keep FS UAE from creating these files in 99.9% of the cases, and guess those file are a performance loss.
And ehm, I bother you now to implement a switch to turn this off. ;-) Because most programs don't require it.

Yes, I always planned this. I actually wanted to have this is per-volume configuration files (.e.g a Amiga-invisible ini file in the root of the volume) so the metadata options follows the hard drive independent of what FS-UAE options are used. But a simple global option for setting default metadata strategy would be relatively quickly implemented, so I can consider doing that first.

Originally Posted by jbl007

I use this to delete the unneeded .uaem-files in Linux:

find /path/to/directory/ -name "*.uaem" -print0 | xargs -0 grep -lZe "----rwed" | xargs -0 rm

Perhaps someone who "speaks" C can give a little help here and change 'need_metadata_file=1' into something like 'if metadata = "----rwed" -> need_metadata_file=0' in fsdb_host.cpp. Or isn't it that easy?

Another important case to consider is when files have associated filenotes (comments). It seems some unpackers require information in filenotes to unpack files (see e.g. Embryo WHDLoad slave). It is easy to spot these files, as .uaem files will be larger in this case (without comments they have a fixed minimum size).

(Regarding the C code, it is *almost* that easy if you want a basic non-configurable on-demand uaem feature )

[jbl007]

I don't know what I'm doing wrong, but fs-uae always creates .uaem files. For example saving a litte text file in MEmacs:

$ LANG=C stat foo
  File: 'foo'
  Size: 18            Blocks: 8          IO Block: 4096   regular file
Device: 25h/37d    Inode: 8052        Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/     stw)   Gid: (  100/   users)
Access: 2014-01-04 20:05:10.540000000 +0100
Modify: 2014-01-04 20:05:10.540000000 +0100
Change: 2014-01-04 20:05:10.542887952 +0100
 Birth: -

$ cat foo.uaem 
----rwed 2014-01-04 20:05:10.54

• It's a new file -> no .uaem existed before
• It has default permissions
• mtime is precise enough
• no filenotes

Why is it needed here?

[FrodeSolheim]

.uaem files create because of the following line(s) in fsdb_host.cpp:
// always write metadata file, for now...
need_metadata_file = 1;
(since I didn't finish implement a configurable option, it uses the most conservative approach = always write files).

Note: removing this line does not make it work as desired, you'd also probably want it to be set to 1 if a comment exists and if permissions are other than ----rwed (and ----rw-d).

[mr_a500]

Can you please add that option to disable creation of .uaem files? I just searched my hard drive and found and deleted 46,857 .uaem files!

[FrodeSolheim]

I'll an add option for that in one of the next development versions (Not that I really think it is *that* problematic with 46,857 .uaem files, since it also means you have at least created that many files and folders from within the emulated Amiga, and an occasional find /dir -name *.uaem -delete can alraedy clear it out )

[mark_k]

Another down side of having many un-needed .uaem files is fragmentation. Even if you delete them (which will also delete the needed ones for files which have comments), the freed space on the drive will then be in thousands of small fragments.

[FrodeSolheim]

2.5.8dev is being prepared now, which supports a new option: uaem_write_flags (controls when .uaem metadata files are created):

0 - disable (never create .uaem files)
1 - default (always create .uaem files)

n - when file notes are added
t - when file time cannot be stored exactly on the host system

h - when the hold bit is set
s - when the script bit is set
p - when the pure bit is set
a - when the archive bit is set

r - when the read bit is NOT set
w - when the write bit is NOT set
e - when the execute bit is NOT set
d - when the delete bit is NOT set

To only create .uaem metadata files when files notes are added:

uaem_write_flags = n

To only create .uaem files when files notes are added or non-default protection bits are set:

uaem_write_flags = nhsparwed

Since file notes are required for the operating of some software, it is recommended to always include the n flag if you use this option.

Regardless of the use of this option, when specific .uaem files already exists, they will always be updated.

[jbl007]

Works great, thanks for implementing this.
I'm using nhsparwed as a default setting now, this makes life a bit easier

Originally Posted by FrodeSolheim

Since file notes are required for the operating of some software, it is recommended to always include the n flag if you use this option.

BTW: Too bad that there is no lha for pc which can take care about file notes. Python-lhafile doesn't support it, does it!?
It would be nice to extract archives at the host side and auto-generate .uaem-files if needed.

[FrodeSolheim]

Originally Posted by jbl007

BTW: Too bad that there is no lha for pc which can take care about file notes. Python-lhafile doesn't support it, does it!?
It would be nice to extract archives at the host side and auto-generate .uaem-files if needed.

Actually, it (python-lhafile) does - I added the functionality so I could use it to create OAGD.net WHDLoad variants from .lha archives (e.g. Embryo needs file notes to work).

I have already thought about created an little unarchiver tool and bundle it with Fs-UAE Launcher...

(the tool would also be useful to extract .zip archives with ISO-8859-1 file encoding - as would be used on Amiga. Many common zip programs do not support this mode of operation...).

[jbl007]

Originally Posted by FrodeSolheim

Actually, it (python-lhafile) does - I added the functionality

Hm, I tried some time ago without success. The .filenote() (or .comment()? - don't remember) methods always returned empty strings. And yes, there were files with file notes in the archive. I think, I'll give it another try...

[FrodeSolheim]

Here is partial code from the archive scanner used for WHDLoad imports (it supports both .lha files and .zip files with embedded .uaem files):

    for info in archive.infolist():
        name = info.filename
        if name.endswith(".uaem"):
            continue
        comment = info.comment
        try:
            metadata = archive.read(name + ".uaem")
        except KeyError:
            pass
        else:
            metadata = metadata.split("\n")[0].strip()
            comment = metadata[32:]
            comment = urllib.unquote(comment)

(archive being a zipfile.ZipFile or lhafile.LhaFile object)