English Amiga Board


Go Back   English Amiga Board > Support > support.FS-UAE

 
 
Thread Tools
Old 26 November 2013, 10:25   #1
edd_jedi
Registered User
edd_jedi's Avatar
 
Join Date: Apr 2010
Location: London / UK
Posts: 396
.uaem and info.uaem files?

I've just noticed that while backing up my real Amiga files to my computer via FS-UAE, it has created the above 2 files for every single file! So for example I now have:

AlienBreed3D (folder)
AlienBreed3D.info (icon)
AlienBreed3D.info.uaem
AlienBreed3D.uaem

so this has obviously doubled the amount of files, which is a considerable amount on a 4GB hard disk!

What are these files, and how do I get rid of them? When I have to copy these back to my real Amiga HD, I don't want to have to exclude all of these files manually, it will take forever.
edd_jedi is offline  
Old 26 November 2013, 11:02   #2
thevoice
Registered User
 
Join Date: Oct 2006
Location: germany
Posts: 104
FS-UAE uses these files to store file attributes and such iirc.
if you play whdloadgames in FS-UAE you might need these files on certain titles. if you play on a real amiga these files are not needed at all.

also each .ueam file is just a few bytes usually whopping 33 bytes.
so even with the entire whdload collection (games/demos/mags) the "wasted" space shouldn't be a big problem at all even on a 4gb hd .

also getting rid of them on the amiga shouldnt be a huge problem. just look for some kind of a script/program which is able to mass delete files with pre defined filtes e.g. *.uaem or something.

Last edited by thevoice; 26 November 2013 at 11:05. Reason: i can almost type i guess
thevoice is offline  
Old 26 November 2013, 14:31   #3
mr_a500
Amiga-based Cyborg
mr_a500's Avatar
 
Join Date: Dec 2004
Location: Canada
Posts: 804
I thought when these were implemented that there would be an option to turn them off.
mr_a500 is offline  
Old 26 November 2013, 20:27   #4
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
What thevoice said, and you probably want to keep them (especially if you want the directory hard drive to function as a backup) in order to keep all metadata intact.

When you copy content back to an Amiga drive (from within FS-UAE), the .uaem files will not be copied (they'll only be read in order to figure out the correct permissions/dates). So you won't have to exclude then manually

@mr_a500 Yes, that was and is still a possibiliy, I haven't implemented that yet (one might want to be able to turn it on/off per volume), and there hasn't been any demand for it... (= no-one has bugged me about it )
FrodeSolheim is offline  
Old 28 November 2013, 14:54   #5
amigafreak68k
Registered User
 
Join Date: Apr 2013
Location: Engelsdorf / Germany
Posts: 403
Maybe the files could be hidden on host OS? In Linux/Mac OS X and - maybe - BSD, a simple "." in front of the file name will hide the file. In Windows there could files be hidden by set it in files properties - maybe this could be a file flag in Windows.
amigafreak68k is offline  
Old 28 November 2013, 18:45   #6
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
@amigafreak68k Yes, I have considered (and rejected) that, see http://eab.abime.net/showpost.php?p=909394&postcount=32.
FrodeSolheim is offline  
Old 01 January 2014, 23:33   #7
mark_k
Registered User
 
Join Date: Aug 2004
Location:
Posts: 2,966
Quote:
Originally Posted by thevoice View Post
also each .ueam file is just a few bytes usually whopping 33 bytes.
so even with the entire whdload collection (games/demos/mags) the "wasted" space shouldn't be a big problem at all even on a 4gb hd .
It depends on the filesystem, but if you have your files on a FAT32 partition (on an SD card or USB flash drive say), even the smallest file typically occupies 32KB.

I was wondering, is there any reason why FS-UAE doesn't use the same _UAEFSDB.___ method as WinUAE? That just adds one file per directory, and would be great for interoperability too, e.g. if the user has shared folders he created using WinUAE.
mark_k is offline  
Old 02 January 2014, 06:21   #8
Der Wanderer
Registered User

Der Wanderer's Avatar
 
Join Date: Dec 2013
Location: Mountain View / USA
Posts: 33
Another idea is instead of brute force creating those files, create them only if an attribute is different from the default (most common) case. Should keep FS UAE from creating these files in 99.9% of the cases, and guess those file are a performance loss.
And ehm, I bother you now to implement a switch to turn this off. ;-) Because most programs don't require it.
Der Wanderer is offline  
Old 04 January 2014, 18:26   #9
jbl007
Registered User
 
Join Date: Mar 2013
Location: Leipzig/Germany
Posts: 371
I use this to delete the unneeded .uaem-files in Linux:
Code:
find /path/to/directory/ -name "*.uaem" -print0 | xargs -0 grep -lZe "----rwed" | xargs -0 rm
Quote:
Originally Posted by Der Wanderer View Post
create them only if an attribute is different from the default (most common) case.
Perhaps someone who "speaks" C can give a little help here and change 'need_metadata_file=1' into something like 'if metadata = "----rwed" -> need_metadata_file=0' in fsdb_host.cpp. Or isn't it that easy?

Edit: In fact there seem to be some people here with some good knowledge. So guys, what about some practical solutions instead of only requesting features?

Last edited by jbl007; 04 January 2014 at 18:59.
jbl007 is offline  
Old 04 January 2014, 19:16   #10
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
Quote:
Originally Posted by mark_k View Post
I was wondering, is there any reason why FS-UAE doesn't use the same _UAEFSDB.___ method as WinUAE? That just adds one file per directory, and would be great for interoperability too, e.g. if the user has shared folders he created using WinUAE.
Very short version: When I implemented file system support, there was a problem with UAEFSDB not working in FS-UAE, and I decided to not try to fix it for several reasons:
- I personally liked the .uaem approach better, more info here http://eab.abime.net/showpost.php?p=909698&postcount=34 and here http://eab.abime.net/showthread.php?t=67882.
- I think there were some concerns with the portability of the UAEFSDB files / file name escaping across multiple OSse because the UAE versions for different platforms didn't agree on what file names had to be rewritten, etc.

I don't even remember all the details now, but I concluded that writing a new meta system would be both safer and nicer (the "nicer" argument being somewhat subjective of course )

Quote:
Originally Posted by Der Wanderer View Post
Another idea is instead of brute force creating those files, create them only if an attribute is different from the default (most common) case. Should keep FS UAE from creating these files in 99.9% of the cases, and guess those file are a performance loss.
And ehm, I bother you now to implement a switch to turn this off. ;-) Because most programs don't require it.
Yes, I always planned this. I actually wanted to have this is per-volume configuration files (.e.g a Amiga-invisible ini file in the root of the volume) so the metadata options follows the hard drive independent of what FS-UAE options are used. But a simple global option for setting default metadata strategy would be relatively quickly implemented, so I can consider doing that first.

Quote:
Originally Posted by jbl007 View Post
I use this to delete the unneeded .uaem-files in Linux:
Code:
find /path/to/directory/ -name "*.uaem" -print0 | xargs -0 grep -lZe "----rwed" | xargs -0 rm
Perhaps someone who "speaks" C can give a little help here and change 'need_metadata_file=1' into something like 'if metadata = "----rwed" -> need_metadata_file=0' in fsdb_host.cpp. Or isn't it that easy?
Another important case to consider is when files have associated filenotes (comments). It seems some unpackers require information in filenotes to unpack files (see e.g. Embryo WHDLoad slave). It is easy to spot these files, as .uaem files will be larger in this case (without comments they have a fixed minimum size).

(Regarding the C code, it is *almost* that easy if you want a basic non-configurable on-demand uaem feature )
FrodeSolheim is offline  
Old 04 January 2014, 21:12   #11
jbl007
Registered User
 
Join Date: Mar 2013
Location: Leipzig/Germany
Posts: 371
I don't know what I'm doing wrong, but fs-uae always creates .uaem files. For example saving a litte text file in MEmacs:

Code:
$ LANG=C stat foo
  File: 'foo'
  Size: 18            Blocks: 8          IO Block: 4096   regular file
Device: 25h/37d    Inode: 8052        Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/     stw)   Gid: (  100/   users)
Access: 2014-01-04 20:05:10.540000000 +0100
Modify: 2014-01-04 20:05:10.540000000 +0100
Change: 2014-01-04 20:05:10.542887952 +0100
 Birth: -

$ cat foo.uaem 
----rwed 2014-01-04 20:05:10.54
  • It's a new file -> no .uaem existed before
  • It has default permissions
  • mtime is precise enough
  • no filenotes
Why is it needed here?
jbl007 is offline  
Old 04 January 2014, 22:20   #12
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
.uaem files create because of the following line(s) in fsdb_host.cpp:
// always write metadata file, for now...
need_metadata_file = 1;
(since I didn't finish implement a configurable option, it uses the most conservative approach = always write files).

Note: removing this line does not make it work as desired, you'd also probably want it to be set to 1 if a comment exists and if permissions are other than ----rwed (and ----rw-d).
FrodeSolheim is offline  
Old 04 March 2014, 14:54   #13
mr_a500
Amiga-based Cyborg
mr_a500's Avatar
 
Join Date: Dec 2004
Location: Canada
Posts: 804
Can you please add that option to disable creation of .uaem files? I just searched my hard drive and found and deleted 46,857 .uaem files!
mr_a500 is offline  
Old 16 March 2014, 15:02   #14
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
I'll an add option for that in one of the next development versions (Not that I really think it is *that* problematic with 46,857 .uaem files, since it also means you have at least created that many files and folders from within the emulated Amiga, and an occasional find /dir -name *.uaem -delete can alraedy clear it out )

Last edited by FrodeSolheim; 16 March 2014 at 15:08.
FrodeSolheim is offline  
Old 16 March 2014, 18:52   #15
mark_k
Registered User
 
Join Date: Aug 2004
Location:
Posts: 2,966
Another down side of having many un-needed .uaem files is fragmentation. Even if you delete them (which will also delete the needed ones for files which have comments), the freed space on the drive will then be in thousands of small fragments.
mark_k is offline  
Old 07 August 2014, 21:19   #16
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
2.5.8dev is being prepared now, which supports a new option: uaem_write_flags (controls when .uaem metadata files are created):

0 - disable (never create .uaem files)
1 - default (always create .uaem files)

n - when file notes are added
t - when file time cannot be stored exactly on the host system

h - when the hold bit is set
s - when the script bit is set
p - when the pure bit is set
a - when the archive bit is set

r - when the read bit is NOT set
w - when the write bit is NOT set
e - when the execute bit is NOT set
d - when the delete bit is NOT set

To only create .uaem metadata files when files notes are added:
Code:
uaem_write_flags = n
To only create .uaem files when files notes are added or non-default protection bits are set:
Code:
uaem_write_flags = nhsparwed
Since file notes are required for the operating of some software, it is recommended to always include the n flag if you use this option.

Regardless of the use of this option, when specific .uaem files already exists, they will always be updated.

Last edited by FrodeSolheim; 08 August 2014 at 03:23. Reason: uaem_write_flags was spelled wrong in the examples
FrodeSolheim is offline  
Old 16 August 2014, 22:35   #17
jbl007
Registered User
 
Join Date: Mar 2013
Location: Leipzig/Germany
Posts: 371
Works great, thanks for implementing this.
I'm using nhsparwed as a default setting now, this makes life a bit easier

Quote:
Originally Posted by FrodeSolheim View Post
Since file notes are required for the operating of some software, it is recommended to always include the n flag if you use this option.
BTW: Too bad that there is no lha for pc which can take care about file notes. Python-lhafile doesn't support it, does it!?
It would be nice to extract archives at the host side and auto-generate .uaem-files if needed.
jbl007 is offline  
Old 16 August 2014, 22:42   #18
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
Quote:
Originally Posted by jbl007 View Post
BTW: Too bad that there is no lha for pc which can take care about file notes. Python-lhafile doesn't support it, does it!?
It would be nice to extract archives at the host side and auto-generate .uaem-files if needed.
Actually, it (python-lhafile) does - I added the functionality so I could use it to create OAGD.net WHDLoad variants from .lha archives (e.g. Embryo needs file notes to work).

I have already thought about created an little unarchiver tool and bundle it with Fs-UAE Launcher...

(the tool would also be useful to extract .zip archives with ISO-8859-1 file encoding - as would be used on Amiga. Many common zip programs do not support this mode of operation...).
FrodeSolheim is offline  
Old 16 August 2014, 23:00   #19
jbl007
Registered User
 
Join Date: Mar 2013
Location: Leipzig/Germany
Posts: 371
Quote:
Originally Posted by FrodeSolheim View Post
Actually, it (python-lhafile) does - I added the functionality
Hm, I tried some time ago without success. The .filenote() (or .comment()? - don't remember) methods always returned empty strings. And yes, there were files with file notes in the archive. I think, I'll give it another try...
jbl007 is offline  
Old 16 August 2014, 23:06   #20
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 37
Posts: 3,427
Here is partial code from the archive scanner used for WHDLoad imports (it supports both .lha files and .zip files with embedded .uaem files):
Code:
    for info in archive.infolist():
        name = info.filename
        if name.endswith(".uaem"):
            continue
        comment = info.comment
        try:
            metadata = archive.read(name + ".uaem")
        except KeyError:
            pass
        else:
            metadata = metadata.split("\n")[0].strip()
            comment = metadata[32:]
            comment = urllib.unquote(comment)
(archive being a zipfile.ZipFile or lhafile.LhaFile object)
FrodeSolheim is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
how to create .info files jimbobrocks92 support.Other 2 31 December 2011 09:03
How to Create .info Files? AmigaFan support.Other 6 25 January 2009 21:18
info on creating info files icons ? Dimlow New to Emulation or Amiga scene 10 16 July 2008 20:28
.info files Aljrob support.WinUAE 5 14 March 2007 16:42
New here with a question about .info files jobro Amiga scene 14 24 November 2005 23:48

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 22:03.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.
Page generated in 0.09428 seconds with 15 queries