File timestamps lose precision in shared folders

[mark_k]

It seems WinUAE keeps less-precise timestamps for files which are in shared folders. I noticed this with WinUAE 2.5.0 on native Windows Vista (shared folder on NTFS boot drive) and Wine (shared folder on ext4 partition).

Amiga native filesystem timestamp precision is to the tick (1/50th second). The NTFS native timestamp is apparently precise to 100ns, so NTFS should support accurate-enough timestamps.

To hopefully demonstrate this issue we'll use Cloanto's DirDiff program. That can compare files in two directories, indicating which are different. It compares file comment, protection bits and datestamp as well as the actual file content.

Try this:

• Configure WinUAE to have at least one shared folder, I'll assume you called it SHARE:
• Download DirDiff from http://aminet.net/biz/cloan/DirDiff.lha. Unpack it to the shared folder.
• Boot WinUAE from an emulated hard disk or floppy disk. (You need at least Kickstart 2.0 to run DirDiff.)
• With a blank/formatted floppy disk in DF1:, open a CLI and type

  Echo "test1" >DF1:test1.txt
  Echo "test2" >DF1:test2.txt
  MakeDir RAM:test
  MakeDir SHARE:sharetest
  Copy DF1:#? RAM:test CLONE
  Copy DF1:#? SHARE:sharetest CLONE
  CD SHARE:DirDiff
  DirDiff 2 -b DF1: RAM:test
  DirDiff 2 -b DF1: SHARE:sharetest

The first DirDiff run, comparing with RAM:test, should show no differences. The second run should show one or both the files in SHARE:sharetest having different datestamps from the DF1: copies.

If you're too lazy let me know and I'll try to make a floppy image that does all that from the startup-sequence.

[Toni Wilen]

Filesystem handler internally uses seconds for some reason.. (losing fractions of seconds part of ticks)
Probably some historic reason.

Will be fixed.

[mark_k]

Maybe the shared folder handler was first introduced with WinUAE (as opposed to the original UNIX UAE), when Windows 95/98 was current and hence FAT was the most common partition format?

FAT file creation times are accurate to 10ms (since DOS 7.0, else 2 seconds). But file modification times are only accurate to 2 seconds. Perhaps the original developer didn't see a need to maintain greater accuracy in the AmigaDOS handler.

On that subject... are Amiga file datestamps stored along with comments in _UAEFSDB.___? That would stop host filesystem datestamp inaccuracy affecting the emulated Amiga.

[Toni Wilen]

It is Unix based and it seems Unix file date resolution is (was?) 1 second.

EDIT: I replaced time_t with timeval: resolution is now microseconds.

_UAEFSDB.___ can't be changed without breaking compatibility with old versions = not going to happen.

[Toni Wilen]

Hmm..

There are some unused fields in UAEFSDB (for example status byte that only contains zero or non-zero), it should be possible to fit small time offset (0-99 ticks = max 2 seconds) somewhere without breaking backwards compatibility.

[mark_k]

That would be one possibility for retro-fitting more accurate file times. But there are others.

I looked at a _UAEFSDB.___ file in a hex editor. It seems 2 × 256 bytes are allotted for each filename. (Each name is stored twice, first the Amiga native name then the corresponding host name. The host name may be longer than the Amiga one if the Amiga name contains characters such as \ which are "illegal" in host filenames.)

What is the maximum Amiga filename length? struct FileInfoBlock would seem to limit it to 107 characters. So that's about 148 free bytes after every Amiga filename entry. You could easily fit a struct DateStamp in there. [Storing a full date would have the advantage of preserving the Amiga datestamps if the user e.g. copies the contents of the shared folder to another directory on the host using a program which doesn't preserve file dates.]

Assuming that's correct, it should be quite simple to make backward-compatible additions to the _UAEFSDB.___ file. Older WinUAE versions might overwrite any data in the extra region with zeros on exit though?

If you did decide to use that extra space, maybe store the used length in the last byte, and perhaps use the second-last byte to indicate the uaefsdb file version. Then future versions of WinUAE would know to preserve that data if the version is later than they understand.

[Toni Wilen]

Yeah, there is more free space but saving complete date twice (FSDB and real file metadata) will cause confusion.

Modify data outside of emulation (or just the date) and it is guaranteed weird date difference will be considered a bug.

This is the reason why I prefer tick offset.

[mark_k]

This is quite a tricky issue to get right. There probably isn't one right thing to do. But if you were to store an Amiga DateStamp in the UAEFSDB entry you could derive the tick offset from that.

The trouble with only storing a tick offset is, what happens if the user copies files elsewhere? Say they move their shared directory from an NTFS volume to a FAT32 one, e.g. a USB stick or SD card. Assume the program they used to do that preserves host file times (as closely as is possible for FAT32).

When previously accessed on the NTFS volume, all tick offset fields will be zero because NTFS datestamps are accurate enough. On copying to the FAT32 volume all host file times will be rounded to the nearest two seconds (or maybe to the previous two-second point, I don't know). And so will file times from the emulated Amiga's point of view, since all tick offsets are zero.

Whereas, if Amiga DateStamps were stored for every file instead of just a tick offset, you could restore/preserve the original file times. WinUAE could compare the host file date with the DateStamp from UAEFSDB. If the Amiga file time is within 2 seconds of the host time then assume the Amiga time is correct. If the host file time differs by more than that just tell Amiga programs that the Amiga file time is the same as the host file time.

Maybe have an option like "prefer UAEFSDB datestamps". That would allow a user who has accidentally moved/copied a shared folder without preserving the host file times to preserve the file times from the emulated Amiga's point of view.


The down size of this change is that every file written by the emulated Amiga would get its own entry in UAEFSDB. Whereas currently (I think) only files with host-illegal names or comments appear in UAEFSDB. So that would cause the UAEFSDB file to become larger than before.

I guess that could be avoided by maintaining a second file that only contains names and DateStamps for those files which don't have either a host-illegal name or a comment (i.e all files except those already in UAEFSDB). But that's more complexity...

[mark_k]

Only 2-second file date precision is supported with WinUAE 2.5.1 for files on FAT partitions. That's a limitation of the FAT last write time. However FAT file creation time supports precision to 10ms which should be enough for tick-accurate datestamps.

Can I suggest a change to support full-precision datestamps for files on FAT partitions:
- When a file is created, no problem since the creation time will be set appropriately.
- When an existing file is written to or the user manually sets its date using the SetDate command, use the SetFileTime function to set its creation time. (Maybe only do that when the file is closed for performance reasons?)
- When returning the Amiga datestamp of an existing file, check whether the creation time and last write time match (as closely as is possible). If they don't, the file may have been modified outside of emulation. So perhaps in that case return the last write time for the Amiga datestamp.

[Toni Wilen]

Good idea, will be done.

EDIT: and done now. Maybe.