ADF & HDF Creator - support for sparse files?

[nexusle]

For the ADF & HDF Creator:

I think, it should create a sparse file, so the size on HOST is related to the real stored data. Is it possible on all platforms?

[FrodeSolheim]

Originally Posted by nexusle

For the ADF & HDF Creator: I think, it should create a sparse file, so the size on HOST is related to the real stored data. Is it possible on all platforms?

It's quite easy to create sparse files, especially on Unix-like systems. Of course, on Windows one needs to use Windows-specific APIs (as usual...).

Whether it is supported or not is usually more dependent on file system than on OS. But as you are a Mac-ish person: If the following information is true: "On MacOS, the default filesystem is HFS+ which does not support sparse files. You can optionally format a volume with UFS which does support sparse files. HFS+ is the default filesystem on MacOS because it supports the archaic "resource fork" stuff which a few programs still use."(http://stackoverflow.com/questions/1...-c-on-mac-os-x), then most OS X users are out of luck regarding sparse files. FS-UAE can still try to create one, the worst thing that will happen if it is not supported is that the file won't be sparse.

[nexusle]

Huh??

Mac OS X itself can create sparse disk images (.dmg). Every virtualization software on OS X support sparse hard files too. From where you have this information?

I think, we should try it out. WinUAE should have code for sparse file creation in the sources

http://awesome.commodore.me/forum_im...e_hardfile.png

[FrodeSolheim]

Originally Posted by nexusle

Every virtualization software on OS X support sparse hard files too. From where you have this information?

Did you read the entire post? I included the link to where I found that information... -Wikipedia also agrees with the statement found there

Originally Posted by nexusle

I think, we should try it out. WinUAE should have code for sparse file creation in the sources

This is (1) not relevant for OS X as the Windows API for creating sparse files is not the same as the API for Unix-like systems, and (2) it is easy to create sparse files, so no help is needed there, your question was whether the platforms supports it..

It is possible you are confusing sparse files (which is a file-system feature where not all blocks are allocated up-front) with specific file formats which are "growable", such as growable HD file formats used by Virtualization software. HDF files are not growable. WinUAE (and most likely FS-UAE as well by inheritance, I just haven't tested) supports growable HD images in VHD format, but this is technically not a sparse file.

[nexusle]

Ok. Maybe I misunderstood something. But the result counts

Is there a ability to implement a feature to create "growable" images to save space on the host system?

[FrodeSolheim]

Originally Posted by nexusle

Is there a ability to implement a feature to create "growable" images to save space on the host system?

In this case, code from WinUAE can be used as reference for how to initialize VDH files, if I were to implement it in the ADF & HDF creator.

In the meantime, you can if you want test if VHD files work in FS-UAE. You can create them with WinUAE, I believe this is the "Dynamic HDF" option (or Microsoft Virtual PC, I guess). I recommend testing in "RDB" mode, if the file is >= 512 MB.

[FrodeSolheim]

A couple of follow-up points:
- I'm not certain I'll bother to implement support for Dynamic VHD files
- They can't be used like HDF files, for example, you can "dd" it to a CF-card for example.
- If the goal is to have a growable HD on your host system, using directory hard drives is probably better and more convenient anyway. Time is probably better spent implementing full support for storing Amiga file metadata...

[nexusle]

Have tried to add a hardfile (dynamically allocated VHD. Size on disk 4KB) created by Virtual Box. This recognized the hard file with the correct size in HDToolBox, but doesn't format the partitions.

After this I created a VHD file with WinUAE and partition it under Install3.1 with WinUAE. Now I moved this file to my Mac and mount it in FS-UAE. Now I see the partitions and tried to install Workbench3.1. It seems ok, but it doesn't copy all files to the destination.

Maybe something is defect in the source code. But generally VHD should work on Mac OS X, Linux and Windows (maybe also FreeBSD).

Please can you have a look on this if you find the time?

[FrodeSolheim]

I think a good use case should be present to justify spending time out (see my above post), when directory hard drives can be a better choice anyway.

(Btw, on OS X / HFS+ you can probably use the file system support for transparent compression to make up for no support for sparse files).

[nexusle]

Just found one reason, to better use hard files then directorys, yesterday. MUI3.8 can't be installed, if it is downloaded end extracted in a directory hard disk. I think there are special characters in file names which are wrong interpretet or translated by the host OS fs (in my case Mac OS X). On a hardfile the same archive can be installed with no problems.

[FrodeSolheim]

I don't doubt your problem with MUI3.8, but that doesn't mean the best solution is to drop hard drive directories and use HDFs instead. The solution depends on what the real problem is

Possible problems:
1) The MUI3.8 depends on specific file metadata / permissions. Solution: Finish implementation of file metadata for directory hard drives in FS-UAE.
2) There's a bug in FS-UAE's directory hard drive support. Solution: Fix the bug.
3) MUI3.8 uses some special file names and you extract the archive outside the emulated Amiga. Solution: Extract the archive from within the Amiga also when using directory hard drives.

Any error message or indication of what the problem might be?

[Toni Wilen]

MUI archive has file with name " " (single space). It must be unpacked inside emulation and afaik not all lha unarchivers handle it correctly.

[FrodeSolheim]

Thanks for the info Toni. My guess is that nexusle extracted the archive outside emulation (please verify?).

The file with a single space will be stored by FS-UAE on the host file system as %20 (because Windows filenames cannot end with " ", the file name is still encoded like this on all platforms for interoperability). I'll test/verify later with the MUI install this evening.

[nexusle]

Hmm. This is a good point. In case I unpacked the archive in Mac OS X using Stuffit Expander. I'll test it this evening unpacking in the emulation.

Thanks for the info!

As a clarification: I would not wish to make the directory hard drive feature in question! It may work well for many users. Everyone should take advantage of what he likes. I also want to force anybody to my will or force some feature where I only see at the moment a benefit. I would just like to discuss it, because I am of the opinion that it is useful. There are - in my opinion - important reasons.

1. If you use hard files and they want to transport from one system to the other, makes a dynamic hard file sense, because then only the really required amount needs to be copied and not the full size of the hard file

2. I personally (!) prefer work with hardfiles, because I think so to be closer to real Amiga, as with directory hard drives.

[TCD]

Originally Posted by nexusle

2. I personally (!) prefer work with hardfiles, because I think so to be closer to real Amiga, as with directory hard drives.

[nexusle]

I took the blue pill

[FrodeSolheim]

Originally Posted by nexusle

Hmm. This is a good point. In case I unpacked the archive in Mac OS X using Stuffit Expander. I'll test it this evening unpacking in the emulation.

That's the problem then (that won't work with the " " file). I tried to install MUI3.8 to a directory hard drive, and everything worked as expected (the " " file was extracted and stored as "%20" on the host system).

Originally Posted by nexusle

I would just like to discuss it, because I am of the opinion that it is useful. There are - in my opinion - important reasons.

Discussing stuff is (often) useful, but I need to prioritize what I work on

Originally Posted by nexusle

1. If you use hard files and they want to transport from one system to the other, makes a dynamic hard file sense, because then only the really required amount needs to be copied and not the full size of the hard file

Well, compared to HDFs perhaps, but directory hard drives also have this advantage! (btw, over time, dynamic hard drives will likely get fragmented and the space savings will diminish, this will not happen with directory hard drives). (btw2, you can compress the HDF for transfer, to get most of the space saving as lots of 0 bytes will compress very well..).

Originally Posted by nexusle

2. I personally (!) prefer work with hardfiles, because I think so to be closer to real Amiga, as with directory hard drives.

Please do But this is not a *really* good reason to work more on support for dynamic hard drive files (if needed?), as when directory hard drives support all metadata, you wouldn't really notice the difference in use.

[TCD]

I think it's more about feeling 'safe' that all files are within one file on the host system. Reading posts here on EAB people seem to prefer HDFs mainly for that reason. I only use directory based Amiga hard drives for years and never had any issues with them (speaking for WinUAE here, but I know that FS-UAE handles them the same). Exactly for the reasons of easy access of text files and not wasting space and/or running into problems with too less space, I'd recommend using a folder based hard drive unless synchronizing with a HD or CF card for a real Amiga is planned.

[FrodeSolheim]

Originally Posted by TheCyberDruid

I think it's more about feeling 'safe' that all files are within one file on the host system.

Probably true, but ironically, a corrupted HDF could destroy all Amiga files, while this is not so likely to happen with directory hard drives

Originally Posted by TheCyberDruid

...WinUAE here, but I know that FS-UAE handles them the same

A big part of it is the same, but the host interface code is completely different (to account for interoperability among multiple OSes among other reasons). I think I must quickly implement full metadata support, because right now I spend more time writing about it missing than I would just implementing it

[nexusle]

Ok. If directory usage are compatible like hdf, I'll do some compromise by formatting a SDCard as FAT32 and use the whole disk as directory hard drive! On OS X handle with real drives as hardfile is a bit tricky.