English Amiga Board


Go Back   English Amiga Board > Support > support.FS-UAE

 
 
Thread Tools
Old 09 February 2013, 10:43   #1
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
ADF & HDF Creator - support for sparse files?

For the ADF & HDF Creator:

I think, it should create a sparse file, so the size on HOST is related to the real stored data. Is it possible on all platforms?

Last edited by prowler; 29 August 2013 at 22:41. Reason: Rewind.
nexusle is offline  
Old 09 February 2013, 13:19   #2
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
Quote:
Originally Posted by nexusle View Post
For the ADF & HDF Creator: I think, it should create a sparse file, so the size on HOST is related to the real stored data. Is it possible on all platforms?
It's quite easy to create sparse files, especially on Unix-like systems. Of course, on Windows one needs to use Windows-specific APIs (as usual...).

Whether it is supported or not is usually more dependent on file system than on OS. But as you are a Mac-ish person: If the following information is true: "On MacOS, the default filesystem is HFS+ which does not support sparse files. You can optionally format a volume with UFS which does support sparse files. HFS+ is the default filesystem on MacOS because it supports the archaic "resource fork" stuff which a few programs still use."(http://stackoverflow.com/questions/1...-c-on-mac-os-x), then most OS X users are out of luck regarding sparse files. FS-UAE can still try to create one, the worst thing that will happen if it is not supported is that the file won't be sparse.
FrodeSolheim is offline  
Old 09 February 2013, 13:33   #3
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
Huh??

Mac OS X itself can create sparse disk images (.dmg). Every virtualization software on OS X support sparse hard files too. From where you have this information?

I think, we should try it out. WinUAE should have code for sparse file creation in the sources

http://awesome.commodore.me/forum_im...e_hardfile.png

Last edited by prowler; 29 August 2013 at 22:42. Reason: Rewind.
nexusle is offline  
Old 09 February 2013, 13:45   #4
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
Quote:
Originally Posted by nexusle View Post
Every virtualization software on OS X support sparse hard files too. From where you have this information?
Did you read the entire post? I included the link to where I found that information... -Wikipedia also agrees with the statement found there

Quote:
Originally Posted by nexusle View Post
I think, we should try it out. WinUAE should have code for sparse file creation in the sources
This is (1) not relevant for OS X as the Windows API for creating sparse files is not the same as the API for Unix-like systems, and (2) it is easy to create sparse files, so no help is needed there, your question was whether the platforms supports it..

It is possible you are confusing sparse files (which is a file-system feature where not all blocks are allocated up-front) with specific file formats which are "growable", such as growable HD file formats used by Virtualization software. HDF files are not growable. WinUAE (and most likely FS-UAE as well by inheritance, I just haven't tested) supports growable HD images in VHD format, but this is technically not a sparse file.

Last edited by FrodeSolheim; 09 February 2013 at 13:58.
FrodeSolheim is offline  
Old 09 February 2013, 13:49   #5
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
Ok. Maybe I misunderstood something. But the result counts

Is there a ability to implement a feature to create "growable" images to save space on the host system?

Last edited by prowler; 29 August 2013 at 22:43. Reason: Rewind.
nexusle is offline  
Old 09 February 2013, 14:06   #6
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
Quote:
Originally Posted by nexusle View Post
Is there a ability to implement a feature to create "growable" images to save space on the host system?
In this case, code from WinUAE can be used as reference for how to initialize VDH files, if I were to implement it in the ADF & HDF creator.

In the meantime, you can if you want test if VHD files work in FS-UAE. You can create them with WinUAE, I believe this is the "Dynamic HDF" option (or Microsoft Virtual PC, I guess). I recommend testing in "RDB" mode, if the file is >= 512 MB.
FrodeSolheim is offline  
Old 09 February 2013, 14:17   #7
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
A couple of follow-up points:
- I'm not certain I'll bother to implement support for Dynamic VHD files
- They can't be used like HDF files, for example, you can "dd" it to a CF-card for example.
- If the goal is to have a growable HD on your host system, using directory hard drives is probably better and more convenient anyway. Time is probably better spent implementing full support for storing Amiga file metadata...

Last edited by FrodeSolheim; 09 February 2013 at 14:35.
FrodeSolheim is offline  
Old 09 February 2013, 14:55   #8
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
Have tried to add a hardfile (dynamically allocated VHD. Size on disk 4KB) created by Virtual Box. This recognized the hard file with the correct size in HDToolBox, but doesn't format the partitions.

After this I created a VHD file with WinUAE and partition it under Install3.1 with WinUAE. Now I moved this file to my Mac and mount it in FS-UAE. Now I see the partitions and tried to install Workbench3.1. It seems ok, but it doesn't copy all files to the destination.

Maybe something is defect in the source code. But generally VHD should work on Mac OS X, Linux and Windows (maybe also FreeBSD).

Please can you have a look on this if you find the time?

Last edited by prowler; 29 August 2013 at 22:44. Reason: Rewind.
nexusle is offline  
Old 09 February 2013, 15:25   #9
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
I think a good use case should be present to justify spending time out (see my above post), when directory hard drives can be a better choice anyway.

(Btw, on OS X / HFS+ you can probably use the file system support for transparent compression to make up for no support for sparse files).
FrodeSolheim is offline  
Old 11 February 2013, 12:16   #10
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
Just found one reason, to better use hard files then directorys, yesterday. MUI3.8 can't be installed, if it is downloaded end extracted in a directory hard disk. I think there are special characters in file names which are wrong interpretet or translated by the host OS fs (in my case Mac OS X). On a hardfile the same archive can be installed with no problems.

Last edited by prowler; 29 August 2013 at 22:45. Reason: Rewind.
nexusle is offline  
Old 11 February 2013, 13:09   #11
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
I don't doubt your problem with MUI3.8, but that doesn't mean the best solution is to drop hard drive directories and use HDFs instead. The solution depends on what the real problem is

Possible problems:
1) The MUI3.8 depends on specific file metadata / permissions. Solution: Finish implementation of file metadata for directory hard drives in FS-UAE.
2) There's a bug in FS-UAE's directory hard drive support. Solution: Fix the bug.
3) MUI3.8 uses some special file names and you extract the archive outside the emulated Amiga. Solution: Extract the archive from within the Amiga also when using directory hard drives.

Any error message or indication of what the problem might be?
FrodeSolheim is offline  
Old 11 February 2013, 13:13   #12
Toni Wilen
WinUAE developer
 
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 44
Posts: 23,268
MUI archive has file with name " " (single space). It must be unpacked inside emulation and afaik not all lha unarchivers handle it correctly.
Toni Wilen is online now  
Old 11 February 2013, 13:27   #13
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
Thanks for the info Toni. My guess is that nexusle extracted the archive outside emulation (please verify?).

The file with a single space will be stored by FS-UAE on the host file system as %20 (because Windows filenames cannot end with " ", the file name is still encoded like this on all platforms for interoperability). I'll test/verify later with the MUI install this evening.

Last edited by FrodeSolheim; 11 February 2013 at 13:34.
FrodeSolheim is offline  
Old 11 February 2013, 15:03   #14
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
Hmm. This is a good point. In case I unpacked the archive in Mac OS X using Stuffit Expander. I'll test it this evening unpacking in the emulation.

Thanks for the info!

As a clarification: I would not wish to make the directory hard drive feature in question! It may work well for many users. Everyone should take advantage of what he likes. I also want to force anybody to my will or force some feature where I only see at the moment a benefit. I would just like to discuss it, because I am of the opinion that it is useful. There are - in my opinion - important reasons.

1. If you use hard files and they want to transport from one system to the other, makes a dynamic hard file sense, because then only the really required amount needs to be copied and not the full size of the hard file

2. I personally (!) prefer work with hardfiles, because I think so to be closer to real Amiga, as with directory hard drives.


Last edited by prowler; 29 August 2013 at 22:46. Reason: Rewind.
nexusle is offline  
Old 11 February 2013, 15:12   #15
TCD
Registered User

TCD's Avatar
 
Join Date: Sep 2006
Location: Germany
Age: 41
Posts: 24,012
Quote:
Originally Posted by nexusle View Post
2. I personally (!) prefer work with hardfiles, because I think so to be closer to real Amiga, as with directory hard drives.


TCD is offline  
Old 11 February 2013, 15:25   #16
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
I took the blue pill

Last edited by prowler; 29 August 2013 at 22:47. Reason: Rewind.
nexusle is offline  
Old 11 February 2013, 18:57   #17
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
Quote:
Originally Posted by nexusle View Post
Hmm. This is a good point. In case I unpacked the archive in Mac OS X using Stuffit Expander. I'll test it this evening unpacking in the emulation.
That's the problem then (that won't work with the " " file). I tried to install MUI3.8 to a directory hard drive, and everything worked as expected (the " " file was extracted and stored as "%20" on the host system).

Quote:
Originally Posted by nexusle View Post
I would just like to discuss it, because I am of the opinion that it is useful. There are - in my opinion - important reasons.
Discussing stuff is (often) useful, but I need to prioritize what I work on

Quote:
Originally Posted by nexusle View Post
1. If you use hard files and they want to transport from one system to the other, makes a dynamic hard file sense, because then only the really required amount needs to be copied and not the full size of the hard file
Well, compared to HDFs perhaps, but directory hard drives also have this advantage! (btw, over time, dynamic hard drives will likely get fragmented and the space savings will diminish, this will not happen with directory hard drives). (btw2, you can compress the HDF for transfer, to get most of the space saving as lots of 0 bytes will compress very well..).

Quote:
Originally Posted by nexusle View Post
2. I personally (!) prefer work with hardfiles, because I think so to be closer to real Amiga, as with directory hard drives.
Please do But this is not a *really* good reason to work more on support for dynamic hard drive files (if needed?), as when directory hard drives support all metadata, you wouldn't really notice the difference in use.

Last edited by FrodeSolheim; 11 February 2013 at 19:13.
FrodeSolheim is offline  
Old 11 February 2013, 19:04   #18
TCD
Registered User

TCD's Avatar
 
Join Date: Sep 2006
Location: Germany
Age: 41
Posts: 24,012
I think it's more about feeling 'safe' that all files are within one file on the host system. Reading posts here on EAB people seem to prefer HDFs mainly for that reason. I only use directory based Amiga hard drives for years and never had any issues with them (speaking for WinUAE here, but I know that FS-UAE handles them the same). Exactly for the reasons of easy access of text files and not wasting space and/or running into problems with too less space, I'd recommend using a folder based hard drive unless synchronizing with a HD or CF card for a real Amiga is planned.
TCD is offline  
Old 11 February 2013, 19:13   #19
FrodeSolheim
FS-UAE Developer

FrodeSolheim's Avatar
 
Join Date: Dec 2011
Location: Førde, Norway
Age: 38
Posts: 3,561
Quote:
Originally Posted by TheCyberDruid View Post
I think it's more about feeling 'safe' that all files are within one file on the host system.
Probably true, but ironically, a corrupted HDF could destroy all Amiga files, while this is not so likely to happen with directory hard drives

Quote:
Originally Posted by TheCyberDruid View Post
...WinUAE here, but I know that FS-UAE handles them the same
A big part of it is the same, but the host interface code is completely different (to account for interoperability among multiple OSes among other reasons). I think I must quickly implement full metadata support, because right now I spend more time writing about it missing than I would just implementing it
FrodeSolheim is offline  
Old 11 February 2013, 19:42   #20
nexusle
Banned
 
Join Date: Jul 2010
Location: Earth
Age: 41
Posts: 500
Ok. If directory usage are compatible like hdf, I'll do some compromise by formatting a SDCard as FAT32 and use the whole disk as directory hard drive! On OS X handle with real drives as hardfile is a bit tricky.

Last edited by prowler; 29 August 2013 at 22:48. Reason: Rewind.
nexusle is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
hdf creator for whdload packs rare_j Amiga scene 68 02 April 2017 21:39
Windows Files to ADF or HDF Technix59 support.WinUAE 8 05 May 2011 21:04
Custom HDF Creator freddix Amiga scene 14 09 May 2009 23:59
Copy files ADF to HDF Heula New to Emulation or Amiga scene 7 11 January 2008 14:48
which adf creator skidrow New to Emulation or Amiga scene 5 24 April 2006 23:33

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 20:35.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Page generated in 0.09195 seconds with 13 queries