Compacting vhd hardfiles

[hrmes]

I recently prepared a 8 GB SFS vhd partition as an application partition for use with a OS 3.9 installation. It contains 450 MB of data but the vhd hardfile has grown to 563 MB in size.

Are there any plans to implement vhd hardfile shrinking in WinUAE ?

I know VirtualPC and VMWare Workstation can do this for vhd and vmdk hardfiles by first marking the unused areas with a tool that runs on the guest OS and then doing the actual shrinking of the hardfile.

[thomas]

Create a new vhd, copy all files from the old to the new one and delete the old vhd.

Why do 100 MB matter when the vhd can take 8 GB? If the file was 4 GB and contained only 500 MB, you could complain, but about 100 MB?

[hrmes]

The point is that you shouldn't have to go through this chore to save some disk space, it should be done by WinUAE.
100 MB doesn't seem much but it is about 25% more space than is needed.
Let's say that you copy another 500 MB of data to the partition and then decide that you don't need it and erase it. The partition will remain close to 1 GB and only half of that would be actual data.

[Toni Wilen]

Not possible.

Emulator does not know which blocks are used, only (possible custom) filesystem driver running in emulation knows it. Problem is that there are multiple filesystems, some which have undocumented structure...

PC virtualization software only needs to know FAT and NTFS and both are quite well documented.

[prowler]

Even if it were possible, it would only pay dividends if the vhd were first defragmented, increasing the risk of data corruption.

[Maccara]

I didn't even think there would be issues with unknown structure. (null bytes are still null bytes after compacting, so i thought it should just work)

Just compacted a defragmented vhd (RDB, FFS) with VirtualPC and it worked fine.

VirtualPC "compactor" just frees the null bytes.

Never had any issues, but probably shouldn't do that then.

[Toni Wilen]

Compacting works and is safe but it can't detect which non-zero blocks are unused..

Zeroed blocks gets becomes non-zero when data gets written and they never get back to zeroed state when you delete files.

You need some filesystem specific cleanup utility/defragmenter that zeroes unused space to make compacting more efficient.

Reorg can be (mis)used to clear unused space on OFS/FFS partitions (as long as partition is completely under first 4G).

[hrmes]

I did some research with SFS and managed to locate the block allocation map/bitmap of that filesystem in my OS3.9 hardfile, using information from this page: http://www.xs4all.nl/~hjohn/SFS/bitmap.htm

Using the Diskmontools editor I saw that the bitmap starts at block 34 and goes on for about 2000 512-byte blocks in my case. The last bitmap block that contained a used data block was 165.
After that all the other bitmap blocks did not contain any data block allocations.

Then I wrote a 86 MB file to that hardfile and saw that the last used data block was allocated at bitmap block 207.
Then I erased that file and also from SFS's trashcan dir, so that all data blocks used by that file to be freed.

I saw with Diskmontools that all data blocks used by that 86 MB file were freed except for one at bitmap block 207, but all other allocated data blocks ended again at block 165 like before.

I was expecting SFS to do some sort of defragmenting on its own as it is supposed to do, so that lone allocated block would be freed.
That finally happened after I erased all files in the SFS trashcan dir.

I checked again with Diskmontools and the last allocated data block was again at bitmap block 165, and all subsequent bitmap blocks were empty.

Could it be that we have enough data to perform SFS hardfile shrinking ?