Uaeunp (26-10-10) extracts corrupted ADF from DSQ archive

[prowler]

Hi Toni,

I encountered a problem with the current version of Uaeunp when I extracted the ADF disk image from the AUI23 Coverdisk DSQ archive uploaded by asymetrix in this post:
http://eab.abime.net/showpost.php?p=771485&postcount=49

uaeunp AUI23-ds.dsq reveals:

  [VDIR] ----RWED 1995/08/24 12:34:04          Image.dsq.DIR
  926384 ----RWED 1995/08/24 12:34:04 6B5C14DB Image.dsq

and uaeunp AUI23-ds.dsq Image.dsq extracts Image.dsq correctly.

Now uaeunp Image.dsq reveals:

   [VDIR] -------- -------------------          Image.dsq.adf.DIR
   [VDIR] -------- -------------------          Image.dsq.DIR
   926384 -------- ------------------- 6B5C14DB Image.dsq
   901120 -------- ------------------- E6E158F0 Image.dsq.adf

and uaeunp Image.dsq Image.dsq.adf extracts Image.dsq.adf correctly.

However, although the checksum of the extracted file is as reported above, the file is corrupt (for example, sector boundaries are displaced).

FYI, I have attached a valid ADF image of the coverdisk extracted from asymetrix's archive using the Amiga DiskSqueeze application under WinUAE. Differences between the uaeunp extracted image and this one can be clearly seen.

Edit: I have tried earlier uaeunp versions dated 17-10-10, 28-06-10, 08-11-09, 07-11-09, 18-06-09, 30-05-09 and 21-05-09 with the same result.
Versions dated 24-09-10 and 12-09-10 give "Couldn't open archive 'Image.dsq'" in response to the command line uaeunp Image.dsq Image.dsq.adf

[prowler]

Asymetrix has now uploaded AUI82 Coverdisk DSQ archive here:
http://eab.abime.net/showpost.php?p=771739&postcount=55

Unlike the AUI23 coverdisk referred to in my opening post, that of AUI82 is DiskSpare formatted.

I successfully extracted previously uploaded AUI DiskSpare formatted coverdisk DSQ archives using the uaeunp version dated 28-06-2010, but both the latest AUI coverdisk examples uploaded by asymetrix (one 880K, one 960K) are corrupted when extracted from the DSQ archives using uaeunp (any version).

Are the DSQ archives uploaded by asymetrix somehow different to those I exttracted previously? Unfortunately, I have not retained those previous archives and they are now missing from the EAB File Server.

[prowler]

I have now extracted a valid Diskspare formatted .ADF image from asymetrix's AUI82 Coverdisk DSQ archive using DiskSqueeze by unpacking it to a custom FMS device.

Differences between this image and the corrupted one extracted from the same archive using uaeunp can be clearly seen, so I have attached it here.

Originally Posted by prowler

Are the DSQ archives uploaded by asymetrix somehow different to those I extracted previously? Unfortunately, I have not retained those previous archives and they are now missing from the EAB File Server.

The only plausible reason I can come up with is that the previously uploaded images were probably derived from coverdisks mounted with Diskspare device driver v2.2, which was provided on the companion 880K coverdisks, whereas asymetrix's archives were derived from coverdisks mounted with version 3.2 of the Diskspare device driver. Would this be sufficient to break the uaeunp extraction routines?

[Toni Wilen]

Should be fixed now (uaeunp-0.8.zip updated)

It seems later versions of DSQ compression added extra header that has byte size of number of sectors per track * 10. No idea what it does but uaeunp now knows about it and skips it.

[prowler]

Thanks, Toni.

Hmm... There are many differences between the ADF images it's extracting now and those it was extracting before, but there are also still many differences between these images and those extracted using DiskSqueeze.

I can't tell whether they're any closer without a more thorough look.

I'll run some HexCmp comparisons and let you know.

[Toni Wilen]

Attach uncompressed versions for comparison. I won't uncompress them manually, it is too boring

[prowler]

Uaeunp uncompressed disk images attached as requested.

The filenames indicate the AUI Coverdisk and Uaeunp version used to extract them. The corresponding Disksqueeze uncompressed disk images, which I have now verified as valid images, are attached to my posts above.

[prowler]

Extracts from my notes:

Adopting 16-byte boundaries, the first difference between the two files occurs at offset 0x594A0, i.e. uaeunp extracts the first 40% of the disk image accurately.

The region 0x594A0-59600 in the target file contains 352 zero bytes, which are missing from the extracted file.

Cutting 352 bytes from the end of Image23(13-08-2011).dsq.adf (where it is padded with zeroes) and pasting them back in at offset 0x594A0 aligns the two files once more.

The two files are now identical up to offset 0x59760.

The region 0x59760-5A010 in the target file contains data, the overwhelming majority of which are zero bytes, which again are missing from the extracted file.

Copying this region from the target file, pasting it into Image23(13-08-2011).dsq.adf at offset 0x59760 and cutting an equal number of bytes from the end of the file brings the two files back into alignment.

The two files are now identical up to offset 0x5AC00 (a sector boundary).

There are 112 zero bytes occupying the region 0x5AC00-5AC70 in the extracted file which are just not present in the target file.

Removing them from Image23(13-08-2011).dsq.adf and appending them to the end of the file again realigns the two files.

The two files are now identical up to offset 0x5B600 (another sector boundary).

Here, the target file contains 176 bytes, again the majority of which are zeroes, which are missing from the extracted file.

Copying these bytes from the target file, pasting them into Image23(13-08-2011).dsq.adf at offset 0x5B600 and cutting an equal number of bytes from the end of the file aligns the two files once more.

The two files are now identical up to offset 0x5C200 (yet another sector boundary).

At this point, there are 176 zero bytes occupying the region 0x5C200-5C2B0 in the extracted file which are just not present in the target file.

Removing them from Image23(13-08-2011).dsq.adf and appending them to the end of the file again realigns the two files.

The two files are now identical up to offset 0x5CBA0.

Here, the target file contains 176 bytes, the majority of which are once again zeroes, which are missing from the extracted file.

Copying these bytes from the target file, pasting them into Image23(13-08-2011).dsq.adf at offset 0x5CBA0 and cutting an equal number of bytes from the end of the file brings the two files back into alignment.

The two files are now identical up to offset 0x5D800 (a sector boundary once more).

...and so it goes on.

528 zero bytes remain at the end of the Image23(13-08-2011).dsq.adf file. Temporarily removing them and prepending them to the start of the file reveals that the last discrepancy between the two files occurs at offset 0xDB65F, i.e. only the last 2464 bytes in the region 0xDB660-DC000 are identical.

It is tempting to suggest that the extraction is breaking when a certain number of consecutive zero bytes are encountered, which is what I thought to start with (hence some of my comments), but there are also many examples of this in parts of the image which have been extracted correctly.

I'm reluctant to continue in this direction because I really doubt whether it will be of much use.

Edit: Did you want me to attach the uncompressed Image.dsq files as well? Just in case, I attach them here.

[Toni Wilen]

Extra header seems to be some kind of bitmap that tells if block contains only zeros (bit is set), problem is mapping those bits to sectors, it isn't using any obvious way.

[prowler]

This extra header: I assume that it is part of the uncompressed Image.dsq file.

I just don't understand why a proprietary format was used here. It would have been far simpler to have used an ADF format and not obfuscated it.

Could you give me a pointer to this extra header so I can try to decipher it, please? I am already familiar with the standard Amiga floppy bitmap format, of course.

[Toni Wilen]

Uncompressed file ("PKD" identifier visible at the start), offset 0x34 (just after "1" at offset 0x33. If it is zero = no extra header included)

Header's size is number of blocks in file / 8 (one bit per block). Set bit = block data (512 bytes) is skipped. (Whats the point? File is supposed to be compressed anyway and 512 bytes of zeros would compress really well..) After header comes normal data.

Perhaps the bitmap to block mapping is simple and I missed something obvious but this is too boring task..

[prowler]

Thanks, Toni!

The Image.dsq files decompressed by uaeunp have got to be the same as those used by the Disksqueeze program before the image is unpacked to disk, because it's only LZX compression after all.

It's useful to have the valid adf image to refer to in each case. I'll compare the AUI23 and AUI82 Image.dsq header bitmaps with the contents of the adf images to see if I can identify how the blocks are mapped.

If the bit is set merely to indicate that the block is full of zeroes, then this is quite different from how the standard Amiga floppy bitmap works, where the bit is set to indicate that the block is not used. In the latter case, of course, it is perfectly possible for an FFS disk block to be full of zeroes and still be in use.

[prowler]

This Disksqueeze "extra" bitmap header seems to share some features with the standard Amiga floppy disk bitmap.

The bitmap starts at the lowest bit of the first long word and, since the two Boot sectors (Blocks 0 and 1) are not included in the bitmap, this particular bit refers to Block 2. The bitmap ends at the third highest bit of the last long word. The highest two bits in the last long word are not used, but both archives have these bits set.

Firstly, I see that only one bit of the Image82.dsq "extra" bitmap header is set ("20" at offset 0xA8), indicating that the AUI82.adf image should contain only one disk block full of zeroes.

Sure enough, there is only a single block of zeroes at offset 0x77E00-77FFF (Disk Block 959).

Further, comparison of uaeunp-extracted image Image82(13-08-2011).dsq.adf with target file AUI82.adf reveals that the first difference occurs at offset 0x77EB0. Thus, uaeunp extracts very nearly the first 50% of the disk image accurately but, more importantly, it would appear that it is the failure of uaeunp to take account of the bitmap header which is breaking the image extraction from this point onwards.

On the other hand, the Image23.dsq bitmap header contains several set bits ("3D" at offset 0x8E and "80" at 0xA2), indicating that the AUI23.adf image should contain 6 disk blocks full of zeroes.

In fact it contains 7, but one of these is a boot sector (Block 1).

The full list of AUI23.adf Disk Blocks containing all zeroes (with offset in brackets) is 1 (0x200-3FF), 714 (0x59400-595FF), 716 (0x59800-599FF), 717 (0x59A00-59BFF), 718 (0x59C00-59DFF), 719 (0x59E00-59FFF) and 881 (0x6E200-6E3FF).

And sure enough, as we have already seen, the first discrepancy betwen the uaeunp-extracted Image23(13-08-2011).dsq.adf and target file AUI23.adf is at offset 0x594A0.

To help make the bitmap easier to visualize, here is an ASCII pictorial representation of an 880K Amiga floppy disk bitmap I prepared some time ago:

Bitmap Block
~~~~~~~~~~~~
$000:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
         Checksum     ^33          ^2  ^65        34^  ^97        66^
$010:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^129       98^  ^161      130^   ^193      162^  ^225      194^
$020:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^257      226^  ^289      258^   ^321      290^  ^353      322^
$030:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^385      354^  ^417      386^   ^449      418^  ^481      450^
$040:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^513      482^  ^545      514^   ^577      546^  ^609      578^
$050:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^641      610^  ^673      642^   ^705      674^  ^737      706^
$060:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^769      738^  ^801      770^   ^833      802^  ^865      834^
$070:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^897      866^  ^929      898^   ^961      930^  ^993      962^
$080:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^1025     994^  ^1057    1026^   ^1089    1056^  ^1121    1090^
$090:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^1153    1122^  ^1185    1154^   ^1217    1186^  ^1249    1218^
$0A0:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^1281    1250^  ^1313    1282^   ^1345    1314^  ^1377    1346^
$0B0:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^1409    1378^  ^1441    1410^   ^1473    1442^  ^1505    1474^
$0C0:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^1537    1506^  ^1569    1538^   ^1601    1570^  ^1633    1602^
$0D0:[00  00  00  00][00  00  00  00]-[00  00  00  00][00  00  00  00]
      ^1665    1634^  ^1697    1666^   ^1729    1698^  ^1761    1730^

I hope this helps!

[prowler]

Hi Toni,

I can confirm that using the above interpretation of the DiskSqueeze "extra" header bitmap, the "20" at offset 0xA8 in the Image82.dsq file correctly indicates that Disk Block 959 is full of zeroes in the AUI82.adf image, and the "3D" at offset 0x8E and "80" at 0xA2 in the Image23.dsq file correctly indicate that Disk Blocks 719, 718, 717, 716, 714 and 881, respectively, are full of zeroes in the AUI23.adf image.

[Toni Wilen]

Fixed now. Maybe.

Quite illogical sector mask. If the point was to skip zeroed blocks and save (uncompressed) space, why does it skip boot blocks? It is not too rare to have empty second boot block sector..

[prowler]

Originally Posted by Toni Wilen

Fixed now. Maybe

Thanks, Toni! I'll test it in a short while.

Edit: Yes, it's fixed!!!

Originally Posted by Toni Wilen

Quite illogical sector mask. If the point was to skip zeroed blocks and save (uncompressed) space, why does it skip boot blocks? It is not too rare to have empty second boot block sector..

I can't see why a proprietary disk image format was chosen here at all. As you said previously, the inclusion of blocks full of zeroes would have little impact on the size of the compressed archive anyway. The DiskSqueeze application provides no way to access this intermediate, uncompressed disk image, and it provides no support for opening LZX decompressed image archives either!

The only justifiable explanation for choosing to copy the Amiga floppy bitmap layout for this task is if it were easier to program than a more direct system, but you would know more about that than I.

Finally, thanks for spending your time on fixing this, Toni!

[orange]

is there a converter adf->dsq that can compress a lot of adfs automatically?
( so we can verify uaeunp results)

[prowler]

Originally Posted by orange

is there a converter adf->dsq that can compress a lot of adfs automatically?
( so we can verify uaeunp results)

AFAIK, there is no adf->dsq utility which would perform such a batch conversion, but why would you use it for this task?

To check the DSQ decompression performance of uaeunp objectively, you should use it to unpack archives produced by DiskSqueeze only.

I would suggest that you use the DiskSqueeze application to convert ADFs to DSQ under WinUAE for this purpose.

[orange]

but I have many .dsq images. I'd like to be sure nothing is lost after converting to .adf
(presumably, .adf can be better compressed with modern archivers and solid packing.)

Can DiskSqueeze be somehow used in batch mode?

[prowler]

Provided your DSQ images were originally produced using DiskSqueeze (and AFAIK there is no other application which might have been used to do that), you have the best means of testing the DSQ decompression performance of uaeunp. With any luck your set of images contains some both with and without the 'extra' header.

What is needed here is to extract the ADFs from all the DSQ archives that you have under WinUAE, and binary compare them (fc file1.adf file2.adf /b can be used for this under Windows) with the ADFs extracted from the same archives using uaeunp.

Unfortunately, not only is DiskSqueeze not a straightforward application to use - especially for DiskSpare images, but also there is no way of using it in batch mode for a large number of images due to the 'diskswapping' needed.

Zipped ADF images are preferred to DSQ archives for use in emulation, because they are generally smaller and the troublesome DiskSqueeze application can be avoided.