08 August 2011, 22:50 | #1 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Uaeunp (26-10-10) extracts corrupted ADF from DSQ archive
Hi Toni,
I encountered a problem with the current version of Uaeunp when I extracted the ADF disk image from the AUI23 Coverdisk DSQ archive uploaded by asymetrix in this post: http://eab.abime.net/showpost.php?p=771485&postcount=49 uaeunp AUI23-ds.dsq reveals: Code:
[VDIR] ----RWED 1995/08/24 12:34:04 Image.dsq.DIR 926384 ----RWED 1995/08/24 12:34:04 6B5C14DB Image.dsq Now uaeunp Image.dsq reveals: Code:
[VDIR] -------- ------------------- Image.dsq.adf.DIR [VDIR] -------- ------------------- Image.dsq.DIR 926384 -------- ------------------- 6B5C14DB Image.dsq 901120 -------- ------------------- E6E158F0 Image.dsq.adf However, although the checksum of the extracted file is as reported above, the file is corrupt (for example, sector boundaries are displaced). FYI, I have attached a valid ADF image of the coverdisk extracted from asymetrix's archive using the Amiga DiskSqueeze application under WinUAE. Differences between the uaeunp extracted image and this one can be clearly seen. Edit: I have tried earlier uaeunp versions dated 17-10-10, 28-06-10, 08-11-09, 07-11-09, 18-06-09, 30-05-09 and 21-05-09 with the same result. Versions dated 24-09-10 and 12-09-10 give "Couldn't open archive 'Image.dsq'" in response to the command line uaeunp Image.dsq Image.dsq.adf Last edited by prowler; 08 August 2011 at 23:35. |
11 August 2011, 00:25 | #2 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Asymetrix has now uploaded AUI82 Coverdisk DSQ archive here:
http://eab.abime.net/showpost.php?p=771739&postcount=55 Unlike the AUI23 coverdisk referred to in my opening post, that of AUI82 is DiskSpare formatted. I successfully extracted previously uploaded AUI DiskSpare formatted coverdisk DSQ archives using the uaeunp version dated 28-06-2010, but both the latest AUI coverdisk examples uploaded by asymetrix (one 880K, one 960K) are corrupted when extracted from the DSQ archives using uaeunp (any version). Are the DSQ archives uploaded by asymetrix somehow different to those I exttracted previously? Unfortunately, I have not retained those previous archives and they are now missing from the EAB File Server. |
13 August 2011, 00:50 | #3 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
I have now extracted a valid Diskspare formatted .ADF image from asymetrix's AUI82 Coverdisk DSQ archive using DiskSqueeze by unpacking it to a custom FMS device.
Differences between this image and the corrupted one extracted from the same archive using uaeunp can be clearly seen, so I have attached it here. The only plausible reason I can come up with is that the previously uploaded images were probably derived from coverdisks mounted with Diskspare device driver v2.2, which was provided on the companion 880K coverdisks, whereas asymetrix's archives were derived from coverdisks mounted with version 3.2 of the Diskspare device driver. Would this be sufficient to break the uaeunp extraction routines? |
13 August 2011, 10:34 | #4 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,574
|
Should be fixed now (uaeunp-0.8.zip updated)
It seems later versions of DSQ compression added extra header that has byte size of number of sectors per track * 10. No idea what it does but uaeunp now knows about it and skips it. |
13 August 2011, 22:57 | #5 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Thanks, Toni.
Hmm... There are many differences between the ADF images it's extracting now and those it was extracting before, but there are also still many differences between these images and those extracted using DiskSqueeze. I can't tell whether they're any closer without a more thorough look. I'll run some HexCmp comparisons and let you know. |
13 August 2011, 23:14 | #6 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,574
|
Attach uncompressed versions for comparison. I won't uncompress them manually, it is too boring
|
13 August 2011, 23:38 | #7 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Uaeunp uncompressed disk images attached as requested.
The filenames indicate the AUI Coverdisk and Uaeunp version used to extract them. The corresponding Disksqueeze uncompressed disk images, which I have now verified as valid images, are attached to my posts above. |
15 August 2011, 23:17 | #8 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Comparison of uaeunp-extracted Image23(13-08-2011).dsq.adf with target file AUI23.ADF
Extracts from my notes:
Adopting 16-byte boundaries, the first difference between the two files occurs at offset 0x594A0, i.e. uaeunp extracts the first 40% of the disk image accurately. The region 0x594A0-59600 in the target file contains 352 zero bytes, which are missing from the extracted file. Cutting 352 bytes from the end of Image23(13-08-2011).dsq.adf (where it is padded with zeroes) and pasting them back in at offset 0x594A0 aligns the two files once more. The two files are now identical up to offset 0x59760. The region 0x59760-5A010 in the target file contains data, the overwhelming majority of which are zero bytes, which again are missing from the extracted file. Copying this region from the target file, pasting it into Image23(13-08-2011).dsq.adf at offset 0x59760 and cutting an equal number of bytes from the end of the file brings the two files back into alignment. The two files are now identical up to offset 0x5AC00 (a sector boundary). There are 112 zero bytes occupying the region 0x5AC00-5AC70 in the extracted file which are just not present in the target file. Removing them from Image23(13-08-2011).dsq.adf and appending them to the end of the file again realigns the two files. The two files are now identical up to offset 0x5B600 (another sector boundary). Here, the target file contains 176 bytes, again the majority of which are zeroes, which are missing from the extracted file. Copying these bytes from the target file, pasting them into Image23(13-08-2011).dsq.adf at offset 0x5B600 and cutting an equal number of bytes from the end of the file aligns the two files once more. The two files are now identical up to offset 0x5C200 (yet another sector boundary). At this point, there are 176 zero bytes occupying the region 0x5C200-5C2B0 in the extracted file which are just not present in the target file. Removing them from Image23(13-08-2011).dsq.adf and appending them to the end of the file again realigns the two files. The two files are now identical up to offset 0x5CBA0. Here, the target file contains 176 bytes, the majority of which are once again zeroes, which are missing from the extracted file. Copying these bytes from the target file, pasting them into Image23(13-08-2011).dsq.adf at offset 0x5CBA0 and cutting an equal number of bytes from the end of the file brings the two files back into alignment. The two files are now identical up to offset 0x5D800 (a sector boundary once more). ...and so it goes on. 528 zero bytes remain at the end of the Image23(13-08-2011).dsq.adf file. Temporarily removing them and prepending them to the start of the file reveals that the last discrepancy between the two files occurs at offset 0xDB65F, i.e. only the last 2464 bytes in the region 0xDB660-DC000 are identical. It is tempting to suggest that the extraction is breaking when a certain number of consecutive zero bytes are encountered, which is what I thought to start with (hence some of my comments), but there are also many examples of this in parts of the image which have been extracted correctly. I'm reluctant to continue in this direction because I really doubt whether it will be of much use. Edit: Did you want me to attach the uncompressed Image.dsq files as well? Just in case, I attach them here. Last edited by prowler; 15 August 2011 at 23:25. |
16 August 2011, 11:14 | #9 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,574
|
Extra header seems to be some kind of bitmap that tells if block contains only zeros (bit is set), problem is mapping those bits to sectors, it isn't using any obvious way.
|
16 August 2011, 21:03 | #10 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
This extra header: I assume that it is part of the uncompressed Image.dsq file.
I just don't understand why a proprietary format was used here. It would have been far simpler to have used an ADF format and not obfuscated it. Could you give me a pointer to this extra header so I can try to decipher it, please? I am already familiar with the standard Amiga floppy bitmap format, of course. |
16 August 2011, 21:11 | #11 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,574
|
Uncompressed file ("PKD" identifier visible at the start), offset 0x34 (just after "1" at offset 0x33. If it is zero = no extra header included)
Header's size is number of blocks in file / 8 (one bit per block). Set bit = block data (512 bytes) is skipped. (Whats the point? File is supposed to be compressed anyway and 512 bytes of zeros would compress really well..) After header comes normal data. Perhaps the bitmap to block mapping is simple and I missed something obvious but this is too boring task.. |
16 August 2011, 21:37 | #12 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Thanks, Toni!
The Image.dsq files decompressed by uaeunp have got to be the same as those used by the Disksqueeze program before the image is unpacked to disk, because it's only LZX compression after all. It's useful to have the valid adf image to refer to in each case. I'll compare the AUI23 and AUI82 Image.dsq header bitmaps with the contents of the adf images to see if I can identify how the blocks are mapped. If the bit is set merely to indicate that the block is full of zeroes, then this is quite different from how the standard Amiga floppy bitmap works, where the bit is set to indicate that the block is not used. In the latter case, of course, it is perfectly possible for an FFS disk block to be full of zeroes and still be in use. Last edited by prowler; 16 August 2011 at 22:05. |
18 August 2011, 00:33 | #13 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
This Disksqueeze "extra" bitmap header seems to share some features with the standard Amiga floppy disk bitmap.
The bitmap starts at the lowest bit of the first long word and, since the two Boot sectors (Blocks 0 and 1) are not included in the bitmap, this particular bit refers to Block 2. The bitmap ends at the third highest bit of the last long word. The highest two bits in the last long word are not used, but both archives have these bits set. Firstly, I see that only one bit of the Image82.dsq "extra" bitmap header is set ("20" at offset 0xA8), indicating that the AUI82.adf image should contain only one disk block full of zeroes. Sure enough, there is only a single block of zeroes at offset 0x77E00-77FFF (Disk Block 959). Further, comparison of uaeunp-extracted image Image82(13-08-2011).dsq.adf with target file AUI82.adf reveals that the first difference occurs at offset 0x77EB0. Thus, uaeunp extracts very nearly the first 50% of the disk image accurately but, more importantly, it would appear that it is the failure of uaeunp to take account of the bitmap header which is breaking the image extraction from this point onwards. On the other hand, the Image23.dsq bitmap header contains several set bits ("3D" at offset 0x8E and "80" at 0xA2), indicating that the AUI23.adf image should contain 6 disk blocks full of zeroes. In fact it contains 7, but one of these is a boot sector (Block 1). The full list of AUI23.adf Disk Blocks containing all zeroes (with offset in brackets) is 1 (0x200-3FF), 714 (0x59400-595FF), 716 (0x59800-599FF), 717 (0x59A00-59BFF), 718 (0x59C00-59DFF), 719 (0x59E00-59FFF) and 881 (0x6E200-6E3FF). And sure enough, as we have already seen, the first discrepancy betwen the uaeunp-extracted Image23(13-08-2011).dsq.adf and target file AUI23.adf is at offset 0x594A0. To help make the bitmap easier to visualize, here is an ASCII pictorial representation of an 880K Amiga floppy disk bitmap I prepared some time ago: Code:
Bitmap Block ~~~~~~~~~~~~ $000:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] Checksum ^33 ^2 ^65 34^ ^97 66^ $010:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^129 98^ ^161 130^ ^193 162^ ^225 194^ $020:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^257 226^ ^289 258^ ^321 290^ ^353 322^ $030:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^385 354^ ^417 386^ ^449 418^ ^481 450^ $040:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^513 482^ ^545 514^ ^577 546^ ^609 578^ $050:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^641 610^ ^673 642^ ^705 674^ ^737 706^ $060:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^769 738^ ^801 770^ ^833 802^ ^865 834^ $070:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^897 866^ ^929 898^ ^961 930^ ^993 962^ $080:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^1025 994^ ^1057 1026^ ^1089 1056^ ^1121 1090^ $090:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^1153 1122^ ^1185 1154^ ^1217 1186^ ^1249 1218^ $0A0:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^1281 1250^ ^1313 1282^ ^1345 1314^ ^1377 1346^ $0B0:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^1409 1378^ ^1441 1410^ ^1473 1442^ ^1505 1474^ $0C0:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^1537 1506^ ^1569 1538^ ^1601 1570^ ^1633 1602^ $0D0:[00 00 00 00][00 00 00 00]-[00 00 00 00][00 00 00 00] ^1665 1634^ ^1697 1666^ ^1729 1698^ ^1761 1730^ |
18 August 2011, 20:11 | #14 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Hi Toni,
I can confirm that using the above interpretation of the DiskSqueeze "extra" header bitmap, the "20" at offset 0xA8 in the Image82.dsq file correctly indicates that Disk Block 959 is full of zeroes in the AUI82.adf image, and the "3D" at offset 0x8E and "80" at 0xA2 in the Image23.dsq file correctly indicate that Disk Blocks 719, 718, 717, 716, 714 and 881, respectively, are full of zeroes in the AUI23.adf image. |
19 August 2011, 21:17 | #15 |
WinUAE developer
Join Date: Aug 2001
Location: Hämeenlinna/Finland
Age: 49
Posts: 26,574
|
Fixed now. Maybe.
Quite illogical sector mask. If the point was to skip zeroed blocks and save (uncompressed) space, why does it skip boot blocks? It is not too rare to have empty second boot block sector.. |
19 August 2011, 22:17 | #16 | |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Thanks, Toni! I'll test it in a short while.
Edit: Yes, it's fixed!!! Quote:
The only justifiable explanation for choosing to copy the Amiga floppy bitmap layout for this task is if it were easier to program than a more direct system, but you would know more about that than I. Finally, thanks for spending your time on fixing this, Toni! Last edited by prowler; 19 August 2011 at 22:43. |
|
11 October 2011, 18:55 | #17 |
Registered User
Join Date: Apr 2007
Location: Belgrade
Posts: 567
|
is there a converter adf->dsq that can compress a lot of adfs automatically?
( so we can verify uaeunp results) |
11 October 2011, 22:36 | #18 | |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Quote:
To check the DSQ decompression performance of uaeunp objectively, you should use it to unpack archives produced by DiskSqueeze only. I would suggest that you use the DiskSqueeze application to convert ADFs to DSQ under WinUAE for this purpose. |
|
12 October 2011, 17:12 | #19 |
Registered User
Join Date: Apr 2007
Location: Belgrade
Posts: 567
|
but I have many .dsq images. I'd like to be sure nothing is lost after converting to .adf
(presumably, .adf can be better compressed with modern archivers and solid packing.) Can DiskSqueeze be somehow used in batch mode? |
12 October 2011, 20:20 | #20 |
Global Moderator
Join Date: Aug 2008
Location: Sidcup, England
Posts: 10,300
|
Provided your DSQ images were originally produced using DiskSqueeze (and AFAIK there is no other application which might have been used to do that), you have the best means of testing the DSQ decompression performance of uaeunp. With any luck your set of images contains some both with and without the 'extra' header.
What is needed here is to extract the ADFs from all the DSQ archives that you have under WinUAE, and binary compare them (fc file1.adf file2.adf /b can be used for this under Windows) with the ADFs extracted from the same archives using uaeunp. Unfortunately, not only is DiskSqueeze not a straightforward application to use - especially for DiskSpare images, but also there is no way of using it in batch mode for a large number of images due to the 'diskswapping' needed. Zipped ADF images are preferred to DSQ archives for use in emulation, because they are generally smaller and the troublesome DiskSqueeze application can be avoided. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Warlocks ADF Archive | Kitty | Retrogaming General Discussion | 8 | 10 August 2018 14:53 |
Downloadable ADF archive? | Kenan | support.Games | 3 | 13 May 2013 10:15 |
Uaeunp (24-09-10) refuses to acknowledge certain formats. | MethodGit | support.WinUAE | 12 | 05 November 2010 00:21 |
Dsq | mai | request.Apps | 2 | 15 September 2008 22:43 |
The real biggest archive for ADF images is????? | Citrus | Retrogaming General Discussion | 9 | 20 July 2001 04:12 |
|
|