Open source decompressors for many "old" amiga compressors

[nocash]

Good to know! I've looked at some Heimdall files... the compression is a bit more customized than just patching the Pack-Ice header. The main difference are the sizes of the "uncompressed length" values:

Pack-Ice: 1,1,2,2,3,8,15 (bits)
RNC-Old:  1,1,2,2,3,10   (bits)

Other differences between RNC and Pack-Ice are: Changed header ID, swapped compressed/uncompressed size header entries, the compressed size excludes the 0Ch-byte headersize. And the Atari picture filter feature is completely removed (the bitstream ends with the last compression bit, without the usual trailing filter flag).
Other than that, it's clearly ICE! based, with the same huffman codes, and even with the very uncommon "if distance>1 then distance=distance+length" feature.

[temisu]

Originally Posted by nocash

Ah, okay. Ah, but wait, now that you have mentioned the Choloks package... peeking at the LZW2 code:

beq.s   .end            ;tagend

That does very much look as if the decompressor did use the end code : )

Yes I know there is end code checking in the decompressor - but if that end code is never used in the bitstream (or in some compressors it is broken) I just treat it as a marker for corrupted bitstream

[Don_Adan]

Originally Posted by nocash

Good to know! I've looked at some Heimdall files... the compression is a bit more customized than just patching the Pack-Ice header. The main difference are the sizes of the "uncompressed length" values:

Pack-Ice: 1,1,2,2,3,8,15 (bits)
RNC-Old:  1,1,2,2,3,10   (bits)

Other differences between RNC and Pack-Ice are: Changed header ID, swapped compressed/uncompressed size header entries, the compressed size excludes the 0Ch-byte headersize. And the Atari picture filter feature is completely removed (the bitstream ends with the last compression bit, without the usual trailing filter flag).
Other than that, it's clearly ICE! based, with the same huffman codes, and even with the very uncommon "if distance>1 then distance=distance+length" feature.

On Atari ST exist minimum 5 (if I remember right) versions of Pack Ice packers which using a few (?) different pack methods. On Amiga mostly was used latest version of Pack Ice within "ICE!" header (as IAM Packer). And one or two versions with "Ice!" header. From my memory Ice! depackers are/was not compatible, or better recognition code must be used than only checking for "Ice!" header. Then RNC1Old can be one of these Ice packers or a few changed/modified. If I remember right, then exist also RNC2Old packer.

More infos about packer headers you can find in T2F doc:

http://aminet.net/package/disk/misc/Track2File

[nocash]

Originally Posted by temisu

Yes I know there is end code checking in the decompressor... I just treat it as a marker for corrupted bitstream

Ah, damn. That's fine in closed source decompressors, but not so good in open source. When removing all those "unneccessary" end codes you have somewhat unfortunately downgraded everything from "this is a very good reference" to "this works for decompression".
Imagine somebody deciding to write a LZW2 compressor based on your decompression code... that will lead to corrupted files without end codes, which will cause buffer overruns and randomly crash during or after decompression when using the original XPK functions : /

I don't know if you want to fix that. But if yes: I can make a list of methods that seem to contain end codes, and we could also verify the cholok source files to be sure that they are really actually required end codes (as opposed to unused/padding bits).

Originally Posted by Don_Adan

If I remember right, then exist also RNC2Old packer.
More infos about packer headers you can find in T2F doc:
http://aminet.net/package/disk/misc/Track2File

Interesting package, the Track2File's .guide file seems to be listing a number of IDs that aren't mentioned on the "Crunched file IDs" webpage. Although at least some of the IDs appear to be slightly wrong, for example <"RNC1"> would be <"RNC",01h> in reality.
I've looked into the rnc2old files from Knightmare. They are also based on Pack-Ice, but with a few more modifications:

Pack-Ice and RNC1old and RNC2old
  rawlen_bits  = (1,1,2,2,3,8,15)                              ;-pack-ice
  rawlen_bits  = (1,1,2,2,3,10)                                ;-rnc1old
  rawlen_bits  = (1,1,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16)  ;-rnc2old
  maxlzlenbits = 10                                            ;\pack-ice
  maxdispbits  = 12                                            ;/and rnc1old
  maxlzlenbits = ([src+0Ch+src_len-1]/10h)+1                   ;\rnc2old
  maxdispbits  = ([src+0Ch+src_len-1] AND 0Fh)+1               ;/
Note: rnc2old rawlen can be max 10000h bytes.

There's one oddity: The Knightmare disk contains compressed and uncompressed .pic files. The uncompressed ones are 9C40h bytes (200*200), the (de-)compressed ones are 4 bytes taller, and have trailing <00h,"PFF"> appended at end of file. Maybe that's just coincedence, unrelated to compression. The compressed seg# files don't have "PFF" appended.

PS. rnc2old disassembly:

                 ;------------------
                 ;RNC2old (modified Pack-Ice compressor) disassembly by nocash
                 ;(similar to RNC1old, but RNC2old has more modifications)
                 ;---
                 ;used by Liberation, Knightmare
                 ;---
                 ;extracted from Knightmare_v1.2_0966.lha\Data\fed_bootfile
                 ;(with the HUNK_CODE block loaded to offset 00000000h)
                 ;------------------
                 decompress_rnc2old:
00001050 48E77FFE pushd   d1-d7,a0-a6
00001054 2248     movd    a1,a0                 ;src_start=src
00001056 610000F4 call    114Ch ;@@get32bit     ;get id
0000105A B2BC52.. cmpd    d1,524E4302h          ;"RNC",02h
00001060 66000110 jnz     1172h ;@@bad_id
00001064 610000E6 call    114Ch ;@@get32bit     ;get decompressed size
00001068 45E80104 movd    a2,104h+a0            ;dst_end=src_start+8+104h
0000106C D5C1     addd    a2,d1                 ;dst_end=src_start+8+104h+dst_siz
0000106E 610000DC call    114Ch ;@@get32bit     ;get compressed size
00001072 4DF01800 movd    a6,0h+a0+d1           ;src_end=src_start+0Ch+src_siz
00001076 4BFA00CA movd    a5,1142h              ;base for data table(s)
0000107A 1826     movb    d4,-[a6]              ;[src_end-1]         ;\
0000107C 0244000F andw    d4,0Fh                ;[src_end-1] AND 0Fh ; maxdispbits-1
00001080 1A84     movb    [a5],d4               ;[1142h]=            ;/
00001082 1816     movb    d4,[a6]               ;[src_end-1]         ;\
00001084 E80C     shrb    d4,4                  ;[src_end-1]/10h     ; lzlenbits-1
00001086 1B44FFF8 movb    [0FFFFFFF8h+a5],d4    ;[113Ah]=            ;/
0000108A 2848     movd    a4,a0                 ;src_limit=src_start+0Ch
0000108C 264A     movd    a3,a2                 ;dst=src_start+8+104h+dst_siz
0000108E 1626     movb    d3,-[a6]              ;collected=[src]
                 @@decompress_lop:
00001090 6166     call    10F8h ;@@getbit       ;\      ;\
00001092 6424     jnc     10B8h ;@@rawlen_none  ;/      ;
00001094 7C00     movd    d6,0h ;rawlen=1       ;\      ;
00001096 6160     call    10F8h ;@@getbit       ;       ;
00001098 6418     jnc     10B2h ;@@rawlen_this  ;/      ;
0000109A 7401     movd    d2,1h ;rawlenmax=1            ; get rawlen
0000109C 7A00     movd    d5,0h ;rawlenbits=1           ;
0000109E 7C01     movd    d6,1h ;rawlenbase=1           ; bits = 1,1,1,2,3,  4,  5,  6,  7,  8,   9,  10,  11,  12,   13,   14,   15,   16
000010A0 6006     jmp     10A8h ;@@get_rawlen_injump    ; base = 0,1,2,3,6,0Dh,1Ch,3Bh,7Ah,F9h,1F8h,3F7h,7F6h,FF5h,1FF4h,3FF3h,7FF2h,FFF1h
                 ;---                                   ;
                 @@get_rawlen_lop:                      ; (theoretically supports infinitly increasing sizes,
000010A2 D442     addw    d2,d2 ;rawlenmax*2            ; but the 16bit registers will overflow at some point)
000010A4 5242     addw    d2,1  ;rawlenmax+1            ;
000010A6 5245     addw    d5,1  ;rawlenbits+1           ;
                 @@get_rawlen_injump:                   ;
000010A8 3005     movw    d0,d5 ;rawlenbits             ;
000010AA 6156     call    1102h ;@@get_d0_plus1_bits    ;
000010AC DC47     addw    d6,d7 ;rawlenbase             ;
000010AE BE42     cmpw    d7,d2 ;rawlenmax              ;
000010B0 6CF0     jge     10A2h ;@@get_rawlen_lop       ;
                 @@rawlen_this:                         ;/
                 @@                                     ;\
000010B2 1726     movb    -[a3],-[a6]                   ; copy uncompressed data
000010B4 51CEFFFC loop    d6,10B2h                      ;/
                 @@rawlen_none:
000010B8 BDCC     cmpd    a6,a4    ;src,src_limit       ;\check if SRC (!) end
000010BA 6F5A     jle     1116h ;@@to_1158h             ;/
                 ;note: above could END even if "collected" would contain up to
                 ;seven remaining unprococessed bits
                 ;however, the following lzlen/lzdisp code takes up (at least):
                 ;  1+1+6 bits for lzlen=2 and disp=6bit
                 ;  2+2+5 bits for lzlen=0bit and disp=5bit
                 ;  5+3   bits for maxlzlen=1bit and maxdisp=1bit
                 ;and additionally, the final rawlen takes up (at least):
                 ;  1     bits for rawlen=0
                 ;those cases sum up to at least 9-10 bits.
                 ;so, it's safe to quit decompression when there are only
                 ;seven or less remaining bits in the bitstream
                 ;(usually there are exactly 0 bits remaining).
                 ;- - -
000010BC 7C01     movd    d6,1h ;lzlen=2                ;\      ;\
000010BE 6138     call    10F8h ;@@getbit               ;       ;
000010C0 6456     jnc     1118h ;@@lzlen2               ;/      ;
000010C2 7202     movd    d1,2h                         ;\      ;
                 @@get_lzlen_lop:                       ;       ; get lzlen
000010C4 6132     call    10F8h ;@@getbit               ;       ;
000010C6 54C9FFFC loopc   d1,10C4h ;@@get_lzlen_lop     ;/      ;
000010CA 103510F9 movb    d0,[0FFFFFFF9h+a5+d1]   ;[113Ah..]    ;
000010CE 6B04     js      10D4h        ;with d6=1 ;lzlenbits    ;
000010D0 6130     call    1102h ;@@get_d0_plus1_bits            ;
000010D2 3C07     movw    d6,d7                                 ;
                 @@                                             ;
000010D4 103510FD movb    d0,[0FFFFFFFDh+a5+d1]   ;[113Eh..]    ;
000010D8 4880     cbw     d0                      ;lzlenbase    ;
000010DA DC40     addw    d6,d0                                 ;
000010DC 7201     movd    d1,1h                                 ;/
                 @@                             ;\
000010DE 6118     call    10F8h ;@@getbit       ;
000010E0 54C9FFFC loopc   d1,10DEh              ;/
000010E4 10351001 movb    d0,[1h+a5+d1]  ;[1142h..] ;dispbits
000010E8 6118     call    1102h ;@@get_d0_plus1_bits ;dispbits
000010EA D241     addw    d1,d1
000010EC DE751006 addw    d7,[6h+a5+d1]  ;[1146h..] ;dispbase
000010F0 6638     jnz     112Ah
000010F2 41EB0001 movd    a0,1h+a3      ;dst+1
000010F6 6038     jmp     1130h
                 ;---
                 @@getbit:
000010F8 D603     addb    d3,d3
000010FA 6604     jnz     1100h
000010FC 1626     movb    d3,-[a6]
000010FE D703     adxb    d3,d3
                 @@
00001100 4E75     ret
                 ;---
                 @@get_d0_plus1_bits:
00001102 4880     cbw     d0
00001104 7E00     movd    d7,0h
00001106 D603     addb    d3,d3
00001108 6604     jnz     110Eh
0000110A 1626     movb    d3,-[a6]
0000110C D703     adxb    d3,d3
                 @@
0000110E DF47     adxw    d7,d7
00001110 51C8FFF4 loop    d0,1106h
00001114 4E75     ret
                 ;---
                 @@to_1158h:
00001116 6040     jmp     1158h
                 ;---
                 @@lzlen2:
00001118 7005     movd    d0,5h   ;bits=6
0000111A 7200     movd    d1,0h   ;base=0
0000111C 61DA     call    10F8h ;@@getbit
0000111E 6404     jnc     1124h
00001120 7008     movd    d0,8h   ;bits=9
00001122 7240     movd    d1,40h  ;base=40h
                 @@
00001124 61DC     call    1102h ;@@get_d0_plus1_bits
00001126 DE41     addw    d7,d1
00001128 67C8     jz      10F2h
                 @@
0000112A DE46     addw    d7,d6
0000112C 41F37000 movd    a0,0h+a3+d7   ;dst+disp
                 @@
                 @@lz_copy_lop:                                 ;\
00001130 1720     movb    -[a3],-[a0]   ;[dst]=[dst+disp]       ; copy
00001132 51CEFFFC loop    d6,1130h ;@@lz_copy_lop               ;/
00001136 6000FF58 jmp     1090h ;@@decompress_lop
                 ;--- --- ---
                 ;data (relative to a5=1142h)
0000113A 09       db 9     ;lzlenbits=10  ;VAR! (modified by footer setting)
0000113B 01       db 1     ;lzlenbits=2
0000113C 00       db 0     ;lzlenbits=1
0000113D FF       db 0FFh  ;lzlenbits=0      ;<-- "0bit"
                 ;---
0000113E 09       db 9     ;lzlenbase=0Ah
0000113F 05       db 5     ;lzlenbase=06h
00001140 03       db 3     ;lzlenbase=04h
00001141 01       db 1     ;lzlenbase=03h-1  ;<-- will act as len=3 (because it adds "d6=1" to the "0bit" value)
                 ;---
00001142 0B       db 0Bh   ;dispbits=12   ;VAR! (modified by footer setting)
00001143 04       db 04h   ;dispbits=5
00001144 07       db 07h   ;dispbits=8
00001145 00       db 0     ;unused
                 ;---
00001146 0120     dw 0120h ;dispbase=120h
00001148 0000     dw 0000h ;dispbase=0
0000114A 0020     dw 0020h ;dispbase=20h
                 ;--- --- ---
                 @@get32bit:
0000114C 7403     movd    d2,3h
                 @@get32bit_lop:
0000114E E189     shld    d1,8
00001150 1218     movb    d1,[a0]+
00001152 51CAFFFA loop    d2,114Eh ;@@get32bit_lop
00001156 4E75     ret
                 ;---
                        ... done
00001158 200A     movd    d0,a2   ;dst_end
0000115A 908B     subd    d0,a3   ;dst_end-dst
0000115C 204B     movd    a0,a3   ;dst
0000115E 2408     movd    d2,a0   ;dst
00001160 9489     subd    d2,a1   ;dst-src_start    ;this SHOULD be 8+104h
00001162 2200     movd    d1,d0   ;dst_end-dst (=dst_siz)
00001164 670C     jz      1172h ;@@skip_final_reloc  ;skip if dst_siz=0
                 @@
00001166 12D8     movb    [a1]+,[a0]+            copy
00001168 5381     subd    d1,1    ;dst_siz-1
0000116A 66FA     jnz     1166h
                 @@
0000116C 4219     clrb    [a1]+                  clear (8+104h bytes)
0000116E 5382     subd    d2,1
00001170 66FA     jnz     116Ch
                 @@skip_final_reloc:
                 @@bad_id:
00001172 4CDF7FFE popd    d1-d7,a0-a6   ;out: d0=dst_end-dst (decompressed size)
00001176 4E75     ret
                 ;------------------

[Don_Adan]

RNC1/RNC2 was standard naming as ascii for mixed ascii and hex headers. Shortest version, even if 1 is not ascii value, but hex value. Same for other packers like LOB with headers like 1LOB3 or 2LOB4.
I dont know if You know WT page, but here you can find more depackers:

http://wt.exotica.org.uk/others.html

[temisu]

Originally Posted by nocash

DHUF
I've found the rare DHUF library here https://archive.org/download/Commodo...9%28XPK%29.zip
and here https://aminet.net/package/misc/os/aros-orca-m68k (huge download, 282Mbyte).
Might be interesting to see how it differs from HUFF and HFMN. I can help on disassembling code in the the library.
But I have no idea how to set up a working boot disc with all required XPK libraries for creating a DHUF test file : /

Took some time, but tested these, they are actually the same file in both. And it is the same DHUF that I had already which does not work at all (purely uncompressed passthrough). I'll comment that it is not missing but broken

[nocash]

Hmmm, that's odd. If DHUF and CBR0 (and perhaps CBR1) don't compress anything... that makes me wonder if they did work with older XPKmaster versions (maybe they have mis-configured something, causing newer XPKmaster versions to refuse to use them).

[temisu]

I actually made few changes now when refactoring - the large tables are gone (mentioned by nocash), now changes are merged to master. It also contain lot more (trivial) fixes here and there. Maybe I even look integrating bytekiller/pack-ice at some point

[temisu]

Originally Posted by nocash

PS. rnc2old disassembly:

Well, thanks for the good analysis. I added it to the ancient (although I took my time to do it). So it was definitely not wasted effort

[copse]

I grabbed xfd package which is quite old yesterday to unpack a cracked executable. In order to get it to work, I had to disassemble the executable and manually fudge things.

Are there any tools from this project that would be more capable or extensive, that are available?

[temisu]

Originally Posted by copse

Are there any tools from this project that would be more capable or extensive, that are available?

Of course ancient is the best but seriously though there are other options:

There is xfdmaster, like you found. Then there is xadmaster and finally there is unpacker library.

Even though every project has a bit different purpose, there is a lot of overlap both in functionality and formats they support. So it all depends what you want to do. What was the compression algorithm in that executable? I've found a lot of bugs in xfdmaster...

[copse]

Ah, I missed the link in the first post because it was not linkified! Apologies.

The executable embedded another executable which did bytekiller decompression. Some arcane Quartex magic. After I extracted the embedded executable xfdDecrunch was able to recognise it as bytekiller and decompress it, although there is definitely some weirdness in there - like sequences of instructions are broken by bytes that Ghidra cannot disassemble. Could be self-modifying code though, not necessarily decompression corruption.

[temisu]

Originally Posted by copse

The executable embedded another executable which did bytekiller decompression. Some arcane Quartex magic.

For bytekiller, xfd is still the best. However, there are a lot of variants and bugs and the decompressor does not have proper error handling so it still could be decompression issue. What if you run the executable in winuae and grab the memory. Does it still look mostly the same?

[copse]

Originally Posted by temisu

For bytekiller, xfd is still the best. However, there are a lot of variants and bugs and the decompressor does not have proper error handling so it still could be decompression issue. What if you run the executable in winuae and grab the memory. Does it still look mostly the same?

That's a great idea, thanks!

[copse]

Okay, here's my steps for anyone interested in how to use WinUAE to do this.

• I know FoFT loads at $800 and jumps there. I set DF0 to an FoFT disk and let it load to AmigaDOS.
• I press SHIFT+F12 to open the WinUAE debugger.
• I type "f 0x800" to set a breakpoint to when the freshly unpacked game code is entered.
• I type 'S "c:\foftdump" 0x800 0x77800' to write out the unpacked memory to disk.

The answer appears to be correct unpacking, but likely self-modifying code bad ghidra absolute address label guessing breaking code flow.