17 June 2023, 04:50 | #61 |
Registered User
Join Date: Feb 2016
Location: Homeless
Posts: 67
|
Good to know! I've looked at some Heimdall files... the compression is a bit more customized than just patching the Pack-Ice header. The main difference are the sizes of the "uncompressed length" values:
Code:
Pack-Ice: 1,1,2,2,3,8,15 (bits) RNC-Old: 1,1,2,2,3,10 (bits) Other than that, it's clearly ICE! based, with the same huffman codes, and even with the very uncommon "if distance>1 then distance=distance+length" feature. |
17 June 2023, 07:37 | #62 |
Registered User
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 74
|
Yes I know there is end code checking in the decompressor - but if that end code is never used in the bitstream (or in some compressors it is broken) I just treat it as a marker for corrupted bitstream
Last edited by temisu; 17 June 2023 at 09:34. |
17 June 2023, 10:24 | #63 | |
Registered User
Join Date: Jan 2008
Location: Warsaw/Poland
Age: 56
Posts: 2,083
|
Quote:
More infos about packer headers you can find in T2F doc: http://aminet.net/package/disk/misc/Track2File |
|
19 June 2023, 00:18 | #64 | ||
Registered User
Join Date: Feb 2016
Location: Homeless
Posts: 67
|
Quote:
Imagine somebody deciding to write a LZW2 compressor based on your decompression code... that will lead to corrupted files without end codes, which will cause buffer overruns and randomly crash during or after decompression when using the original XPK functions : / I don't know if you want to fix that. But if yes: I can make a list of methods that seem to contain end codes, and we could also verify the cholok source files to be sure that they are really actually required end codes (as opposed to unused/padding bits). Quote:
I've looked into the rnc2old files from Knightmare. They are also based on Pack-Ice, but with a few more modifications: Code:
Pack-Ice and RNC1old and RNC2old rawlen_bits = (1,1,2,2,3,8,15) ;-pack-ice rawlen_bits = (1,1,2,2,3,10) ;-rnc1old rawlen_bits = (1,1,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16) ;-rnc2old maxlzlenbits = 10 ;\pack-ice maxdispbits = 12 ;/and rnc1old maxlzlenbits = ([src+0Ch+src_len-1]/10h)+1 ;\rnc2old maxdispbits = ([src+0Ch+src_len-1] AND 0Fh)+1 ;/ Note: rnc2old rawlen can be max 10000h bytes. PS. rnc2old disassembly: Code:
;------------------ ;RNC2old (modified Pack-Ice compressor) disassembly by nocash ;(similar to RNC1old, but RNC2old has more modifications) ;--- ;used by Liberation, Knightmare ;--- ;extracted from Knightmare_v1.2_0966.lha\Data\fed_bootfile ;(with the HUNK_CODE block loaded to offset 00000000h) ;------------------ decompress_rnc2old: 00001050 48E77FFE pushd d1-d7,a0-a6 00001054 2248 movd a1,a0 ;src_start=src 00001056 610000F4 call 114Ch ;@@get32bit ;get id 0000105A B2BC52.. cmpd d1,524E4302h ;"RNC",02h 00001060 66000110 jnz 1172h ;@@bad_id 00001064 610000E6 call 114Ch ;@@get32bit ;get decompressed size 00001068 45E80104 movd a2,104h+a0 ;dst_end=src_start+8+104h 0000106C D5C1 addd a2,d1 ;dst_end=src_start+8+104h+dst_siz 0000106E 610000DC call 114Ch ;@@get32bit ;get compressed size 00001072 4DF01800 movd a6,0h+a0+d1 ;src_end=src_start+0Ch+src_siz 00001076 4BFA00CA movd a5,1142h ;base for data table(s) 0000107A 1826 movb d4,-[a6] ;[src_end-1] ;\ 0000107C 0244000F andw d4,0Fh ;[src_end-1] AND 0Fh ; maxdispbits-1 00001080 1A84 movb [a5],d4 ;[1142h]= ;/ 00001082 1816 movb d4,[a6] ;[src_end-1] ;\ 00001084 E80C shrb d4,4 ;[src_end-1]/10h ; lzlenbits-1 00001086 1B44FFF8 movb [0FFFFFFF8h+a5],d4 ;[113Ah]= ;/ 0000108A 2848 movd a4,a0 ;src_limit=src_start+0Ch 0000108C 264A movd a3,a2 ;dst=src_start+8+104h+dst_siz 0000108E 1626 movb d3,-[a6] ;collected=[src] @@decompress_lop: 00001090 6166 call 10F8h ;@@getbit ;\ ;\ 00001092 6424 jnc 10B8h ;@@rawlen_none ;/ ; 00001094 7C00 movd d6,0h ;rawlen=1 ;\ ; 00001096 6160 call 10F8h ;@@getbit ; ; 00001098 6418 jnc 10B2h ;@@rawlen_this ;/ ; 0000109A 7401 movd d2,1h ;rawlenmax=1 ; get rawlen 0000109C 7A00 movd d5,0h ;rawlenbits=1 ; 0000109E 7C01 movd d6,1h ;rawlenbase=1 ; bits = 1,1,1,2,3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 000010A0 6006 jmp 10A8h ;@@get_rawlen_injump ; base = 0,1,2,3,6,0Dh,1Ch,3Bh,7Ah,F9h,1F8h,3F7h,7F6h,FF5h,1FF4h,3FF3h,7FF2h,FFF1h ;--- ; @@get_rawlen_lop: ; (theoretically supports infinitly increasing sizes, 000010A2 D442 addw d2,d2 ;rawlenmax*2 ; but the 16bit registers will overflow at some point) 000010A4 5242 addw d2,1 ;rawlenmax+1 ; 000010A6 5245 addw d5,1 ;rawlenbits+1 ; @@get_rawlen_injump: ; 000010A8 3005 movw d0,d5 ;rawlenbits ; 000010AA 6156 call 1102h ;@@get_d0_plus1_bits ; 000010AC DC47 addw d6,d7 ;rawlenbase ; 000010AE BE42 cmpw d7,d2 ;rawlenmax ; 000010B0 6CF0 jge 10A2h ;@@get_rawlen_lop ; @@rawlen_this: ;/ @@ ;\ 000010B2 1726 movb -[a3],-[a6] ; copy uncompressed data 000010B4 51CEFFFC loop d6,10B2h ;/ @@rawlen_none: 000010B8 BDCC cmpd a6,a4 ;src,src_limit ;\check if SRC (!) end 000010BA 6F5A jle 1116h ;@@to_1158h ;/ ;note: above could END even if "collected" would contain up to ;seven remaining unprococessed bits ;however, the following lzlen/lzdisp code takes up (at least): ; 1+1+6 bits for lzlen=2 and disp=6bit ; 2+2+5 bits for lzlen=0bit and disp=5bit ; 5+3 bits for maxlzlen=1bit and maxdisp=1bit ;and additionally, the final rawlen takes up (at least): ; 1 bits for rawlen=0 ;those cases sum up to at least 9-10 bits. ;so, it's safe to quit decompression when there are only ;seven or less remaining bits in the bitstream ;(usually there are exactly 0 bits remaining). ;- - - 000010BC 7C01 movd d6,1h ;lzlen=2 ;\ ;\ 000010BE 6138 call 10F8h ;@@getbit ; ; 000010C0 6456 jnc 1118h ;@@lzlen2 ;/ ; 000010C2 7202 movd d1,2h ;\ ; @@get_lzlen_lop: ; ; get lzlen 000010C4 6132 call 10F8h ;@@getbit ; ; 000010C6 54C9FFFC loopc d1,10C4h ;@@get_lzlen_lop ;/ ; 000010CA 103510F9 movb d0,[0FFFFFFF9h+a5+d1] ;[113Ah..] ; 000010CE 6B04 js 10D4h ;with d6=1 ;lzlenbits ; 000010D0 6130 call 1102h ;@@get_d0_plus1_bits ; 000010D2 3C07 movw d6,d7 ; @@ ; 000010D4 103510FD movb d0,[0FFFFFFFDh+a5+d1] ;[113Eh..] ; 000010D8 4880 cbw d0 ;lzlenbase ; 000010DA DC40 addw d6,d0 ; 000010DC 7201 movd d1,1h ;/ @@ ;\ 000010DE 6118 call 10F8h ;@@getbit ; 000010E0 54C9FFFC loopc d1,10DEh ;/ 000010E4 10351001 movb d0,[1h+a5+d1] ;[1142h..] ;dispbits 000010E8 6118 call 1102h ;@@get_d0_plus1_bits ;dispbits 000010EA D241 addw d1,d1 000010EC DE751006 addw d7,[6h+a5+d1] ;[1146h..] ;dispbase 000010F0 6638 jnz 112Ah 000010F2 41EB0001 movd a0,1h+a3 ;dst+1 000010F6 6038 jmp 1130h ;--- @@getbit: 000010F8 D603 addb d3,d3 000010FA 6604 jnz 1100h 000010FC 1626 movb d3,-[a6] 000010FE D703 adxb d3,d3 @@ 00001100 4E75 ret ;--- @@get_d0_plus1_bits: 00001102 4880 cbw d0 00001104 7E00 movd d7,0h 00001106 D603 addb d3,d3 00001108 6604 jnz 110Eh 0000110A 1626 movb d3,-[a6] 0000110C D703 adxb d3,d3 @@ 0000110E DF47 adxw d7,d7 00001110 51C8FFF4 loop d0,1106h 00001114 4E75 ret ;--- @@to_1158h: 00001116 6040 jmp 1158h ;--- @@lzlen2: 00001118 7005 movd d0,5h ;bits=6 0000111A 7200 movd d1,0h ;base=0 0000111C 61DA call 10F8h ;@@getbit 0000111E 6404 jnc 1124h 00001120 7008 movd d0,8h ;bits=9 00001122 7240 movd d1,40h ;base=40h @@ 00001124 61DC call 1102h ;@@get_d0_plus1_bits 00001126 DE41 addw d7,d1 00001128 67C8 jz 10F2h @@ 0000112A DE46 addw d7,d6 0000112C 41F37000 movd a0,0h+a3+d7 ;dst+disp @@ @@lz_copy_lop: ;\ 00001130 1720 movb -[a3],-[a0] ;[dst]=[dst+disp] ; copy 00001132 51CEFFFC loop d6,1130h ;@@lz_copy_lop ;/ 00001136 6000FF58 jmp 1090h ;@@decompress_lop ;--- --- --- ;data (relative to a5=1142h) 0000113A 09 db 9 ;lzlenbits=10 ;VAR! (modified by footer setting) 0000113B 01 db 1 ;lzlenbits=2 0000113C 00 db 0 ;lzlenbits=1 0000113D FF db 0FFh ;lzlenbits=0 ;<-- "0bit" ;--- 0000113E 09 db 9 ;lzlenbase=0Ah 0000113F 05 db 5 ;lzlenbase=06h 00001140 03 db 3 ;lzlenbase=04h 00001141 01 db 1 ;lzlenbase=03h-1 ;<-- will act as len=3 (because it adds "d6=1" to the "0bit" value) ;--- 00001142 0B db 0Bh ;dispbits=12 ;VAR! (modified by footer setting) 00001143 04 db 04h ;dispbits=5 00001144 07 db 07h ;dispbits=8 00001145 00 db 0 ;unused ;--- 00001146 0120 dw 0120h ;dispbase=120h 00001148 0000 dw 0000h ;dispbase=0 0000114A 0020 dw 0020h ;dispbase=20h ;--- --- --- @@get32bit: 0000114C 7403 movd d2,3h @@get32bit_lop: 0000114E E189 shld d1,8 00001150 1218 movb d1,[a0]+ 00001152 51CAFFFA loop d2,114Eh ;@@get32bit_lop 00001156 4E75 ret ;--- ... done 00001158 200A movd d0,a2 ;dst_end 0000115A 908B subd d0,a3 ;dst_end-dst 0000115C 204B movd a0,a3 ;dst 0000115E 2408 movd d2,a0 ;dst 00001160 9489 subd d2,a1 ;dst-src_start ;this SHOULD be 8+104h 00001162 2200 movd d1,d0 ;dst_end-dst (=dst_siz) 00001164 670C jz 1172h ;@@skip_final_reloc ;skip if dst_siz=0 @@ 00001166 12D8 movb [a1]+,[a0]+ copy 00001168 5381 subd d1,1 ;dst_siz-1 0000116A 66FA jnz 1166h @@ 0000116C 4219 clrb [a1]+ clear (8+104h bytes) 0000116E 5382 subd d2,1 00001170 66FA jnz 116Ch @@skip_final_reloc: @@bad_id: 00001172 4CDF7FFE popd d1-d7,a0-a6 ;out: d0=dst_end-dst (decompressed size) 00001176 4E75 ret ;------------------ Last edited by nocash; 19 June 2023 at 01:22. |
||
19 June 2023, 09:47 | #65 |
Registered User
Join Date: Jan 2008
Location: Warsaw/Poland
Age: 56
Posts: 2,083
|
RNC1/RNC2 was standard naming as ascii for mixed ascii and hex headers. Shortest version, even if 1 is not ascii value, but hex value. Same for other packers like LOB with headers like 1LOB3 or 2LOB4.
I dont know if You know WT page, but here you can find more depackers: http://wt.exotica.org.uk/others.html |
27 June 2023, 21:44 | #66 | |
Registered User
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 74
|
Quote:
|
|
28 June 2023, 18:37 | #67 |
Registered User
Join Date: Feb 2016
Location: Homeless
Posts: 67
|
Hmmm, that's odd. If DHUF and CBR0 (and perhaps CBR1) don't compress anything... that makes me wonder if they did work with older XPKmaster versions (maybe they have mis-configured something, causing newer XPKmaster versions to refuse to use them).
|
06 August 2023, 14:05 | #68 |
Registered User
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 74
|
I actually made few changes now when refactoring - the large tables are gone (mentioned by nocash), now changes are merged to master. It also contain lot more (trivial) fixes here and there. Maybe I even look integrating bytekiller/pack-ice at some point
|
23 November 2023, 21:39 | #69 |
Registered User
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 74
|
|
06 December 2023, 02:49 | #70 |
Registered User
Join Date: Jul 2009
Location: Lala Land
Posts: 608
|
I grabbed xfd package which is quite old yesterday to unpack a cracked executable. In order to get it to work, I had to disassemble the executable and manually fudge things.
Are there any tools from this project that would be more capable or extensive, that are available? Last edited by copse; 06 December 2023 at 04:42. |
07 December 2023, 00:16 | #71 | |
Registered User
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 74
|
Quote:
There is xfdmaster, like you found. Then there is xadmaster and finally there is unpacker library. Even though every project has a bit different purpose, there is a lot of overlap both in functionality and formats they support. So it all depends what you want to do. What was the compression algorithm in that executable? I've found a lot of bugs in xfdmaster... |
|
07 December 2023, 00:47 | #72 |
Registered User
Join Date: Jul 2009
Location: Lala Land
Posts: 608
|
Ah, I missed the link in the first post because it was not linkified! Apologies.
The executable embedded another executable which did bytekiller decompression. Some arcane Quartex magic. After I extracted the embedded executable xfdDecrunch was able to recognise it as bytekiller and decompress it, although there is definitely some weirdness in there - like sequences of instructions are broken by bytes that Ghidra cannot disassemble. Could be self-modifying code though, not necessarily decompression corruption. |
07 December 2023, 16:10 | #73 |
Registered User
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 74
|
For bytekiller, xfd is still the best. However, there are a lot of variants and bugs and the decompressor does not have proper error handling so it still could be decompression issue. What if you run the executable in winuae and grab the memory. Does it still look mostly the same?
|
07 December 2023, 21:01 | #74 | |
Registered User
Join Date: Jul 2009
Location: Lala Land
Posts: 608
|
Quote:
|
|
08 December 2023, 05:36 | #75 |
Registered User
Join Date: Jul 2009
Location: Lala Land
Posts: 608
|
Okay, here's my steps for anyone interested in how to use WinUAE to do this.
The answer appears to be correct unpacking, but likely self-modifying code bad ghidra absolute address label guessing breaking code flow. Last edited by copse; 08 December 2023 at 20:19. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
"Voices8" 8 Channel Soundtracker "DemoSongI" song - "This is the Amiga with 8 Voices" | DemosongIHunter | request.Music | 45 | 23 May 2022 20:07 |
"Reminder "Lincs Amiga User Group aka "LAG" Meet Sat 5th of January 2013" | rockape | News | 4 | 30 January 2013 00:06 |
After creating OS 3.9 Emergency Disk: Cannot open "cd.device" unit 2 | Snowwie | support.Other | 2 | 31 March 2012 14:44 |
A source of cheap LCD for "small Amiga" projects... | mabus | support.Hardware | 2 | 14 February 2008 00:25 |
Who hides behind the handle "Source / The Source"? | andreas | Retrogaming General Discussion | 15 | 04 January 2005 16:02 |
|
|