Cross-developer executable packer

[Keir]

I have just completed a tool for creating self-unpacking Amiga executables in a cross-development environment.

To build, you need bebbo's amiga-gcc6 installed. Then:

git clone https://github.com/keirf/Amiga-Stuff
cd Amiga-Stuff/inflate && make

To pack an executable:

python3 path/to/inflate/pack_exe.py original.exe packed.exe

This uses gzip -9 Zopfli for the compression, so it's quite decent, and the unpack times aren't too bad.

[Keir]

I have applied some fixes and improved the script's compression summary. I'm using it for packing game executables as part of patching for FlashFloppy's auto-disk-swap feature. For example on Monkey Island 2:

python3 ../pack/pack_exe.py "Monkey2 Disk 1/monkey2" monkey2.pack
 [Nr] Type      File (   delta,      %)    Memory (delta)
-----------------------------------------------------------
 [--] Header     144 (      +4,  +2.9%)
 [00] Code     72060 (  -79024, -52.3%)    143432 (   +8)
 [01] Data       128 (    -128, -50.0%)       148 (   +4)
 [02] BSS          8 (      -4, -33.3%)         8 (   +0)
 [03] BSS          8 (      -4, -33.3%)         8 (   +0)
 [04] BSS          8 (      -4, -33.3%)      2048 (   +0)
 [05] Data       160 (    -208, -56.5%)       188 (   +4)
 [06] BSS          8 (      -4, -33.3%)         4 (   +0)
 [07] Data        20 (      -4, -16.7%)        12 (   +0)
 [08] BSS          8 (      -4, -33.3%)       208 (   +0)
 [09] Data        24 (     -20, -45.5%)        36 (   +4)
 [10] BSS          8 (      -4, -33.3%)       584 (   +0)
 [11] BSS          8 (      -4, -33.3%)       528 (   +0)
 [12] Data        32 (     -12, -27.3%)        36 (   +4)
 [13] BSS          8 (      -4, -33.3%)       912 (   +0)
 [14] BSS          8 (      -4, -33.3%)       148 (   +0)
 [15] BSS          8 (      -4, -33.3%)       396 (   +0)
 [16] Data        16 (      -4, -20.0%)         8 (   +0)
 [17] BSS          8 (      -4, -33.3%)       132 (   +0)
 [18] Data        32 (     -12, -27.3%)        36 (   +4)
 [19] BSS          8 (      -4, -33.3%)       960 (   +0)
 [20] Data        12 (      -4, -25.0%)         4 (   +0)
 [21] BSS          8 (      -4, -33.3%)       436 (   +0)
 [22] Data        76 (      -4,  -5.0%)        68 (   +0)
 [23] Data       248 (    -604, -70.9%)       844 (   +4)
 [24] Data        88 (     -44, -33.3%)       100 (   +4)
 [25] BSS          8 (      -4, -33.3%)      5072 (   +0)
 [26] Data        60 (     -24, -28.6%)        76 (   +4)
 [27] BSS          8 (      -4, -33.3%)        52 (   +0)
 [28] BSS          8 (      -4, -33.3%)         8 (   +0)
 [29] Data      2952 (   -5764, -66.1%)     30572 (   +0)
 [30] DEPACK    2448                         3980
-----------------------------------------------------------
               78628 (  -83468, -51.5%)    191044 (+4020)
After depack:                              187064 (  +40)

** RESULT:
** Original: Monkey2 Disk 1/monkey2 = 162096 bytes
** Compressed: monkey2.pack = 78628 bytes
** Shrunk 83468 bytes (51.5%)

[WayneK]

Can we have a 'raw data' version of the unpacker, for use in cracks etc? I'm sure we can handle gzipping the files + altering headers, but if you wanted to add a 'raw' mode to the packer script too that would make it even easier

[Keir]

Originally Posted by WayneK

Can we have a 'raw data' version of the unpacker, for use in cracks etc? I'm sure we can handle gzipping the files + altering headers, but if you wanted to add a 'raw' mode to the packer script too that would make it even easier

The script shells out to another tool to extract the raw DEFLATE stream. See degzip in the same Amiga-Stuff/inflate dir as the script.

It also tells you the number of security bytes (aka leeway) that you need if you unpack in place (ie. with the packed data at the end of the unpack buffer, how many bytes longer than the unpacked data must the buffer be).

[ross]

Originally Posted by kaffer

I have just completed a tool for creating self-unpacking Amiga executables in a cross-development environment.

This uses gzip -9 for the compression, so it's quite decent, and the unpack times aren't too bad.

Where you got the unpacker?
There is somewhere on board a very fast routine available.

Originally Posted by kaffer

It also tells you the number of security bytes (aka leeway) that you need if you unpack in place (ie. with the packed data at the end of the unpack buffer, how many bytes longer than the unpacked data must the buffer be).

What are the values on average for the leeway?

[Keir]

Originally Posted by ross

Where you got the unpacker?
There is somewhere on board a very fast routine available.

I use my own, it's fairly fast especially given its size and the complexity of DEFLATE stream handling. Let's say it processes on a basic A500: 25kB/s of compressed input stream, generating approx 50kB/s of output.
(EDIT: This approximately offsets the reduced disk-access time if reading from floppy.)

If there is a faster routine I would like to know, but it depends how big it is: my routine is about 930 bytes, or 630 bytes for a smaller but slower (50%) build.

What are the values on average for the leeway?

It's almost always 2 bytes, which gets rounded up to a longword per compressed hunk.

EDIT: You can see this in the output example above, the last summary column is leeway added to each hunk. Where that is zero is usually because the hunk data did not exist or did not compress. The first hunk has +8 bytes because I need room at the end of the segment to write my epilogue code that jumps to the original exe (after the main Depacker segment has freed itself):

movem.l (sp)+,d0-d7/a0-a6
rts

[Keir]

Update: The packer now shells out to Zopfli instead of Gzip. Zopfli is a modern drop-in replacement for gzip with significantly better compression ratio (usually 3-4%).

So I did a test with v2.37 hrtmon.data, as Foul gave figures for some other exe packers in another thread. From Foul (in descending size order):
PowerPacker 5.0b : 55724
Cranker 0.53 : 51168
Imploder 4.0 : 50712
RNC Pro Pack 2.08 : 50500

Now I add my Zopfli-based packer, and also Shrinkler:
Zopfli : 43888
Shrinkler : 39116 (!)

Shrinkler wins, but of course is very slow to unpack (not measured, but figure on approx 20 seconds or so here). It's about 10x slower than my INFLATE routine, which would take about 2 seconds here.