Fastest depacker

Galahad/FLT · 16 June 2019, 01:15

I'm doing a project that needs to have realtime depacking, but my usual goto choice of RNC Propack is too slow.

Each file is 3200 bytes in length, and whilst I appreciate the fastest depacker might not get the best results packing, faster depacking speed is more essential.

I'm after a data file packer not an executable one.

so has anyone ever done any experiments with packers and evaluated which one by far was the fastest to depack, even if its packing results were not among the best?

hitchhikr · 16 June 2019, 02:27

Did you try lzo ?

a/b · 16 June 2019, 03:43

I generally don't have very strict constraints when it comes to depacking, so I stick with xkp: Nuke for general stuff, Duke for mods, Fast if I need a really fast decompression. So maybe Fast would qualify. All 3 are available in source form on aminet (xkp_source.lha).

ross · 16 June 2019, 09:23

You need to quantify how "fast"

There is so many packer..

But if you are not too concerned about packing ratio then LZ4 is for you.

A version by Leonard:
https://github.com/arnaud-carre/lz4-68k
[lz4_fastest.asm is 6.81 times faster than ARJ and 4.45 times faster than UPX]

Speaking of UPX.. I'm working in a new toolchain for nrv2b

There is still something to squeeze out from it..

meynaf · 16 June 2019, 10:07

Fastest depacker i know of is simple RLE, aka cmpbyterun1 (like used in iff ilbm images).
But the packing ratio, of course, is also the worse...

DanScott · 16 June 2019, 11:16

We used a combination of LZ4 & ARJ in De Profundis

LZ4 is incredibly quick to decompress.. the roto-zoomer images (32k each) are decompressed in the background on the fly using idle CPU time (about 30% of a frame)

Galahad/FLT · 16 June 2019, 11:56

Thanks guys, LZ4 gets the vote it seems, will try that out

Retro1234 · 16 June 2019, 11:57

If it was for gaming probably some basic hand written lossless would be best and could probably decompressed on the fly.

Galahad/FLT · 16 June 2019, 12:01

Quote:

Originally Posted by Retro1234

If it was for gaming probably some basic hand written low loss would be best and could probably decompressed on the fly.

It is for a game to replace an interrupt loader. If LZ4 doesn't work I will have to tackle it a different way.

ross · 16 June 2019, 12:29

Quote:

Originally Posted by Galahad/FLT

It is for a game to replace an interrupt loader. If LZ4 doesn't work I will have to tackle it a different way.

If LZ4 is not enough, there is also LZ4W.
Its word-based bitcode can reach terrific decompression speeds (>700kb/s on a bare A500).
However normally at the expense of compression ratio.
Anyway for small packets (like yours) can beat standard byte-based LZ4.

It's somewhere on SGDK (aka Sega Genesis Dev Kit).
https://github.com/Stephane-D/SGDK/b...r/bin/lz4w.txt

Galahad/FLT · 16 June 2019, 14:12

What are people using as an LZ4 packer then?

WayneK · 16 June 2019, 18:11

Just the reference Google LZ4 code probably, I think Anima posted all his LZ4 stuff on atari-forum (which I know you're a member of) - the packer is probably there somewhere!

Antiriad_UK · 16 June 2019, 18:35

This looks to be a win32 binary. https://github.com/lz4/lz4/releases/tag/v1.9.1

ross · 16 June 2019, 18:49

I use this:
https://github.com/encode84/lz4x

It's by a well known coder in compression scene.

mcgeezer · 16 June 2019, 22:29

I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking?

I also use RNC ProPack, but i'm really now down to squeezing every remaining byte out of my project and it could be the difference of it going on to two disks as opposed to just one which I would very much prefer.

Geezer

Galahad/FLT · 16 June 2019, 22:42

Quote:

Originally Posted by mcgeezer

I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking?

I also use RNC ProPack, but i'm really now down to squeezing every remaining byte out of my project and it could be the difference of it going on to two disks as opposed to just one which I would very much prefer.

Geezer

Propack is usually very good, Shrinkler is better, but its VERY slow to depack, but Shrinkler was the deciding factor in me getting Nitro to fit on one disk like the original, Propack couldn't quite manage it.

Propack is useful for most applications, its just on this one i'm doing, I need realtime depacking because of the way the project works, otherwise I wouldn't bother at all.

DrBong · 16 June 2019, 23:22

Quote:

Originally Posted by mcgeezer

I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking?

Yeh, what Phill said. Shrinkler will usually get the best bang for your buck for compression, but it's very very slow (e.g. took ~3 minutes IIRC to decompress the 1.4MB Crazy Priest game exec on a standard A500 when I one-disked the game). Not surprising as Shrinkler was likely designed to compress demos.

Cranker is much the same, but will nicely decompress a file as it's loading and won't cost you any real additional memory to do so. Otherwise ProPack, XPK SHRI and PowerData are worth trying for A500 configs IMHO.

mcgeezer · 17 June 2019, 21:29

Quote:

Originally Posted by Galahad/FLT

Propack is usually very good, Shrinkler is better, but its VERY slow to depack, but Shrinkler was the deciding factor in me getting Nitro to fit on one disk like the original, Propack couldn't quite manage it.

Propack is useful for most applications, its just on this one i'm doing, I need realtime depacking because of the way the project works, otherwise I wouldn't bother at all.

Quote:

Originally Posted by DrBong

Yeh, what Phill said. Shrinkler will usually get the best bang for your buck for compression, but it's very very slow (e.g. took ~3 minutes IIRC to decompress the 1.4MB Crazy Priest game exec on a standard A500 when I one-disked the game). Not surprising as Shrinkler was likely designed to compress demos.

Cranker is much the same, but will nicely decompress a file as it's loading and won't cost you any real additional memory to do so. Otherwise ProPack, XPK SHRI and PowerData are worth trying for A500 configs IMHO.

Thanks gents, I'll give these a try and post my results back into this thread when I've done it.

Geezer

Keir · 18 June 2019, 14:30

I will also plug my own gzip-based exe packer. Compression ratio is generally better than Cranker and ProPack, it unpacks in place so memory overhead is low, and it's reasonably fast (in the same ballpark as the usual suspects; not as fast as Cranker which unpacks as it loads).

http://eab.abime.net/showthread.php?t=96129

hitchhikr · 18 June 2019, 20:36

Alternatively, you can try your luck with my own packfire: https://www.pouet.net/prod.php?which=54840

16 June 2019, 01:15	#1
Galahad/FLT Going nowhere Join Date: Oct 2001 Location: United Kingdom Age: 50 Posts: 8,986	Fastest depacker I'm doing a project that needs to have realtime depacking, but my usual goto choice of RNC Propack is too slow. Each file is 3200 bytes in length, and whilst I appreciate the fastest depacker might not get the best results packing, faster depacking speed is more essential. I'm after a data file packer not an executable one. so has anyone ever done any experiments with packers and evaluated which one by far was the fastest to depack, even if its packing results were not among the best?

16 June 2019, 11:57	#8
Retro1234 Phone Homer Join Date: Jun 2006 Location: 5150 Posts: 5,773	If it was for gaming probably some basic hand written lossless would be best and could probably decompressed on the fly. Last edited by Retro1234; 16 June 2019 at 12:04.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What depacker to use? Premier Manager II	pepsimaxman	Coders. General	13	15 July 2019 11:17
Fastest Amiga Games	Djay	Nostalgia & memories	73	09 June 2016 21:42
Fastest unZIP on 030?	Amiga1992	support.Apps	7	04 October 2010 01:15
fastest hardfile or directory ?	turrican3	New to Emulation or Amiga scene	10	06 June 2007 19:08
RNC Data File Depacker v2.1	Nico	New to Emulation or Amiga scene	8	05 May 2002 18:05

16 June 2019, 02:27	#2
hitchhikr Registered User Join Date: Jun 2008 Location: somewhere else Posts: 511	Did you try lzo ?

16 June 2019, 03:43	#3
a/b Registered User Join Date: Jun 2016 Location: europe Posts: 1,039	I generally don't have very strict constraints when it comes to depacking, so I stick with xkp: Nuke for general stuff, Duke for mods, Fast if I need a really fast decompression. So maybe Fast would qualify. All 3 are available in source form on aminet (xkp_source.lha).

16 June 2019, 09:23	#4
ross Defendit numerus Join Date: Mar 2017 Location: Crossing the Rubicon Age: 53 Posts: 4,468	You need to quantify how "fast" There is so many packer.. But if you are not too concerned about packing ratio then LZ4 is for you. A version by Leonard: https://github.com/arnaud-carre/lz4-68k [lz4_fastest.asm is 6.81 times faster than ARJ and 4.45 times faster than UPX] Speaking of UPX.. I'm working in a new toolchain for nrv2b There is still something to squeeze out from it..

16 June 2019, 10:07	#5
meynaf son of 68k Join Date: Nov 2007 Location: Lyon / France Age: 51 Posts: 5,323	Fastest depacker i know of is simple RLE, aka cmpbyterun1 (like used in iff ilbm images). But the packing ratio, of course, is also the worse...

16 June 2019, 11:16	#6
DanScott Lemon. / Core Design Join Date: Mar 2016 Location: Tier 5 Posts: 1,212	We used a combination of LZ4 & ARJ in De Profundis LZ4 is incredibly quick to decompress.. the roto-zoomer images (32k each) are decompressed in the background on the fly using idle CPU time (about 30% of a frame)

16 June 2019, 11:56	#7
Galahad/FLT Going nowhere Join Date: Oct 2001 Location: United Kingdom Age: 50 Posts: 8,986	Thanks guys, LZ4 gets the vote it seems, will try that out

16 June 2019, 14:12	#11
Galahad/FLT Going nowhere Join Date: Oct 2001 Location: United Kingdom Age: 50 Posts: 8,986	What are people using as an LZ4 packer then?

16 June 2019, 18:11	#12
WayneK Registered User Join Date: May 2004 Location: Somewhere secret Age: 50 Posts: 365	Just the reference Google LZ4 code probably, I think Anima posted all his LZ4 stuff on atari-forum (which I know you're a member of) - the packer is probably there somewhere!

16 June 2019, 18:35	#13
Antiriad_UK OCS forever! Join Date: Mar 2019 Location: Birmingham, UK Posts: 418	This looks to be a win32 binary. https://github.com/lz4/lz4/releases/tag/v1.9.1

16 June 2019, 18:49	#14
ross Defendit numerus Join Date: Mar 2017 Location: Crossing the Rubicon Age: 53 Posts: 4,468	I use this: https://github.com/encode84/lz4x It's by a well known coder in compression scene.

16 June 2019, 22:29	#15
mcgeezer Registered User Join Date: Oct 2017 Location: Sunderland, England Posts: 2,702	I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking? I also use RNC ProPack, but i'm really now down to squeezing every remaining byte out of my project and it could be the difference of it going on to two disks as opposed to just one which I would very much prefer. Geezer

18 June 2019, 14:30	#19
Keir Registered User Join Date: May 2011 Location: Cambridge Posts: 682	I will also plug my own gzip-based exe packer. Compression ratio is generally better than Cranker and ProPack, it unpacks in place so memory overhead is low, and it's reasonably fast (in the same ballpark as the usual suspects; not as fast as Cranker which unpacks as it loads). http://eab.abime.net/showthread.php?t=96129

18 June 2019, 20:36	#20
hitchhikr Registered User Join Date: Jun 2008 Location: somewhere else Posts: 511	Alternatively, you can try your luck with my own packfire: https://www.pouet.net/prod.php?which=54840

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)