English Amiga Board


Go Back   English Amiga Board > Coders > Coders. General

 
 
Thread Tools
Old 16 June 2019, 02:15   #1
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,326
Fastest depacker

I'm doing a project that needs to have realtime depacking, but my usual goto choice of RNC Propack is too slow.

Each file is 3200 bytes in length, and whilst I appreciate the fastest depacker might not get the best results packing, faster depacking speed is more essential.

I'm after a data file packer not an executable one.

so has anyone ever done any experiments with packers and evaluated which one by far was the fastest to depack, even if its packing results were not among the best?
Galahad/FLT is offline  
Old 16 June 2019, 03:27   #2
hitchhikr
Registered User

 
Join Date: Jun 2008
Location: somewhere else
Posts: 406
Did you try lzo ?
hitchhikr is offline  
Old 16 June 2019, 04:43   #3
a/b
Registered User

 
Join Date: Jun 2016
Location: europe
Posts: 155
I generally don't have very strict constraints when it comes to depacking, so I stick with xkp: Nuke for general stuff, Duke for mods, Fast if I need a really fast decompression. So maybe Fast would qualify. All 3 are available in source form on aminet (xkp_source.lha).
a/b is offline  
Old 16 June 2019, 10:23   #4
ross
Per aspera ad astra

ross's Avatar
 
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 49
Posts: 2,237
You need to quantify how "fast"
There is so many packer..

But if you are not too concerned about packing ratio then LZ4 is for you.

A version by Leonard:
https://github.com/arnaud-carre/lz4-68k
[lz4_fastest.asm is 6.81 times faster than ARJ and 4.45 times faster than UPX]

Speaking of UPX.. I'm working in a new toolchain for nrv2b
There is still something to squeeze out from it..
ross is offline  
Old 16 June 2019, 11:07   #5
meynaf
son of 68k
meynaf's Avatar
 
Join Date: Nov 2007
Location: Lyon / France
Age: 46
Posts: 3,617
Fastest depacker i know of is simple RLE, aka cmpbyterun1 (like used in iff ilbm images).
But the packing ratio, of course, is also the worse...
meynaf is offline  
Old 16 June 2019, 12:16   #6
DanScott
Lemon. / Core Design

DanScott's Avatar
 
Join Date: Mar 2016
Location: Sunny Bournemouth, UK
Posts: 480
We used a combination of LZ4 & ARJ in De Profundis

LZ4 is incredibly quick to decompress.. the roto-zoomer images (32k each) are decompressed in the background on the fly using idle CPU time (about 30% of a frame)
DanScott is offline  
Old 16 June 2019, 12:56   #7
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,326
Thanks guys, LZ4 gets the vote it seems, will try that out
Galahad/FLT is offline  
Old 16 June 2019, 12:57   #8
Retro1234
Boo

Retro1234's Avatar
 
Join Date: Jun 2006
Location: 5150
Posts: 4,373
If it was for gaming probably some basic hand written lossless would be best and could probably decompressed on the fly.

Last edited by Retro1234; 16 June 2019 at 13:04.
Retro1234 is offline  
Old 16 June 2019, 13:01   #9
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,326
Quote:
Originally Posted by Retro1234 View Post
If it was for gaming probably some basic hand written low loss would be best and could probably decompressed on the fly.
It is for a game to replace an interrupt loader. If LZ4 doesn't work I will have to tackle it a different way.
Galahad/FLT is offline  
Old 16 June 2019, 13:29   #10
ross
Per aspera ad astra

ross's Avatar
 
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 49
Posts: 2,237
Quote:
Originally Posted by Galahad/FLT View Post
It is for a game to replace an interrupt loader. If LZ4 doesn't work I will have to tackle it a different way.
If LZ4 is not enough, there is also LZ4W.
Its word-based bitcode can reach terrific decompression speeds (>700kb/s on a bare A500).
However normally at the expense of compression ratio.
Anyway for small packets (like yours) can beat standard byte-based LZ4.

It's somewhere on SGDK (aka Sega Genesis Dev Kit).
https://github.com/Stephane-D/SGDK/b...r/bin/lz4w.txt
ross is offline  
Old 16 June 2019, 15:12   #11
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,326
What are people using as an LZ4 packer then?
Galahad/FLT is offline  
Old 16 June 2019, 19:11   #12
WayneK
Registered User
 
Join Date: May 2004
Location: Somewhere secret
Age: 45
Posts: 294
Just the reference Google LZ4 code probably, I think Anima posted all his LZ4 stuff on atari-forum (which I know you're a member of) - the packer is probably there somewhere!
WayneK is offline  
Old 16 June 2019, 19:35   #13
Antiriad_UK
Registered User

 
Join Date: Mar 2019
Location: Birmingham, UK
Posts: 142
This looks to be a win32 binary. https://github.com/lz4/lz4/releases/tag/v1.9.1
Antiriad_UK is offline  
Old 16 June 2019, 19:49   #14
ross
Per aspera ad astra

ross's Avatar
 
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 49
Posts: 2,237
I use this:
https://github.com/encode84/lz4x

It's by a well known coder in compression scene.
ross is offline  
Old 16 June 2019, 23:29   #15
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,510
I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking?

I also use RNC ProPack, but i'm really now down to squeezing every remaining byte out of my project and it could be the difference of it going on to two disks as opposed to just one which I would very much prefer.

Geezer
mcgeezer is offline  
Old 16 June 2019, 23:42   #16
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,326
Quote:
Originally Posted by mcgeezer View Post
I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking?

I also use RNC ProPack, but i'm really now down to squeezing every remaining byte out of my project and it could be the difference of it going on to two disks as opposed to just one which I would very much prefer.

Geezer
Propack is usually very good, Shrinkler is better, but its VERY slow to depack, but Shrinkler was the deciding factor in me getting Nitro to fit on one disk like the original, Propack couldn't quite manage it.

Propack is useful for most applications, its just on this one i'm doing, I need realtime depacking because of the way the project works, otherwise I wouldn't bother at all.
Galahad/FLT is offline  
Old 17 June 2019, 00:22   #17
DrBong
HOL / AMR Team Member

DrBong's Avatar
 
Join Date: Dec 2001
Location: Australia
Posts: 2,384
Quote:
Originally Posted by mcgeezer View Post
I don't really want to hijack this thread but would it be possible for one or two people to state what they think the best packers are for compression as opposed to speed of unpacking?
Yeh, what Phill said. Shrinkler will usually get the best bang for your buck for compression, but it's very very slow (e.g. took ~3 minutes IIRC to decompress the 1.4MB Crazy Priest game exec on a standard A500 when I one-disked the game). Not surprising as Shrinkler was likely designed to compress demos.

Cranker is much the same, but will nicely decompress a file as it's loading and won't cost you any real additional memory to do so. Otherwise ProPack, XPK SHRI and PowerData are worth trying for A500 configs IMHO.

Last edited by DrBong; 17 June 2019 at 00:33.
DrBong is offline  
Old 17 June 2019, 22:29   #18
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,510
Quote:
Originally Posted by Galahad/FLT View Post
Propack is usually very good, Shrinkler is better, but its VERY slow to depack, but Shrinkler was the deciding factor in me getting Nitro to fit on one disk like the original, Propack couldn't quite manage it.

Propack is useful for most applications, its just on this one i'm doing, I need realtime depacking because of the way the project works, otherwise I wouldn't bother at all.
Quote:
Originally Posted by DrBong View Post
Yeh, what Phill said. Shrinkler will usually get the best bang for your buck for compression, but it's very very slow (e.g. took ~3 minutes IIRC to decompress the 1.4MB Crazy Priest game exec on a standard A500 when I one-disked the game). Not surprising as Shrinkler was likely designed to compress demos.

Cranker is much the same, but will nicely decompress a file as it's loading and won't cost you any real additional memory to do so. Otherwise ProPack, XPK SHRI and PowerData are worth trying for A500 configs IMHO.

Thanks gents, I'll give these a try and post my results back into this thread when I've done it.

Geezer
mcgeezer is offline  
Old 18 June 2019, 15:30   #19
kaffer
Registered User
 
Join Date: May 2011
Location: Cambridge
Posts: 581
I will also plug my own gzip-based exe packer. Compression ratio is generally better than Cranker and ProPack, it unpacks in place so memory overhead is low, and it's reasonably fast (in the same ballpark as the usual suspects; not as fast as Cranker which unpacks as it loads).

http://eab.abime.net/showthread.php?t=96129
kaffer is offline  
Old 18 June 2019, 21:36   #20
hitchhikr
Registered User

 
Join Date: Jun 2008
Location: somewhere else
Posts: 406
Alternatively, you can try your luck with my own packfire: https://www.pouet.net/prod.php?which=54840
hitchhikr is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
What depacker to use? Premier Manager II pepsimaxman Coders. General 13 15 July 2019 12:17
Fastest Amiga Games Djay Nostalgia & memories 73 09 June 2016 22:42
Fastest unZIP on 030? Akira support.Apps 7 04 October 2010 02:15
fastest hardfile or directory ? turrican3 New to Emulation or Amiga scene 10 06 June 2007 20:08
RNC Data File Depacker v2.1 Nico New to Emulation or Amiga scene 8 05 May 2002 19:05

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 13:06.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Page generated in 0.09124 seconds with 13 queries