English Amiga Board


Go Back   English Amiga Board > Coders > Coders. Asm / Hardware

 
 
Thread Tools
Old 10 April 2019, 17:42   #1
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,506
In-Place Packers/Unpackers

Hi All,

For my project I'm currently using the Rob Northen packer/depacker because mainly it was the first one I found and it did a job. The problem with it is that it doesn't support inplace depack and some of the file I pack take a while to pack using WinUAE even with warp mode.

Could anyone suggest a better packer to use that is pretty straight forward to use for packing data files only? It needs to support in place depack as I can get back 160Kb of ram if I do. I'll need the assembler unpack routine as well to swap out.

Cheers,
Geezer
mcgeezer is offline  
Old 10 April 2019, 21:01   #2
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,322
Quote:
Originally Posted by mcgeezer View Post
Hi All,

For my project I'm currently using the Rob Northen packer/depacker because mainly it was the first one I found and it did a job. The problem with it is that it doesn't support inplace depack and some of the file I pack take a while to pack using WinUAE even with warp mode.

Could anyone suggest a better packer to use that is pretty straight forward to use for packing data files only? It needs to support in place depack as I can get back 160Kb of ram if I do. I'll need the assembler unpack routine as well to swap out.

Cheers,
Geezer
If what you mean by "in place" I.e. if you load to address $40000, you need to also depack to the same address, RNC Propack most certainly does support that.

It uses quite a large ish stack space though, $180+ bytes to achieve it.

There are 2 types of Propack, look at the header, it will say RNC and then after that a value of $01 or $02.
Galahad/FLT is offline  
Old 10 April 2019, 21:07   #3
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,506
Quote:
Originally Posted by Galahad/FLT View Post
If what you mean by "in place" I.e. if you load to address $40000, you need to also depack to the same address, RNC Propack most certainly does support that.

It uses quite a large ish stack space though, $180+ bytes to achieve it.

There are 2 types of Propack, look at the header, it will say RNC and then after that a value of $01 or $02.
In the version I have the header in the source says this:

Code:
*------------------------------------------------------------------------------ 
* PRO-PACK Unpack Routine (Compact Version) - MC68000, Method 1 
* 
* on entry, 
*	a0.l = start address of packed file 
*	a1.l = start address to unpack file 
*	(note: a1 cannot be equal to a0) 
*	stack space required: $1DC bytes 
* 
*	all other registers are preserved 
*------------------------------------------------------------------------------
So i guess I need Method 2 right?
mcgeezer is offline  
Old 11 April 2019, 06:05   #4
FSizzle
Registered User

 
Join Date: Nov 2017
Location: Los Angeles
Posts: 27
It is quite common for packers which can decompress in-place to require the source to be aligned to the end of the buffer rather than the start.

i.e. if input size = $400 and output size = $1000 then the input data should start at $600 bytes in the buffer.

Is that possibly a requirement of propacker?
FSizzle is offline  
Old 11 April 2019, 06:18   #5
alpine9000
Registered User

 
Join Date: Mar 2016
Location: Australia
Posts: 730
As Galahad said, use method 2 and then you can de-pack in place.

As for packing speed, if you're cross building you can use this on your host machine:

https://github.com/lab313ru/rnc_propack_source

Creates 100% identical files to the RNC version run on the Amiga.
alpine9000 is offline  
Old 11 April 2019, 12:08   #6
Galahad/FLT
Going nowhere

Galahad/FLT's Avatar
 
Join Date: Oct 2001
Location: United Kingdom
Age: 46
Posts: 7,322
I've figured it out, you're using the COMPACT version of RNC 1 to unpack.

You can use RNC 2 or RNC 1 so long as its the non COMPACT version, check out the zone, i've put what I use there
Galahad/FLT is offline  
Old 11 April 2019, 12:20   #7
phx
Natteravn

phx's Avatar
 
Join Date: Nov 2009
Location: Herford / Germany
Posts: 1,484
Another option besides unpacking in place would be to do it while loading. So you only need a small additional buffer, let's say 512 bytes, for the next block.

There are some disadvantages though: It is not as fast, because you have to fill the buffer from time to time, and it only works with packers which use a single data stream. Don't know if Propack falls into this category, but the LZ-packer, which I used for my last games, did not.

Simple packers, like Bytekiller, are usually fine for this method.
phx is offline  
Old 11 April 2019, 12:21   #8
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,506
Quote:
Originally Posted by Galahad/FLT View Post
I've figured it out, you're using the COMPACT version of RNC 1 to unpack.

You can use RNC 2 or RNC 1 so long as its the non COMPACT version, check out the zone, i've put what I use there
Nice one Galahad, will take a look!
mcgeezer is offline  
Old 11 April 2019, 12:23   #9
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,506
Quote:
Originally Posted by phx View Post
Another option besides unpacking in place would be to do it while loading. So you only need a small additional buffer, let's say 512 bytes, for the next block.

There are some disadvantages though: It is not as fast, because you have to fill the buffer from time to time, and it only works with packers which use a single data stream. Don't know if Propack falls into this category, but the LZ-packer, which I used for my last games, did not.

Simple packers, like Bytekiller, are usually fine for this method.
I did have a look on Aminet and seen something called cranker... but it looked a bit over the top for my needs.
mcgeezer is offline  
Old 11 April 2019, 14:09   #10
ross
Per aspera ad astra

ross's Avatar
 
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 49
Posts: 2,228
Damn! I would have so much to say on the subject that I can't find the time to do it...
ross is offline  
Old 11 April 2019, 15:49   #11
dodke
Registered User

 
Join Date: Feb 2018
Location: London / UK
Posts: 73
For very fast on the fly data decompression I've used lz77 a lot.
There are sources here for decompression on 68k and also a C program for compression and decompression which works on any platform:
http://privat.bahnhof.se/wb800787/da...h/lz77_v13.zip

Recently I've started using lz48 though which is pretty much as fast and still simple but slightly better:
http://www.cpcwiki.eu/forum/programm...herdecruncher/
dodke is offline  
Old 11 April 2019, 16:23   #12
SyX
Registered User

 
Join Date: Sep 2004
Location: Brasil
Age: 45
Posts: 153
Quote:
Originally Posted by ross View Post
Damn! I would have so much to say on the subject that I can't find the time to do it...
I would love to see your super optimized aplib decruncher variant, @ross... maybe in a few months you will get enough free time

Those days i use ZX7, although the compressor is slower for big files and doesn't crunch so well than aplib (but never more that a few dozens of bytes); the zx7 decruncher is faster and smaller than aplib; support forward and backward decrunch, of course; and everything is open source, while we will never get the aplib cruncher sources.

Another great feature is that the delta (extra bytes for decrunching in place) is guaranteed that is never bigger than 4 bytes; because that for decrunching in place, you only need to reserve in memory the decrunched file size + an extra long, load the crunched file at the end of this zone and decrunch

And this extra long can be used as a flag for signaling that data was loaded or another thing that you like.

... and i forget, a nice feature is that it is super easy to modify the decruncher for adding a function that changes the calculation of the destination address. That is useful for making funny FX effects while you are decrunching data directly on the screen or if you want to sort your data in other way for getting a better compression rate (for example, storing screen as tiles or interleaved screens as independent planes, ...)

Last edited by SyX; 11 April 2019 at 16:30. Reason: Last note about customizing the decruncher...
SyX is offline  
Old 12 April 2019, 16:30   #13
modrobert
old bearded fool

modrobert's Avatar
 
Join Date: Jan 2010
Location: Bangkok
Age: 52
Posts: 525
I'm assuming this is available in some shape or form on Amiga?

https://en.wikipedia.org/wiki/LZ77_and_LZ78

A personal favourite is the V.42bis variant of this compression modems used back in the day, where data which was already compressed passed through without performance loss, also relatively easy to implement in hardware.

EDIT: Great, found it two posts above mine. Thanks dodke.

Last edited by modrobert; 14 April 2019 at 04:05.
modrobert is offline  
Old 12 April 2019, 23:49   #14
Photon
Moderator

Photon's Avatar
 
Join Date: Nov 2004
Location: Eksjö / Sweden
Posts: 4,781
Galahad and phx have given help, so all that is left is a timid logical reaction: In-place decompression so completely goes against the grain of what data compression is.

Load-to-top-of-dest or even just a few bytes to spare for a load offset (instead of having to copying to/from the stack to slow things down, spending more bytes) allows maximum decompression speed, making your program appear 'zippier' to the user!
Photon is offline  
Old 13 April 2019, 00:20   #15
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,506
Quote:
Originally Posted by Photon View Post
Galahad and phx have given help, so all that is left is a timid logical reaction: In-place decompression so completely goes against the grain of what data compression is.

Load-to-top-of-dest or even just a few bytes to spare for a load offset (instead of having to copying to/from the stack to slow things down, spending more bytes) allows maximum decompression speed, making your program appear 'zippier' to the user!
The lesser complexity benefit out ways the speed benefit for my needs.
mcgeezer is offline  
Old 13 April 2019, 10:37   #16
ross
Per aspera ad astra

ross's Avatar
 
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 49
Posts: 2,228
It's all about costs/benefits.
The simplest thing would be that the compressed data were at the beginning of the buffer containing the final destination.
In this way the user should not worry about anything except to allocate the buffer, load the compressed data at start address and call the unpacker.
But in this case, even neglecting a securyty zone and doing a pure LZ compression, it's a necessity to arrange the bitcoding for backward references access,
which is certainly slightly slower than a forward access.
For the forward case you can think of copying the data to the end of buffer (loss of speed in the unpacker)
or load compressed data directly at the top (slightly complicating the logic of the loader).

I have to quantify the loss of speed for a backward fetch bitcoding.
ross is offline  
Old 13 April 2019, 10:55   #17
ross
Per aspera ad astra

ross's Avatar
 
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 49
Posts: 2,228
Quote:
Originally Posted by SyX View Post
Another great feature is that the delta (extra bytes for decrunching in place) is guaranteed that is never bigger than 4 bytes; because that for decrunching in place, you only need to reserve in memory the decrunched file size + an extra long, load the crunched file at the end of this zone and decrunch
This is interesting.
So if you use the 'bitchoice' token as a longword you can make a real in place decompressor (with backward fetch).
The problem is that it would be rather slow in a bare 68k (no problem for 020+).
But I can't think of any possibility to guarantee a safety margin of 4 bytes except by cutting off the compression at some point for pathological cases.
In any case, for normal graphics and code data, it is rare to find yourself in such negative situations, especially if your compressor uses 'literal sequences' and therefore pure encoding cannot expand the stream much.
ross is offline  
Old 13 April 2019, 16:46   #18
SyX
Registered User

 
Join Date: Sep 2004
Location: Brasil
Age: 45
Posts: 153
Quote:
Originally Posted by ross View Post
In any case, for normal graphics and code data, it is rare to find yourself in such negative situations, especially if your compressor uses 'literal sequences' and therefore pure encoding cannot expand the stream much.
Exactly! And in the days before the new improved winuae debugger, I used the vamos project for testing that the normal case worked well and the same happened with data prepared specially for trying to exploit those pathological cases.

Usually we try to consider every possibility, but sometimes the probability of happening every one of those are very low... not impossible, but well, we have tests for that reason.
SyX is offline  
Old 14 April 2019, 00:47   #19
Photon
Moderator

Photon's Avatar
 
Join Date: Nov 2004
Location: Eksjö / Sweden
Posts: 4,781
Quote:
Originally Posted by mcgeezer View Post
The lesser complexity benefit out ways the speed benefit for my needs.
There's a meme for such a statement



There's absolutely zero complexity from adding an offset.

Rather, someone who made the cruncher for you to use, would facepalm and ask, "What the hell were you thinking, decompressing to the same place?? Don't you even understand the basics of coding? Like, source/destination overwrite, ring a bell?"

Anyway. Understand the tools you use. Load to an appropriate address. Decompress to destination. Simple & lets you select the best compression tool from all the compression tools there are, for your needs.

Aahh..! Now I can go back to my timid self again.

Last edited by Photon; 14 April 2019 at 00:53.
Photon is offline  
Old 14 April 2019, 15:02   #20
mcgeezer
Registered User

 
Join Date: Oct 2017
Location: Sunderland, England
Posts: 1,506
Quote:
Originally Posted by Photon View Post
There's a meme for such a statement



There's absolutely zero complexity from adding an offset.

Rather, someone who made the cruncher for you to use, would facepalm and ask, "What the hell were you thinking, decompressing to the same place?? Don't you even understand the basics of coding? Like, source/destination overwrite, ring a bell?"

Anyway. Understand the tools you use. Load to an appropriate address. Decompress to destination. Simple & lets you select the best compression tool from all the compression tools there are, for your needs.

Aahh..! Now I can go back to my timid self again.
There's no need for this response, you're a moderator but more importantly have some respect for a fellow coder, I suspect you should owe me an apology.

Do I understand the basics of coding? yes I fucking well do! You should try going upward of 20,000 lines of assembler into a game under a time limit and things tend to get a bit complex.

All I asked for was a bit of help with de-packing in place, the speed of which isn't important at this time, the reason was to save on memory (which I have thanks to Galahad). I have used the in place de-pack routine he put into the zone because as it stands I allocate ram, load the file to the start of that allocation and then de-pack using the same address, fair enough I could have loaded the file to an extra offset - but that adds some complexity.

Go back to being your normal self for all I care, but keep things respectful and we'll get along just peachy.

Last edited by mcgeezer; 14 April 2019 at 16:29. Reason: Typos and clarification.
mcgeezer is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
Data Packers bjadams Coders. Blitz Basic 3 21 September 2018 00:41
What are MOD Packers ? chip New to Emulation or Amiga scene 9 20 August 2018 17:01
Runtime data unpackers similar to PPDATA/RTDD? BarryB support.Apps 4 21 December 2012 19:25
A better place to be! fatboy Amiga scene 10 29 September 2011 18:54

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 17:22.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Page generated in 0.09225 seconds with 13 queries