decruncing on the fly...

[JoeJoe]

Hello everybody,

there is a tutorial or a piece of source code that briefly shows how to unpack onethefly data. With the packer "Shrinkler" or "Cranker" I can pack the raw data. Now I want to read this file piece by piece and unpack it immediately. The unpacking as a whole is no problem.

Thanks and greetings
JoeJoe

[phx]

Originally Posted by JoeJoe

there is a tutorial or a piece of source code that briefly shows how to unpack onethefly data.

Where?

With the packer "Shrinkler" or "Cranker" I can pack the raw data. Now I want to read this file piece by piece and unpack it immediately.

You mean you have a limited buffer and want to read one chunk of the compressed stream after another while decompressing it?

You need a compression format which is represented as a single continuous stream. This works fine for simple compressors like Bytekiller, but it doesn't for LZ compressors, for example, which need a second stream for the literal buffer.

Some of my games use Bytekiller compression and I'm just reading one 512 bytes block after another into a buffer to decompress it on the fly. I have no experience with Shrinkler or Cranker, though.

[JoeJoe]

Sorry, I am looking for a tutorial. Here I have found a good code fragment. https://github.com/keirf/Amiga-Stuff/tree/master/inflate

[Keir]

Originally Posted by JoeJoe

Sorry, I am looking for a tutorial. Here I have found a good code fragment. https://github.com/keirf/Amiga-Stuff/tree/master/inflate

Is your aim to unpack in a small RAM buffer? That doesn't work for inflate as the output stream is your dictionary.

Or is it to hide the unpack time behind disk latency? This is tricky: You would need a pair of coroutines, one for disk, which yields after issuing a disk-read DMA, and one for the compressor, which would yield on a newly-implemented "yield symbol". I'm interested in doing this but had no time (or real purpose for it) so far. It's obviously non-trivial.

[Bartman]

„Titanics Cruncher decrunches while loading...“

[WayneK]

It's quite tricky/unusual on Amiga, I know Mr.Spiv implemented a combined disk-loader/decrunch (almost certainly based on STC?) for the game "Elfmania" but 99.9% of games/demos simply load entire file then decrunch...

[Hannibal]

We used a trackloader+decruncher in 3d demo II intro (which was dumb because it was practically a single-file intro). The design is straightforward: double-buffer the memory that track is loaded into, and while waiting for track N to load into buffer A, decode track N-1 from buffer B into the destination buffer, and the decompressor (which uses an algorithm that allows in-place decompression, and can store its' state and resume) runs through the newly loaded bytes.

In pseudo-code:

start DMA read of track 0 into track buffer 0
while(there are more tracks to read)
{
Wait for DMA track read finish
start DMA read of track N into track buffer (N&1)
Decode buffer track N-1 from track buffer ((N-1)&1) 
Decompress the decoded track N-1
N++
}
Wait for DMA track read finish
Decode buffer track N-1 from track buffer ((N-1)&1) 
Decompress the decoded track N-1

And for the in-place decompression, if you use something like LZ4, you can this by allocating a few bytes extra, and loading the compressed file at the end of the buffer. For example:
if you have a file that's 100000 bytes long when uncompressed, but 30000 bytes when compressed, I'd allocate, say 100008 bytes, and load the compressed file starting 70008 bytes into the buffer.

[JoeJoe]

Thanks guys, very interesting contributions just the example with the trackloader and unpack. If you now adapt the trackloader to read a "DOS" file, it should be possible to load and unpack it.

[mr.spiv]

The original "Smarty and Nasty Gluttons" used track loader that decrunched a track while DMA was loading the next one.. Similar what Hannibal was describing. And if my memory is serving me right Elf-Mania had a similar loader (at least I got paid for providing the (de)compression program for that purpose )

The "new" Smarty and Nasty Gluttons has a hw banding FFS file loader that decrunches files per sector basic. Alas it has no DMA interleaving, which makes the loader slightly slow..