Need Help: Packing and Unpacking

[sandruzzo]

I need to pack and unpack data into my exe,for incoming game, never done before. What utils should I use? Any suggestion?

[jotd]

RNC cruncher comes to mind. There are others too but that one has packers & unpackers from Windows & Amiga. Amiga unpack is pure asm & can do in-place decrunch.

A lot of games from the 1990 era used it.

[JoeJoe]

I use the LZ4

[sandruzzo]

@JoeJoe

Any link to get stuffs?

[DanScott]

http://aminet.net/package/util/pack/RNC_ProPack

[jotd]

in C: https://github.com/lab313ru/rnc_propack_source

[JoeJoe]

unpack for 68k

[sandruzzo]

Thanks to all! First time working around pack and unpack on Amiga!

[sandruzzo]

I'll work around ProPack. First I have to pack my projects' datas.

[sparhawk]

I use zlib. Portable and easy to use and can also do inmem de-/packing.

[defor]

Originally Posted by JoeJoe

I use the LZ4

Hello. Hopefully someone can help me.

I wanted to use LZ4 in my current project. However I'm out of luck to make decompression to work.
I use 68k depack routine by Arnaud Carré (Leonard) (https://github.com/arnaud-carre/lz4-68k) - lz4_normal.asm, and to pack data I use original Win64 CLI (https://lz4.github.io/lz4/).
I figured out that I have to compress data using the legacy format ("-l" parameter), otherwise depack routine crashes. However, even then my data are not correctly depacked. I tried various command-line parameters (block sizes, compression values, ...) - no luck. I also tried LZ4X and smallz4 - same result.
What am I missing? What packer do you use?

[a/b]

Did you use --no-frame-crc? A while ago I was experimenting with lz4 and didn't have any problems with decompression.

[defor]

Originally Posted by a/b

Did you use --no-frame-crc? A while ago I was experimenting with lz4 and didn't have any problems with decompression.

Yes, I did. Surprisingly the compressed file size is the same with or without crc check. Is it alright? I also tried various block sizes (from "-B4" to "-B7") and "-BI" option. I tried various lz4.exe versions, from 1.9.2 down to 1.7.3. Maybe I should go even lower?
I tried normal and fast LZ4 decrunch routines.
Just to be sure that I don't have bug in a different part of my code I compressed data (4bpl 320x192 image in this case) by RNC packer (Lab313 version) and used ProPack routine from Aminet -- everything is fine. So currently I'm rather inclined to use RNC instead of LZ4. In my case, the compression is slightly better and speed is comparable. Yet, it haunts me what am I doing wrong?

[a/b]

Hmm, ok.. Looking at his code now. He assumes there is no header, and a0 points to a packed data block. Now depending on compression flags (like CRC or no CRC) you have to skip a few bytes containing the header. With no CRC and no other flags that would be 11 bytes (4 magic ID, 3 frame descriptor, 4 block length in little-endian format). If you wanted to include the original file length (--content-size flag) it would be additional 8 bytes, for example (if I remember correctly).
Or alternatively, if you want to parse the frame descriptor you can figure out which parts of the header are present and which are not, and based on that skip as many bytes as needed.

[Antiriad_UK]

Check out my github where I wrapped up various routines like LZ4, NRV2S, RNC, cranker, shrinkler, etc.

ConvertAssets.cmd shows the correct command line for the various utils.
IntroLibrary.s shows how to call the depack routines with a source in a0, dest in a1 type api.

https://github.com/jonathanbennett73/amiga-depack

[leonard]

Originally Posted by defor

I figured out that I have to compress data using the legacy format ("-l" parameter), otherwise depack routine crashes. However, even then my data are not correctly depacked. I tried various command-line parameters (block sizes, compression values, ...) - no luck. I also tried LZ4X and smallz4 - same result.
What am I missing? What packer do you use?

lz4_small.asm etc are working on raw LZ4 blocks, and not on LZ4 file header. To work with LZ4 file header ( so what lz4.exe produces ), just include https://github.com/arnaud-carre/lz4-.../lz4_frame.asm

As the comment says, it depacks data packed with this exact command line:

; Depack data produced by LZ4.exe command line:
; lz4.exe -9 --no-frame-crc <input_file> <output_file>

https://github.com/arnaud-carre/lz4-68k#lz4-frame

Hope it helps!

[defor]

Originally Posted by Antiriad_UK

Check out my github where I wrapped up various routines like LZ4, NRV2S, RNC, cranker, shrinkler, etc.

ConvertAssets.cmd shows the correct command line for the various utils.
IntroLibrary.s shows how to call the depack routines with a source in a0, dest in a1 type api.

https://github.com/jonathanbennett73/amiga-depack

Thanks for your tip!
So obviously I was really just missing something important -- I haven't looked at "lz4_frame.asm" where it's already correctly implemented! Well, shame on me.
Thanks again. Your collection is quite handy.

[defor]

Originally Posted by leonard

lz4_small.asm etc are working on raw LZ4 blocks, and not on LZ4 file header. To work with LZ4 file header ( so what lz4.exe produces ), just include https://github.com/arnaud-carre/lz4-.../lz4_frame.asm

As the comment says, it depacks data packed with this exact command line:

; Depack data produced by LZ4.exe command line:
; lz4.exe -9 --no-frame-crc <input_file> <output_file>

https://github.com/arnaud-carre/lz4-68k#lz4-frame

Hope it helps!

Yes, at last I noticed my mistake when browsing through Antiriad's code. I was so focused on fastest/normal/smallest source code files that I completely missed importance of lz4_frame.s!
Thank you all for helping me.