RNC Propack "in place" decompression does not always work

[roondar]

Originally Posted by alpine9000

I’ve used RNC in place for thousands of different files over the past couple of years and have yet to have one that doesn’t work. Are you guys sure you have enough free stack space?

Yes, definitely. I tested this extensively because originally I couldn't believe it didn't work. It's not a stack problem, it's an RNC problem. Note that it only happened to me on one specific file. All others I've ever used worked without a hitch.

[alpine9000]

Originally Posted by roondar

Yes, definitely. I tested this extensively because originally I couldn't believe it didn't work. It's not a stack problem, it's an RNC problem. Note that it only happened to me on one specific file. All others I've ever used worked without a hitch.

I must just be lucky, I can’t even really fathom how many different files I must have packed as in addition to all the assets my engine also packs my executable code for every build I do, which would be in the thousands for Metro Siege.

Can you send me the file that fails?

[roondar]

Originally Posted by alpine9000

I must just be lucky, I can’t even really fathom how many different files I must have packed as in addition to all the assets my engine also packs my executable code for every build I do, which would be in the thousands for Metro Siege.

Can you send me the file that fails?

It could be that I just got really unlucky
Anyway, sorry, I don't have that file any more - it's long been overwritten by a newer version

[ross]

Could it be only bad unpacker code and not the data intself? (see message on the different decompressor in Heimdall).
Could it be that only one of the two versions of RNC is affected?
However, I seem to remember that in the past I also had a file that failed, but that happened too long ago, so I can't provide any precise information or test file.

In any case, I haven't used RNC for a long time.
Depending on my needs, I use doynamite, nrv2x or arjm7.

[Giants]

And you don't have again the source of your nrv2x cruncher ?
a 68000 version of your cruncher will be... great

I don't need 'source' of cruncher, et's say i would prefer to crunch my file directly on amiga.
So juste 'need' a nrv2x 68000 binary.

I like the pc but if I can do all the steps on amiga, it's better. (for me)

[ross]

No, sorry.
Only available for Win platform

[Muzza]

Originally Posted by ross

I do not find the nrv2r, but I'm pretty sure I've posted it somewhere, so here the generic one, extract the version you need:
Compressor from a previous link in github (only the exe).

Hey Ross, is this the latest version of nrv2r?
I've been trying it for in-place decompression and it generally works great but I've come across a file that the decompressor fails on.

Problem file here.

I'm compressing it using: nrv2x.exe -es -k -f -o compressed.dat problem.dat
It gives the output: load encoded file at buffer offset 0x0000355A

In my code, I allocate 26040 bytes (original file size). I load the compressed file into this buffer + 0x355A.
Then I call nrv2x_decoder with the buffer pointer and an offset of 0x355A.
Comparing the output, it is trashing just the last 5 bytes. The end of file is changes from:

38 3E 00 00 00 10
to:
38 38 01 e1 b2 00

Using the .exe to decompress works fine, so it seems to be just the Amiga decompressor. Could I be doing something wrong, or is this a possible bug?

[DanScott]

Are you sure that the compressed file size + 0x355A, is not exceeding the 26040 bytes you allocated ?

[ross]

Originally Posted by Muzza

Hey Ross, is this the latest version of nrv2r?
I've been trying it for in-place decompression and it generally works great but I've come across a file that the decompressor fails on.

Problem file here.

I'm compressing it using: nrv2x.exe -es -k -f -o compressed.dat problem.dat
It gives the output: load encoded file at buffer offset 0x0000355A

In my code, I allocate 26040 bytes (original file size). I load the compressed file into this buffer + 0x355A.
Then I call nrv2x_decoder with the buffer pointer and an offset of 0x355A.
Comparing the output, it is trashing just the last 5 bytes. The end of file is changes from:

38 3E 00 00 00 10
to:
38 38 01 e1 b2 00

Using the .exe to decompress works fine, so it seems to be just the Amiga decompressor. Could I be doing something wrong, or is this a possible bug?

Yep, in-place decompression fail for this file and from some quick checks I think I understand where the problem lies.

nrv2x.exe -es -k -f -o compressed.dat problem.dat

nrv2x v2.3, by ross (68k raw data target)
compressing file "problem.dat"...

upx raw data start at 516, 11381 tokens found
11370 tokens used for the new encoding, 99008 raw bits
refactoring for in-place decoding... successful
11369 in-place tokens, 1 from stack, 99047 raw bits

input size: 26040 bytes
output size: 12381 bytes (padded to 12382 bytes)
load encoded file at buffer offset 0x0000355A

If I enlarge the buffer by 1 byte:

buffer
        ds.b    26040-12382
        ds.b    1
        incbin  compressed.dat

and I change the unpacker input to:

        lea buffer,a0
        move.l  #$355a+1,d0

unpacked data is right.

This:

(padded to 12382 bytes)

and this:

99047 raw bits

==

(12381*8-1...)

makes me think that I have miscalculated the padding by 1 bit somewhere and therefore it would be enough to simply shift the stream..

Please try the attached compressed.dat, load it at 0x355a (in the same buffer you have) but pass 0x355b to unpacker.

If it works I will fix it in the next version
Thanks!

PS:
nrv2x behaves really well with this file.
Your raw data zipped in problem.zip is 14226 bytes, output from nrv2x is 12381 bytes!

[ross]

Originally Posted by Muzza

Using the .exe to decompress works fine, so it seems to be just the Amiga decompressor. Could I be doing something wrong, or is this a possible bug?

ah!, just noticed that the internal dummy '68k emulator' detect it (-t option)

nrv2x.exe -es -k -f -t -o compressed.dat problem.dat
...
decoding overflow, please report

Of course I tried hundreds of files but let alone one that made the problem explicit..
Luckily I left the test code inside

And it's just a very rare in-place refactoring bitcode problem, nothing about the general encoding or the decompression code.

[Muzza]

Thanks Ross, I can confirm that with your new file and passing in 0x355B to the unpacker, it works for me.

[a/b]

Originally Posted by Muzza

38 3E 00 00 00 10

This brings back memories. It looks like you have figured out to replace the original 32-bit (or even longer, it's been awhile) match offset terminator with a 16-bit one :P. I remember my converter creating a few bytes shorter files than reported here http://eab.abime.net/showthread.php?t=89467 because of that.

UPX compressor likes to create quite some junk at the end, like a literal(!) that's too long, unneeded match, and then finally a match offset terminator. WTH, why not just using a shorter literal, completely skipping the bogus match, and go straight for the terminator?!? Lots of small things to check for no apparent reason .

[ross]

Originally Posted by a/b

This brings back memories. It looks like you have figured out to replace the original 32-bit (or even longer, it's been awhile) match offset terminator with a 16-bit one :P. I remember my converter creating a few bytes shorter files than reported here http://eab.abime.net/showthread.php?t=89467 because of that.

UPX compressor likes to create quite some junk at the end, like a literal(!) that's too long, unneeded match, and then finally a match offset terminator. WTH, why not just using a shorter literal, completely skipping the bogus match, and go straight for the terminator?!? Lots of small things to check for no apparent reason .

Eheh, a lot has changed from the old reported thread, check the decoder

But Muzza trashed bytes regard a damn bit that collides and is overwritten by the in-place decompression
(due to an unfortunate combination of events, bitcoding is fully right).

So I used, in the bugfix release, 1 more token from the stack (i.e. it is no longer directly in the main stream but pre-cached).
At the end the file size is (bit) identical, the decompression offset is identical but the bits/byte are shuffled differently.

[StingRay]

A bit of thread necromancy, but for a reason!

When working on my Wrath of the Demon release I encountered a problem which was caused by the Rob Norten decruncher. I am referring to the "Mode 2" decruncher which supports decrunching to overlapping source/destination areas.

Sometimes, there was graphics trash on screen and I always suspected that it had to do with the ProPack decruncher and after a lot of debugging I finally tracked it down and it indeed was entirely the fault of the decruncher. When the source and destination memory areas overlap, the decruncher temporarily trashes a few bytes (2-4 I think) at the end of the destination area. Once decrunching is done, these trashed bytes will be restored. So far, so (not) good! Because this behaviour can lead to hard to find problems as in the case of WOTD. Directly after the decrunched data, a copperlist followed. Now, if this copperlist was active during decrunching (game uses more than one copperlist in this level and switches them each frame), the first 2 bytes in the copperlist were trashed which in turn caused the screen corruption. If the other copperlist was active, everything was fine. Very annoying problem which took me quite some hours to track down due to the "random" nature.

Another case was the game Blastar, here the behaviour of the decruncher caused writes outside the 512k chip memory boundary which isn't exactly nice either.

Long story short: If you use the ProPack decruncher with overlapping memory areas, make sure you have a few bytes safety space directly after the end of the decunched data or prepare for trouble!

My final conclusion is: "Thank you" Rob Northen...

[ross]

Originally Posted by StingRay

..I encountered a problem which was caused by the Rob Norten decruncher.

Yep, I also noticed this damn bug.
I solved it in the most drastic way in my version of WotD: completely removed RNC from it (nrv2x instead)

This has brought several advantages, which sooner or later I will make public.

[StingRay]

Originally Posted by ross

Yep, I also noticed this damn bug.
I solved it in the most drastic way in my version of WotD: completely removed RNC from it (nrv2x instead)

I actually tried your nrv2x too but due to time constraints I didn't redo the whole game using nrv2x instead of ProPack.