crunchers with c and asm sources

[xxxxx]

Hey guys, I'm looking for a data cruncher for my current project, and I'm hoping to not write one from scratch. Do you know if there are some good ones out there that:
1. have c source for the cruncher (so I can run it on PC)
2. have asm source for decruncher, so it runs on an unaccelerated a500
3. deals with raw data, not amiga hunks
4. is fast at decrunching

Have any of you ever used (or seen) any code that can help with this?

[StingRay]

Back in the day of PSX demo coding Gargoyle/Polka Brothers (if memory serves me right) ported ByteKiller to C. I should still have the source somewhere if that helps you? Decruncher in assembler is not a problem either of course. If you can live with the (nowadays) poor crunching efficiency it should be exactly what you need.

[WayneK]

There was also a C port of the RNC mode 1 decruncher (and there is a PC native dos exe for propack to crunch your data), which was quite a good packer although perhaps not fast enough decrunch for you. I believe the C port was intended for the Nintendo64 scene - if you can find a mirror of the old "Dextrose" website you might track it down.
Also there is a short/fast lz77 and lz78 implementation by Ray/TsCC which comes with 680x0 decrunch and C source.

[Mrs Beanbag]

I have asm code for my own LZ-variant I wrote for Mr Beanbag, I recently wrote a much faster compressor but I haven't got round to writing the decoder yet :/

[xxxxx]

the Bytekiller c source would be great. Could you post it? I remember Bytekiller from back in the day, I may even have used it.
The Ray/TsCC lz77 code is also an option, thanks for the tip.

It sounds like both of these have been used in multiple productions, so I will look at them first.

I looked at RNC before, but I couldn't find source for the cruncher, only for the decruncher. I didn't want to rely on an old dos program for crunching :-)

thanks for the help

[hitchhikr]

Source for a RNC cruncher can be found here: http://www.yoda.arachsys.com/dk/utils.html

[phx]

Originally Posted by xxxxx

the Bytekiller c source would be great. Could you post it?

I had published my Bytekiller C source, which I wrote for Sqrxz, some time ago. Here:

http://eab.abime.net/showthread.php?t=65683

[xxxxx]

thanks again, you guys rock :-)

[jotd]

Hi,

I can't make RNC from Dungeon Keeper Small Utilities work

It doesn't run natively, so I tried running it on DOSBox, but I get the usage each time, or "internal fault", whatever the options I'm trying.

I checked the source code, and it seems that "mode" isn't initialized. So I recompiled, and after fixing the "mode" stuff, I got "unable to handle leeway > 255"...

Sounds that this one is very broken. Anyone could make that work? it seems to be the only program able to compress RNC files with C source

(a long time ago I tried to make it work, and I could compress some files but on some other files compression/decompression failed so the program is not portable to 32/64 or even gcc I suppose)

[alpine9000]

https://github.com/AxisOxy/Planet-Ro...s/doynamite68k

This has a portable C compressor with asm depacker.

[jotd]

Thanks, bit I want to unpack/pack for RNC because of a project involving a classic game which uses RNC (same problem as with SWOS team editor)

[Foul]

http://aminet.net/package/util/pack/cranker ?

[jotd]

Hi,

Just rebuilt the RNC/DERNC from Dungeon Keeper Utilities (in the zone) for windows (sources included)

I have fixed the command line (well, at least RNC "file" works, -o doesn't, and it didn't even with the original exe on dosbox: never could make the options work) & a few other things.

- dernc: rnc decompressor: works great
- rnc: rnc compressor: decompresses OK with dernc or xfddecrunch. I still had crashes with some games, so maybe there's a limitation with some game decrunchers (I know there's a CRC which isn't there in some cases, but that doesn't seem to be that)

I'll let you test it.

[temisu]

Originally Posted by jotd

- rnc: rnc compressor: decompresses OK with dernc or xfddecrunch. I still had crashes with some games, so maybe there's a limitation with some game decrunchers (I know there's a CRC which isn't there in some cases, but that doesn't seem to be that)
I'll let you test it.

Dungeon keeper RNC-utility only support 1 out of the 3 different RNC-subtypes, so it is not really a generic solution. However, it supports the most popular one though

Most (i.e. not the old RNC1) of RNC1/RNC2 files do have CRC for both packed and unpacked data thus verifying RNC-files should be straightforward ... However, in my tests I have seen unexpected CRC-failures and concluded that there can only be modded implementations used in many places

[jotd]

I think that the CRC is ignored most of the time. It's useful to tell between old and new RNC1: if CRC fails, then it's not CRC it's RNC1. WHDLoad & CD32load use that property to know which unpacking method to use with RNC1 headers.

The issue I have with this RNC packing code is that it doesn't work with asm unpackers from games (Magic Pockets PC & Amiga version crash when I feed it with this packed file). XFDdecrunch has no issue depacking it, though. Any thoughts?

[temisu]

Originally Posted by jotd

I think that the CRC is ignored most of the time. It's useful to tell between old and new RNC1: if CRC fails, then it's not CRC it's RNC1. WHDLoad & CD32load use that property to know which unpacking method to use with RNC1 headers.

I don't solely believe using CRC to detect the different versions. For me it is the last step, since there is element of error built into that (not to mention calculating CRC of a file is rather heavy method to detect a format)

This is what I use:

	if (hdr==FourCC('RNC\001'))
	{
		// now detect between old and new version
		// since the old and the new version share the same id, there is no foolproof way
		// to tell them apart. It is easier to prove that it is not something by finding
		// specific invalid bitstream content.

		// if we can't access the old-stream we are nothing (since old stream is smaller the new stream by header 12 vs. 18)
		uint8_t oldStreamStart,newStreamStart;
		uint16_t hdrCrc,crc;
		if (!packedData.read(_packedSize+11,oldStreamStart)) return;

		// well, this is silly though but lets assume someone has made old format RNC1 with total size less than 19
		if (!packedData.read(18,newStreamStart))
			_ver=Version::RNC1Old;

		// Check that stream starts with a literal(s)
		else if (!(oldStreamStart&0x80))
			_ver=Version::RNC1New;

		// New stream have two bits in start as a filler on new stream. Those are always 0
		// (although this is not strictly mandated)
		// +
		// Even though it is possible to make new RNC1 stream which starts with zero literal table size,
		// it is extremely unlikely
		else if ((newStreamStart&3) || !(newStreamStart&0x7c))
			_ver=Version::RNC1Old;

		// now the last resort: check CRC.
		else if (!packedData.readBE(14,hdrCrc)) return;
		else if (RNCCRC(_packedData,18,_packedSize,crc) && crc==hdrCrc)
			_ver=Version::RNC1New;
		else
			_ver=Version::RNC1Old;
	} else if (hdr==FourCC('RNC\002')) {
		_ver=Version::RNC2;

Excuse me for the if-else-hell, there is no clean way to write it

Originally Posted by jotd

The issue I have with this RNC packing code is that it doesn't work with asm unpackers from games (Magic Pockets PC & Amiga version crash when I feed it with this packed file). XFDdecrunch has no issue depacking it, though. Any thoughts?

Do you have example files? I could peek in those what they are made of

[jotd]

> Do you have example files? I could peek in those what they are made of

I was hoping you'd say that

I zoned "rnc_files.zip" for you. There are 3 files in it:

- NEW1.PIN: the file packed with rnc (dungeon utilities)
- NEW1_orig.PIN: original file packed with "official" propack
- NEW1_orig_unp.PIN: unpacked file

I hope you can find why it fails. Do you have any RNC cruncher source handy? You've shown some nice C++ code for RNC unpack, maybe you have the pack side too and want to share?

[temisu]

Originally Posted by jotd

- NEW1.PIN: the file packed with rnc (dungeon utilities)

Well, I know why it fails. The dungeon keeper rnc has only compressed 16384 of the 16662 bytes of the file properly. There is one more chunk and remaining bytes are there just fine but that chunk is out of bounds. In short chunk count is one less what it should be. Not many decompressors check this and just proceed until the end...

Happy news is that my own decompressor actually detected this just fine

Originally Posted by jotd

I hope you can find why it fails. Do you have any RNC cruncher source handy? You've shown some nice C++ code for RNC unpack, maybe you have the pack side too and want to share?

Unfortunately no. But I can check what is wrong in dungeon keeper RNC. Probably this very easy to fix now when we now the root cause.

[temisu]

I got it working by adding this (your luck might be different, the rnc seems to be very sensitive to compilers...)

Here is a diff (to the original sources). In theory the chunk count should not be filled at all, but I can see something is there...

--- utilsrc/rnc.c
+++ utilsrc_mod/rnc.c
@@ -194,6 +161,8 @@
     bitbuf = 0;
     write_bits (0, 2);
 
+    int chunks;
+    chunks = 0;
     while (datalen > 0) {
        blklen = datalen > BLOCKMAX ? BLOCKMAX : datalen;
        blkstart = WINMAX - BLOCKMAX;
@@ -209,6 +178,7 @@
        data += bpos - blkstart;
        datalen -= bpos - blkstart;
        write_block();
+       chunks++;
     }
 
     if (bitcount > 0) {
@@ -222,7 +192,8 @@
     bwrite (packed+12, rnc_crc(packed+18, packedpos-18));
     bwrite (packed+10, rnc_crc(original, origlen));
     bwrite (packed+8, packedpos-18);
-    packed[16] = packed[17] = 0;
+    packed[16] = 0;
+    packed[17] = chunks;
 
     return packed;
 }

[jotd]

that's it!! now it decrunches properly and doesn't crash anymore.

The funny part is that I figured that out when making a diff between original & DK crunched files, but somehow I must have zapped it or changed the value in the binary and overwritten with a bad file again.

That's the first RNC packer with source freely available that works. To reduce the risk of compiler abuse, I fixed the "int" as 16-bit values (like the original compiler did). There was also a lot of warnings & undefined variables. Now it works fine, thanks a million.

Source & exes in the zone. Will put that on my website too.