English Amiga Board


Go Back   English Amiga Board > Coders > Coders. General

 
 
Thread Tools
Old 30 November 2012, 09:23   #1
xxxxx
Registered User
 
Join Date: Jan 2012
Location: N/A
Posts: 38
crunchers with c and asm sources

Hey guys, I'm looking for a data cruncher for my current project, and I'm hoping to not write one from scratch. Do you know if there are some good ones out there that:
1. have c source for the cruncher (so I can run it on PC)
2. have asm source for decruncher, so it runs on an unaccelerated a500
3. deals with raw data, not amiga hunks
4. is fast at decrunching

Have any of you ever used (or seen) any code that can help with this?
xxxxx is offline  
AdSense AdSense  
Old 30 November 2012, 09:28   #2
StingRay
move.l #$c0ff33,throat

StingRay's Avatar
 
Join Date: Dec 2005
Location: Berlin/Joymoney
Posts: 5,572
Back in the day of PSX demo coding Gargoyle/Polka Brothers (if memory serves me right) ported ByteKiller to C. I should still have the source somewhere if that helps you? Decruncher in assembler is not a problem either of course. If you can live with the (nowadays) poor crunching efficiency it should be exactly what you need.
StingRay is online now  
Old 30 November 2012, 13:07   #3
WayneK
Registered User
 
Join Date: May 2004
Location: Somewhere secret
Age: 43
Posts: 241
There was also a C port of the RNC mode 1 decruncher (and there is a PC native dos exe for propack to crunch your data), which was quite a good packer although perhaps not fast enough decrunch for you. I believe the C port was intended for the Nintendo64 scene - if you can find a mirror of the old "Dextrose" website you might track it down.
Also there is a short/fast lz77 and lz78 implementation by Ray/TsCC which comes with 680x0 decrunch and C source.
WayneK is offline  
Old 30 November 2012, 15:55   #4
Mrs Beanbag
Glastonbridge Software
Mrs Beanbag's Avatar
 
Join Date: Jan 2012
Location: Edinburgh/Scotland
Posts: 2,202
I have asm code for my own LZ-variant I wrote for Mr Beanbag, I recently wrote a much faster compressor but I haven't got round to writing the decoder yet :/
Mrs Beanbag is offline  
Old 30 November 2012, 18:49   #5
xxxxx
Registered User
 
Join Date: Jan 2012
Location: N/A
Posts: 38
the Bytekiller c source would be great. Could you post it? I remember Bytekiller from back in the day, I may even have used it.
The Ray/TsCC lz77 code is also an option, thanks for the tip.

It sounds like both of these have been used in multiple productions, so I will look at them first.

I looked at RNC before, but I couldn't find source for the cruncher, only for the decruncher. I didn't want to rely on an old dos program for crunching :-)

thanks for the help
xxxxx is offline  
Old 30 November 2012, 19:28   #6
hitchhikr
Registered User

 
Join Date: Jun 2008
Location: somewhere
Posts: 360
Source for a RNC cruncher can be found here: http://www.yoda.arachsys.com/dk/utils.html
hitchhikr is offline  
Old 30 November 2012, 20:18   #7
phx
Natteravn

phx's Avatar
 
Join Date: Nov 2009
Location: Herford / Germany
Posts: 1,004
Quote:
Originally Posted by xxxxx View Post
the Bytekiller c source would be great. Could you post it?
I had published my Bytekiller C source, which I wrote for Sqrxz, some time ago. Here:

http://eab.abime.net/showthread.php?t=65683
phx is offline  
Old 01 December 2012, 06:14   #8
xxxxx
Registered User
 
Join Date: Jan 2012
Location: N/A
Posts: 38
thanks again, you guys rock :-)
xxxxx is offline  
Old 26 May 2017, 20:25   #9
jotd
Cat freak
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 46
Posts: 1,934
Hi,

I can't make RNC from Dungeon Keeper Small Utilities work

It doesn't run natively, so I tried running it on DOSBox, but I get the usage each time, or "internal fault", whatever the options I'm trying.

I checked the source code, and it seems that "mode" isn't initialized. So I recompiled, and after fixing the "mode" stuff, I got "unable to handle leeway > 255"...

Sounds that this one is very broken. Anyone could make that work? it seems to be the only program able to compress RNC files with C source

(a long time ago I tried to make it work, and I could compress some files but on some other files compression/decompression failed so the program is not portable to 32/64 or even gcc I suppose)
jotd is offline  
Old 27 May 2017, 02:21   #10
alpine9000
Registered User

 
Join Date: Mar 2016
Location: Australia
Posts: 361
https://github.com/AxisOxy/Planet-Ro...s/doynamite68k

This has a portable C compressor with asm depacker.
alpine9000 is offline  
Old 27 May 2017, 09:43   #11
jotd
Cat freak
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 46
Posts: 1,934
Thanks, bit I want to unpack/pack for RNC because of a project involving a classic game which uses RNC (same problem as with SWOS team editor)
jotd is offline  
Old 27 May 2017, 16:54   #12
Foul
Registered User

Foul's Avatar
 
Join Date: Jun 2009
Location: Perigueux/France
Age: 43
Posts: 1,263
Send a message via ICQ to Foul Send a message via MSN to Foul Send a message via Skype™ to Foul
http://aminet.net/package/util/pack/cranker ?
Foul is offline  
Old 29 May 2017, 23:19   #13
jotd
Cat freak
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 46
Posts: 1,934
Hi,

Just rebuilt the RNC/DERNC from Dungeon Keeper Utilities (in the zone) for windows (sources included)

I have fixed the command line (well, at least RNC "file" works, -o doesn't, and it didn't even with the original exe on dosbox: never could make the options work) & a few other things.

- dernc: rnc decompressor: works great
- rnc: rnc compressor: decompresses OK with dernc or xfddecrunch. I still had crashes with some games, so maybe there's a limitation with some game decrunchers (I know there's a CRC which isn't there in some cases, but that doesn't seem to be that)

I'll let you test it.
jotd is offline  
Old 30 May 2017, 10:22   #14
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 20
Quote:
Originally Posted by jotd View Post
- rnc: rnc compressor: decompresses OK with dernc or xfddecrunch. I still had crashes with some games, so maybe there's a limitation with some game decrunchers (I know there's a CRC which isn't there in some cases, but that doesn't seem to be that)
I'll let you test it.
Dungeon keeper RNC-utility only support 1 out of the 3 different RNC-subtypes, so it is not really a generic solution. However, it supports the most popular one though

Most (i.e. not the old RNC1) of RNC1/RNC2 files do have CRC for both packed and unpacked data thus verifying RNC-files should be straightforward ... However, in my tests I have seen unexpected CRC-failures and concluded that there can only be modded implementations used in many places
temisu is offline  
Old 30 May 2017, 12:19   #15
jotd
Cat freak
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 46
Posts: 1,934
I think that the CRC is ignored most of the time. It's useful to tell between old and new RNC1: if CRC fails, then it's not CRC it's RNC1. WHDLoad & CD32load use that property to know which unpacking method to use with RNC1 headers.

The issue I have with this RNC packing code is that it doesn't work with asm unpackers from games (Magic Pockets PC & Amiga version crash when I feed it with this packed file). XFDdecrunch has no issue depacking it, though. Any thoughts?
jotd is offline  
Old 30 May 2017, 12:34   #16
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 20
Quote:
Originally Posted by jotd View Post
I think that the CRC is ignored most of the time. It's useful to tell between old and new RNC1: if CRC fails, then it's not CRC it's RNC1. WHDLoad & CD32load use that property to know which unpacking method to use with RNC1 headers.
I don't solely believe using CRC to detect the different versions. For me it is the last step, since there is element of error built into that (not to mention calculating CRC of a file is rather heavy method to detect a format)

This is what I use:

Code:
	if (hdr==FourCC('RNC\001'))
	{
		// now detect between old and new version
		// since the old and the new version share the same id, there is no foolproof way
		// to tell them apart. It is easier to prove that it is not something by finding
		// specific invalid bitstream content.

		// if we can't access the old-stream we are nothing (since old stream is smaller the new stream by header 12 vs. 18)
		uint8_t oldStreamStart,newStreamStart;
		uint16_t hdrCrc,crc;
		if (!packedData.read(_packedSize+11,oldStreamStart)) return;

		// well, this is silly though but lets assume someone has made old format RNC1 with total size less than 19
		if (!packedData.read(18,newStreamStart))
			_ver=Version::RNC1Old;

		// Check that stream starts with a literal(s)
		else if (!(oldStreamStart&0x80))
			_ver=Version::RNC1New;

		// New stream have two bits in start as a filler on new stream. Those are always 0
		// (although this is not strictly mandated)
		// +
		// Even though it is possible to make new RNC1 stream which starts with zero literal table size,
		// it is extremely unlikely
		else if ((newStreamStart&3) || !(newStreamStart&0x7c))
			_ver=Version::RNC1Old;

		// now the last resort: check CRC.
		else if (!packedData.readBE(14,hdrCrc)) return;
		else if (RNCCRC(_packedData,18,_packedSize,crc) && crc==hdrCrc)
			_ver=Version::RNC1New;
		else
			_ver=Version::RNC1Old;
	} else if (hdr==FourCC('RNC\002')) {
		_ver=Version::RNC2;
Excuse me for the if-else-hell, there is no clean way to write it

Quote:
Originally Posted by jotd View Post
The issue I have with this RNC packing code is that it doesn't work with asm unpackers from games (Magic Pockets PC & Amiga version crash when I feed it with this packed file). XFDdecrunch has no issue depacking it, though. Any thoughts?
Do you have example files? I could peek in those what they are made of
temisu is offline  
Old 31 May 2017, 00:44   #17
jotd
Cat freak
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 46
Posts: 1,934
> Do you have example files? I could peek in those what they are made of

I was hoping you'd say that

I zoned "rnc_files.zip" for you. There are 3 files in it:

- NEW1.PIN: the file packed with rnc (dungeon utilities)
- NEW1_orig.PIN: original file packed with "official" propack
- NEW1_orig_unp.PIN: unpacked file

I hope you can find why it fails. Do you have any RNC cruncher source handy? You've shown some nice C++ code for RNC unpack, maybe you have the pack side too and want to share?
jotd is offline  
Old 31 May 2017, 08:45   #18
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 20
Quote:
Originally Posted by jotd View Post
- NEW1.PIN: the file packed with rnc (dungeon utilities)
Well, I know why it fails. The dungeon keeper rnc has only compressed 16384 of the 16662 bytes of the file properly. There is one more chunk and remaining bytes are there just fine but that chunk is out of bounds. In short chunk count is one less what it should be. Not many decompressors check this and just proceed until the end...

Happy news is that my own decompressor actually detected this just fine

Quote:
Originally Posted by jotd View Post
I hope you can find why it fails. Do you have any RNC cruncher source handy? You've shown some nice C++ code for RNC unpack, maybe you have the pack side too and want to share?
Unfortunately no. But I can check what is wrong in dungeon keeper RNC. Probably this very easy to fix now when we now the root cause.
temisu is offline  
Old 31 May 2017, 09:30   #19
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 20
I got it working by adding this (your luck might be different, the rnc seems to be very sensitive to compilers...)

Here is a diff (to the original sources). In theory the chunk count should not be filled at all, but I can see something is there...

Code:
--- utilsrc/rnc.c
+++ utilsrc_mod/rnc.c
@@ -194,6 +161,8 @@
     bitbuf = 0;
     write_bits (0, 2);
 
+    int chunks;
+    chunks = 0;
     while (datalen > 0) {
        blklen = datalen > BLOCKMAX ? BLOCKMAX : datalen;
        blkstart = WINMAX - BLOCKMAX;
@@ -209,6 +178,7 @@
        data += bpos - blkstart;
        datalen -= bpos - blkstart;
        write_block();
+       chunks++;
     }
 
     if (bitcount > 0) {
@@ -222,7 +192,8 @@
     bwrite (packed+12, rnc_crc(packed+18, packedpos-18));
     bwrite (packed+10, rnc_crc(original, origlen));
     bwrite (packed+8, packedpos-18);
-    packed[16] = packed[17] = 0;
+    packed[16] = 0;
+    packed[17] = chunks;
 
     return packed;
 }

Last edited by temisu; 31 May 2017 at 09:42.
temisu is offline  
Old 31 May 2017, 23:14   #20
jotd
Cat freak
jotd's Avatar
 
Join Date: Dec 2004
Location: FRANCE
Age: 46
Posts: 1,934
that's it!! now it decrunches properly and doesn't crash anymore.

The funny part is that I figured that out when making a diff between original & DK crunched files, but somehow I must have zapped it or changed the value in the binary and overwritten with a bad file again.

That's the first RNC packer with source freely available that works. To reduce the risk of compiler abuse, I fixed the "int" as 16-bit values (like the original compiler did). There was also a lot of warnings & undefined variables. Now it works fine, thanks a million.

Source & exes in the zone. Will put that on my website too.
jotd is offline  
AdSense AdSense  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
Crunchers that support overlays? BarryB support.Apps 9 07 March 2017 12:58
What Crunchers do you guys prefer? Plagueis/KRX Coders. General 67 31 July 2014 00:20
Old crunchers absence Coders. General 3 25 June 2012 19:47
ASM - Rolling Thunder sources - accurate delay Asman Coders. General 5 22 September 2010 00:15
Old crunchers for A500/WB1.3? Photon Coders. General 21 03 December 2004 03:14

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 00:26.


Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Page generated in 0.34000 seconds with 12 queries