The new Scoopex Cruncher for the Amiga!

[Photon]

Scoopex Cruncher V1.0!

Just finished a small and simple, but nice cruncher. I made it mainly for oldskool demo coders (i.e., myself ) but it should work on 'all' Amigas.

It crunches well and decrunches _very_ fast. For anyone (anyone? hello?) coding games or demos for the classic 512K chip A500, it's grrreat! It uses only a couple of bytes for buffers and leaves the rest of the precious memory free, so you can utilize memory from $20000-7E000 at least. Probably more like $1F000-7FD00. (A 512K chip A500 with two drives.) And unlike many other crunchers, it can crunch big files on a 512K machine too, it only steals about 7k of the available memory.

Enough blowing my own horn (toot!), if you're a demo coder, check it out here:

http://coppershade.org/amiga/files/scxcrunch.lha

The program is of course free, as is the decrunch source, if you want to customize it. If you think it's good and want to write a hunk parser for it (masochist!! ), contact me! I don't see why it couldn't be adapted to modern Amigas.

[AmiGer]

It seems that the first unpacked byte is messed up (see example).
It has to be 48 not e8 in my case !?

- at least disable the interrupts before decrunching to prevent crashes
- make the source/assembler case sensitive (.jumptbl)

[girv]

@Photon: what algorithm do you use? Just curious - data compression/encryption is a pet interest of mine (sad but true )

[Photon]

Originally Posted by AmiGer

It seems that the first unpacked byte is messed up (see example).
It has to be 48 not e8 in my case !?

Grr... mumble... I thought I fixed that, because I had that problem, corrected it, and the decrunched file worked Will correct it.

Originally Posted by AmiGer

- at least disable the interrupts before decrunching to prevent crashes

That would be the safe way to start a "normal" demo, yes. BUT:
1) what if the decrunched program wants to have interrupts on, for example load a file of its own?
2) what if the decrunched program disables interrupts itself? I seem to remember two disables in a row would cause a Guru, but I may be wrong, please advise in that case.
3) if a Disable() is necessary because the deruncher overwrites low memory, it will hang on exit instead.

If you have a good idea to make it perfect, please suggest it!

Originally Posted by AmiGer

- make the source/assembler case sensitive (.jumptbl)

I personally think case sensitive compilers/assembler are spawns of the brainchild of Satan, but it's easy to fix, so I'll fix it.

Cool, thanks for the suggestions!

[Photon]

Originally Posted by girv

@Photon: what algorithm do you use? Just curious - data compression/encryption is a pet interest of mine (sad but true )

It's basically a block/pattern recognizer. Most such crunchers can't compete with LZW type algorithms but this one has special optimizations. It scans the whole file first to find how frequent each byte value is. Then it uses 12 special tokens to denote a crunched sequence of bytes. These tokens work like semi-intelligent code, decrunching its own pattern or block. The "intelligence" is applied when crunching, that's why it's almost as slow as the Titanics cruncher (right now at least).

By 'special optimizations' I mean, when I decided what these tokens would be, I took stats of files and studied them. I had to throw out a few tokens that I thought "these tokens will gain lots of bytes for sure!", when it turned out they didn't! The stats also told me how to balance the "bytes gained/bytes used in output file" ratio.

Right now the cruncher uses a scan range (for patterns) of up to 2048 bytes. Maybe a few bytes could be shaved off if >4-byte patterns are found far away, in that case it's easy to add a token. But I would have to implement the pointertable speedup first, since scanning a 350K file with a range of 65536 bytes takes *forever*!

I think the reason I get better results than other block/pattern crunchers is because I took those stats and used them to save BITS too, not just bytes. The bits are of course saved in the output of the crunched file, I don't go into the bit level when I'm reading the file!


I would love to get some comparisons of crunched file ratios, since I can't run all crunchers on my setup! It's made to auto-decrunch executables (like Imploder), not data files, so it would probably not be as good for crunching .IFF files and the like, compared to LHA, for example. But I haven't compared many data files yet, I'm coding on my demo now.

[AmiGer]

Well, I assume that you decrunch the program to a fixed address and you don't want to return to the OS because you don't know what you're overwriting in the memory. So it would be better to disable all interrupts during decrunching to prevent a crash.
If you still want to use OS functions later in the code I would do a correct AllocMem and would decrunch the (hopefully relocatable) program to this address.

Originally Posted by Photon

1) what if the decrunched program wants to have interrupts on, for example load a file of its own?.

decide at the beginning:
- use OS-functions and code the "official" (boring) way with AllocMem and so on...
- program the hardware directly like I do... and possibly make a custom disk loader...

2) what if the decrunched program disables interrupts itself? I seem to remember two disables in a row would cause a Guru, but I may be wrong, please advise in that case..

Don't know. You could also disable/enable interrupts by tweaking the custom chip registers directly. Look at the other thread "Taking over the system"...

[Photon]

But what would the WHDLoad guys say??!?

Frog wanted me to make something OS-friendly, so I'd like to make it as friendly as a can. But a demo is a demo... it's supposed to use the machine fully... if it needs all memory from $c0 to $7ffff it's not gonna be a pussy and ask pretty please for it. But if it doesn't, you'd like a clean exit.

I think I'll detect if the decrunch start and end address in within the chip chunk that I allocated for it, _before_ decrunching. If it is, fine, keep all OS-stuff and don't Disable(). Otherwise the decruncher will Disable(), decrunch, start the demo, and on exit, do a reset. I think that's a good solution.

[Photon]

BUG EXTERMINATED!!

Wasn't too hard to fix.

New archive file at:

http://student.hcedu.net/ws03hen/Amiga/scxcrunch.lha


I even managed to save 4 bytes in the decruncher!

[andreas]

Originally Posted by Photon

Grr... mumble... I thought I fixed that, because I had that problem, corrected it, and the decrunched file worked Will correct it.

Ehh .. why not make it an OPTION?
Otherwise the next one's gonna ask and moan about the case sensitivity again

[Photon]

He has a case sensitive assembler and I "missed" a capital J and a capital T in one of the labels. It really wasn't that hard to fix so now it works in both case sensitive assemblers and proper ones

[AmiGer]

	moveq #0,d0
	moveq #256/8-1,d6
.filll:	move.l d0,(a3)+
	move.l d0,(a3)+
	dbf d6,.filll

you should take a look at the moveq commands in the decrunch routine, its better to use moveq.l, otherwise you could get a problem (.w is the deault I think).

Is it right, that you need just 256 additional Bytes behind the destination memory ?

[AmiGer]

I tried your new version but I have still the same problem with the first damaged byte.

[Photon]

Originally Posted by AmiGer

	moveq #0,d0
	moveq #256/8-1,d6
.filll:	move.l d0,(a3)+
	move.l d0,(a3)+
	dbf d6,.filll

you should take a look at the moveq commands in the decrunch routine, its better to use moveq.l, otherwise you could get a problem (.w is the deault I think).

Is it right, that you need just 256 additional Bytes behind the destination memory ?

Moveq has no size. It is always longword size. So if you have any moveq.w or moveq.l in your sources you should correct them

And yes, I only need 256 bytes because the JumpTbl is so small that a +/-128 byte offset is enough.

About the cruncher: It works perfectly for me! I tried both IFF files and executables. I will do some more tests, if you check maybe your browser cache so you didn't get the old file again? The file SCXcrunch should be 6032 bytes long and SCXdecrunch.S should be case sensitive.

Thanks for the feedback anyh00!

[AmiGer]

Yes, of course you're right. moveq is always 32-bit. Funny that the assembler accepts moveq.w and moveq.b !?

I downloaded the zip again and now it changed.
Will do some tests and give feedback soon...

[AmiGer]

...runs perfectly now. I have built this small wrapper to alloc and free memory, works fine.

Good work, Photon !

_LVOAllocMem		equ	-198
_LVOFreeMem		equ	-210

MEMF_PUBLIC		equ	0
MEMF_CHIP		equ	1<<1
MEMF_FAST		equ	1<<2
MEMF_CLEAR		equ	1<<16 

oDataLen=0
oCrunchLen=4
oJumpAddr=8
oTokens=12
oData=28

	SECTION	Startup, CODE, CHIP
	
	movem.l	d2-d7/a2-a6,-(sp)
	lea 	StartOfData(pc),a5	
	move.l	(a5),d0
	add.l	#256,d0
	lea	DataLen(pc),a0
	move.l	d0,(a0)
	move.l	#MEMF_CLEAR!MEMF_CHIP,d1
	move.l	4.w,a6
	jsr	_LVOAllocMem(a6)
	tst.l	d0
	beq	MNoMem
	lea	_Data(pc),a0
	move.l	d0,(a0)
	move.l	d0,oJumpAddr(a5)
	
	bsr	DeCrunchIt

	move.l	_Data(pc),a1
	move.l	DataLen(pc),d0
	move.l	4.w,a6
	jsr	_LVOFreeMem(a6)
	
MNoMem	movem.l	(sp)+,d2-d7/a2-a6
	moveq.l	#0,d0
	rts

_Data	dc.l	0
DataLen	dc.l	0

[AmiGer]

Just another thought: if you add a "(pc)" to this line

lea StartOfData,a5

the decrunch routine is fully relocatable.

I also changed the decrunch colour to background because some machines doesn't display anything when booting. Hope this isn't contrary to any copyrights ?

[Photon]

No, the decrunch source is totally free, do with it what you please.

I guess a (PC) wouldn't hurt, though the loaded data is very seldom right after the decrunch routine but either in another (data) hunk or loaded from disk tracks to some faraway place. I've added it now anyway.

[Photon]

Just finished V1.01 of the cruncher. Small optimizations and a bug fix, enjoy!

I put it in The Zone since my hosting seems to be down atm.

[Photon]

Got a request for the file, it's on Coppershade.org

[cdoty]

Originally Posted by AmiGer

Yes, of course you're right. moveq is always 32-bit. Funny that the assembler accepts moveq.w and moveq.b !?

He he, I just ran across this same thing less than a month ago, while working on the CD-I. The OS-9 assembler also accepted it.