English Amiga Board


Go Back   English Amiga Board > Coders > Coders. General

 
 
Thread Tools
Old 03 December 2004, 03:04   #1
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
The new Scoopex Cruncher for the Amiga!

Scoopex Cruncher V1.0!

Just finished a small and simple, but nice cruncher. I made it mainly for oldskool demo coders (i.e., myself ) but it should work on 'all' Amigas.

It crunches well and decrunches _very_ fast. For anyone (anyone? hello?) coding games or demos for the classic 512K chip A500, it's grrreat! It uses only a couple of bytes for buffers and leaves the rest of the precious memory free, so you can utilize memory from $20000-7E000 at least. Probably more like $1F000-7FD00. (A 512K chip A500 with two drives.) And unlike many other crunchers, it can crunch big files on a 512K machine too, it only steals about 7k of the available memory.

Enough blowing my own horn (toot!), if you're a demo coder, check it out here:

http://coppershade.org/amiga/files/scxcrunch.lha

The program is of course free, as is the decrunch source, if you want to customize it. If you think it's good and want to write a hunk parser for it (masochist!! ), contact me! I don't see why it couldn't be adapted to modern Amigas.

Last edited by Photon; 11 September 2007 at 17:22. Reason: updated download url
Photon is offline  
AdSense AdSense  
Old 03 December 2004, 10:05   #2
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
It seems that the first unpacked byte is messed up (see example).
It has to be 48 not e8 in my case !?

- at least disable the interrupts before decrunching to prevent crashes
- make the source/assembler case sensitive (.jumptbl)
Attached Thumbnails
Click image for larger version

Name:	debug.jpg
Views:	388
Size:	72.3 KB
ID:	7652  
AmiGer is offline  
Old 03 December 2004, 16:53   #3
girv
Reticulator of Splines
girv's Avatar
 
Join Date: Aug 2004
Location: Northern Ireland
Posts: 920
Send a message via ICQ to girv Send a message via AIM to girv Send a message via MSN to girv Send a message via Yahoo to girv
@Photon: what algorithm do you use? Just curious - data compression/encryption is a pet interest of mine (sad but true )
girv is offline  
Old 03 December 2004, 19:56   #4
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
Quote:
Originally Posted by AmiGer
It seems that the first unpacked byte is messed up (see example).
It has to be 48 not e8 in my case !?
Grr... mumble... I thought I fixed that, because I had that problem, corrected it, and the decrunched file worked Will correct it.
Quote:
Originally Posted by AmiGer
- at least disable the interrupts before decrunching to prevent crashes
That would be the safe way to start a "normal" demo, yes. BUT:
1) what if the decrunched program wants to have interrupts on, for example load a file of its own?
2) what if the decrunched program disables interrupts itself? I seem to remember two disables in a row would cause a Guru, but I may be wrong, please advise in that case.
3) if a Disable() is necessary because the deruncher overwrites low memory, it will hang on exit instead.

If you have a good idea to make it perfect, please suggest it!

Quote:
Originally Posted by AmiGer
- make the source/assembler case sensitive (.jumptbl)
I personally think case sensitive compilers/assembler are spawns of the brainchild of Satan, but it's easy to fix, so I'll fix it.

Cool, thanks for the suggestions!
Photon is offline  
Old 03 December 2004, 20:22   #5
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
Quote:
Originally Posted by girv
@Photon: what algorithm do you use? Just curious - data compression/encryption is a pet interest of mine (sad but true )
It's basically a block/pattern recognizer. Most such crunchers can't compete with LZW type algorithms but this one has special optimizations. It scans the whole file first to find how frequent each byte value is. Then it uses 12 special tokens to denote a crunched sequence of bytes. These tokens work like semi-intelligent code, decrunching its own pattern or block. The "intelligence" is applied when crunching, that's why it's almost as slow as the Titanics cruncher (right now at least).

By 'special optimizations' I mean, when I decided what these tokens would be, I took stats of files and studied them. I had to throw out a few tokens that I thought "these tokens will gain lots of bytes for sure!", when it turned out they didn't! The stats also told me how to balance the "bytes gained/bytes used in output file" ratio.

Right now the cruncher uses a scan range (for patterns) of up to 2048 bytes. Maybe a few bytes could be shaved off if >4-byte patterns are found far away, in that case it's easy to add a token. But I would have to implement the pointertable speedup first, since scanning a 350K file with a range of 65536 bytes takes *forever*!

I think the reason I get better results than other block/pattern crunchers is because I took those stats and used them to save BITS too, not just bytes. The bits are of course saved in the output of the crunched file, I don't go into the bit level when I'm reading the file!


I would love to get some comparisons of crunched file ratios, since I can't run all crunchers on my setup! It's made to auto-decrunch executables (like Imploder), not data files, so it would probably not be as good for crunching .IFF files and the like, compared to LHA, for example. But I haven't compared many data files yet, I'm coding on my demo now.

Last edited by Photon; 03 December 2004 at 20:24. Reason: speling eror
Photon is offline  
Old 03 December 2004, 21:15   #6
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
Well, I assume that you decrunch the program to a fixed address and you don't want to return to the OS because you don't know what you're overwriting in the memory. So it would be better to disable all interrupts during decrunching to prevent a crash.
If you still want to use OS functions later in the code I would do a correct AllocMem and would decrunch the (hopefully relocatable) program to this address.

Quote:
Originally Posted by Photon
1) what if the decrunched program wants to have interrupts on, for example load a file of its own?.
decide at the beginning:
- use OS-functions and code the "official" (boring) way with AllocMem and so on...
- program the hardware directly like I do... and possibly make a custom disk loader...

Quote:
2) what if the decrunched program disables interrupts itself? I seem to remember two disables in a row would cause a Guru, but I may be wrong, please advise in that case..
Don't know. You could also disable/enable interrupts by tweaking the custom chip registers directly. Look at the other thread "Taking over the system"...
AmiGer is offline  
Old 03 December 2004, 21:33   #7
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
But what would the WHDLoad guys say??!?

Frog wanted me to make something OS-friendly, so I'd like to make it as friendly as a can. But a demo is a demo... it's supposed to use the machine fully... if it needs all memory from $c0 to $7ffff it's not gonna be a pussy and ask pretty please for it. But if it doesn't, you'd like a clean exit.

I think I'll detect if the decrunch start and end address in within the chip chunk that I allocated for it, _before_ decrunching. If it is, fine, keep all OS-stuff and don't Disable(). Otherwise the decruncher will Disable(), decrunch, start the demo, and on exit, do a reset. I think that's a good solution.
Photon is offline  
Old 06 December 2004, 00:30   #8
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
BUG EXTERMINATED!!

Wasn't too hard to fix.

New archive file at:

http://student.hcedu.net/ws03hen/Amiga/scxcrunch.lha


I even managed to save 4 bytes in the decruncher!
Photon is offline  
Old 07 December 2004, 22:03   #9
andreas
Zone Friend
 
Join Date: Jun 2001
Location: Germany
Age: 43
Posts: 5,858
Send a message via ICQ to andreas Send a message via AIM to andreas
Quote:
Originally Posted by Photon
Quote:
Originally Posted by AmiGer
- make the source/assembler case sensitive (.jumptbl)
Grr... mumble... I thought I fixed that, because I had that problem, corrected it, and the decrunched file worked Will correct it.
Ehh .. why not make it an OPTION?
Otherwise the next one's gonna ask and moan about the case sensitivity again
andreas is offline  
Old 07 December 2004, 23:31   #10
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
He has a case sensitive assembler and I "missed" a capital J and a capital T in one of the labels. It really wasn't that hard to fix so now it works in both case sensitive assemblers and proper ones
Photon is offline  
Old 15 December 2004, 08:56   #11
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
Code:
	moveq #0,d0
	moveq #256/8-1,d6
.filll:	move.l d0,(a3)+
	move.l d0,(a3)+
	dbf d6,.filll
you should take a look at the moveq commands in the decrunch routine, its better to use moveq.l, otherwise you could get a problem (.w is the deault I think).

Is it right, that you need just 256 additional Bytes behind the destination memory ?
AmiGer is offline  
Old 15 December 2004, 09:24   #12
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
I tried your new version but I have still the same problem with the first damaged byte.
Attached Thumbnails
Click image for larger version

Name:	scx.jpg
Views:	224
Size:	148.9 KB
ID:	7681  
AmiGer is offline  
Old 15 December 2004, 14:03   #13
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
Quote:
Originally Posted by AmiGer
Code:
	moveq #0,d0
	moveq #256/8-1,d6
.filll:	move.l d0,(a3)+
	move.l d0,(a3)+
	dbf d6,.filll
you should take a look at the moveq commands in the decrunch routine, its better to use moveq.l, otherwise you could get a problem (.w is the deault I think).

Is it right, that you need just 256 additional Bytes behind the destination memory ?
Moveq has no size. It is always longword size. So if you have any moveq.w or moveq.l in your sources you should correct them

And yes, I only need 256 bytes because the JumpTbl is so small that a +/-128 byte offset is enough.

About the cruncher: It works perfectly for me! I tried both IFF files and executables. I will do some more tests, if you check maybe your browser cache so you didn't get the old file again? The file SCXcrunch should be 6032 bytes long and SCXdecrunch.S should be case sensitive.

Thanks for the feedback anyh00!
Photon is offline  
Old 15 December 2004, 14:43   #14
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
Yes, of course you're right. moveq is always 32-bit. Funny that the assembler accepts moveq.w and moveq.b !?

I downloaded the zip again and now it changed.
Will do some tests and give feedback soon...
AmiGer is offline  
Old 15 December 2004, 16:07   #15
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
...runs perfectly now. I have built this small wrapper to alloc and free memory, works fine.

Good work, Photon !

Code:
_LVOAllocMem		equ	-198
_LVOFreeMem		equ	-210

MEMF_PUBLIC		equ	0
MEMF_CHIP		equ	1<<1
MEMF_FAST		equ	1<<2
MEMF_CLEAR		equ	1<<16 

oDataLen=0
oCrunchLen=4
oJumpAddr=8
oTokens=12
oData=28

	SECTION	Startup, CODE, CHIP
	
	movem.l	d2-d7/a2-a6,-(sp)
	lea 	StartOfData(pc),a5	
	move.l	(a5),d0
	add.l	#256,d0
	lea	DataLen(pc),a0
	move.l	d0,(a0)
	move.l	#MEMF_CLEAR!MEMF_CHIP,d1
	move.l	4.w,a6
	jsr	_LVOAllocMem(a6)
	tst.l	d0
	beq	MNoMem
	lea	_Data(pc),a0
	move.l	d0,(a0)
	move.l	d0,oJumpAddr(a5)
	
	bsr	DeCrunchIt

	move.l	_Data(pc),a1
	move.l	DataLen(pc),d0
	move.l	4.w,a6
	jsr	_LVOFreeMem(a6)
	
MNoMem	movem.l	(sp)+,d2-d7/a2-a6
	moveq.l	#0,d0
	rts

_Data	dc.l	0
DataLen	dc.l	0
AmiGer is offline  
Old 15 December 2004, 16:31   #16
AmiGer
Registered User
AmiGer's Avatar
 
Join Date: Sep 2002
Location: Germany
Posts: 349
Just another thought: if you add a "(pc)" to this line

lea StartOfData,a5

the decrunch routine is fully relocatable.

I also changed the decrunch colour to background because some machines doesn't display anything when booting. Hope this isn't contrary to any copyrights ?
AmiGer is offline  
Old 16 December 2004, 01:44   #17
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
No, the decrunch source is totally free, do with it what you please.

I guess a (PC) wouldn't hurt, though the loaded data is very seldom right after the decrunch routine but either in another (data) hunk or loaded from disk tracks to some faraway place. I've added it now anyway.
Photon is offline  
Old 04 January 2005, 02:13   #18
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
Just finished V1.01 of the cruncher. Small optimizations and a bug fix, enjoy!

I put it in The Zone since my hosting seems to be down atm.
Photon is offline  
Old 18 August 2007, 00:58   #19
Photon
Moderator
Photon's Avatar
 
Join Date: Nov 2004
Location: Hult / Sweden
Posts: 4,451
Got a request for the file, it's on Coppershade.org
Photon is offline  
Old 18 August 2007, 06:40   #20
cdoty
RasterSoft Dev

cdoty's Avatar
 
Join Date: Feb 2005
Location: Manteno IL
Posts: 58
Send a message via ICQ to cdoty Send a message via AIM to cdoty
Quote:
Originally Posted by AmiGer View Post
Yes, of course you're right. moveq is always 32-bit. Funny that the assembler accepts moveq.w and moveq.b !?
He he, I just ran across this same thing less than a month ago, while working on the CD-I. The OS-9 assembler also accepted it.
cdoty is offline  
AdSense AdSense  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
Portable bytekiller cruncher phx Coders. Releases 7 23 July 2016 09:31
Cruncher ...... what ? Another World New to Emulation or Amiga scene 27 16 December 2008 20:48
Cruncher Factory plasmatron support.Games 2 20 February 2008 18:40
Scoopex J.O.E.Slideshow II zerohero support.WinUAE 2 01 February 2004 12:57
DEFJAM-Cruncher andreas request.Apps 5 14 September 2001 20:39

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 21:27.


Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Page generated in 0.24519 seconds with 13 queries