The Nibbler cruncher

[Photon]

Made another cruncher

Basically, it compresses like lha but decompresses more than five times faster.

It's a cruncher made to decrunch extremely fast while meeting the ratios of famous archivers. In the ACA500, it decompresses the 512K Kickstart 3.1 in 1.7 seconds. On a stock A500, you will typically see speeds of 156-192k/s.

You can use it to crunch demos, but it doesn't relocate object files. You can however decrunch an executable binary with the de-archiver command option, or with the included example source.

You can also use it as a normal archiver and script the decompression, such as for a slideshow. Note that crunching is quite slow due to the large "sliding window". Due to the 3-stage compression it also requires at least twice the RAM of the file you want to compress.

It works well with code, gfx, modules, and text, or a mix of them like you would see in a one-file demo. It was made for large files, so it's less efficient for files < 8k, though it still decrunches extremely fast. Its larger sliding window starts to really kick in at around 40k.

If you want to try it, download it here. Be happy for feedback / comparisons

[Galahad/FLT]

Originally Posted by Photon

Made another cruncher

Basically, it compresses like lha but decompresses more than five times faster.

It's a cruncher made to decrunch extremely fast while meeting the ratios of famous archivers. In the ACA500, it decompresses the 512K Kickstart 3.1 in 1.7 seconds. On a stock A500, you will typically see speeds of 156-192k/s.

You can use it to crunch demos, but it doesn't relocate object files. You can however decrunch an executable binary with the de-archiver command option, or with the included example source.

You can also use it as a normal archiver and script the decompression, such as for a slideshow. Note that crunching is quite slow due to the large "sliding window". Due to the 3-stage compression it also requires at least twice the RAM of the file you want to compress.

It works well with code, gfx, modules, and text, or a mix of them like you would see in a one-file demo. It was made for large files, so it's less efficient for files < 8k, though it still decrunches extremely fast. Its larger sliding window starts to really kick in at around 40k.

If you want to try it, download it here. Be happy for feedback / comparisons

I tested it on an old cracktro of mine, Propack crunched to 9k, Nibbler did it to 15k.

Original size of file was just over 60k.

If I could suggest an improvement, when it successfully loads a file to crunch, it would be good for Nibbler to say so, maybe print out "Nibbler is now packing" as its not entirely evident that its doing anything, it just sits there, and you're thinking "is it working or has it crashed."

[Photon]

Heh, thanks for testing Galahad, hmm so you're getting a cracktro going?

This looks like the result I get from files containing large blocks of zeros, such as a DCB 40960,0 instead of allocating a screen buffer. I can't tell for sure unless I get the original or .nibbed file, though.

Normally the compressor works like an archiver would, bit "anonymous", but if you type Nibble without any parameters you can discover the classic BOING option which will work like you want methinks

[Galahad/FLT]

Originally Posted by Photon

Heh, thanks for testing Galahad, hmm so you're getting a cracktro going?

This looks like the result I get from files containing large blocks of zeros, such as a DCB 40960,0 instead of allocating a screen buffer. I can't tell for sure unless I get the original or .nibbed file, though.

Normally the compressor works like an archiver would, bit "anonymous", but if you type Nibble without any parameters you can discover the classic BOING option which will work like you want methinks

No its just one I had to hand, I was too lazy to assemble something, so depacked an old AGA fix intro I did for X-Out (the pretend Kickstart 1.3 intro), hence why I had the Propacked file sizes.

[Mark Wright]

It's quick but too quiet :-( Can't we force Paul van der Valk (or even Jimmy!) out of retirement to create a short musical accompaniment? ;-)

[phx]

I'm currently compressing all data of my current game project (code, graphics, music, samples) with a portable C-version of Bytekiller, which I wrote. So it was interesting to me to do some comparisons.

Here are some test:

1. Tiny graphics file (a menu pointer)
Original: 310
Bytekiller: 116
Nibbler: 480

2. Bigger graphics file (320x240 image with some animation phases)
Original: 82170
Bytekiller: 5576
Nibbler: 11915

3. Protracker MOD (only the header and tracks part)
Original: 58428
Bytekiller: 8468
Nibbler: 10432

4. Protracker MOD (only the samples)
Original: 62870
Bytekiller: 53392
Nibbler: 46844

Looks like Nibbler is a good cruncher for samples! For everything else I would continue using the old Bytekiller.

[StingRay]

Originally Posted by phx

Looks like Nibbler is a good cruncher for samples! For everything else I would continue using the old Bytekiller.

The ByteKiller ratio can be increased by using a 2nd pass using a simple RLE algo. Check for example TetraPack, it does exactly this and is basically just a "pimped" 2 pass ByteKiller. Other than that there are much better crunchers than ByteKiller available nowadays (CrunchMania, StoneCracker, ProPack etc. pp).

[phx]

Originally Posted by StingRay

The ByteKiller ratio can be increased by using a 2nd pass using a simple RLE algo.

Hmm... my tests didn't confirm that. A simple RLE over a bytepacked file always created a slightly bigger result.

Other than that there are much better crunchers than ByteKiller available nowadays

For sure!
But Bytekiller was simple enough to rewrite it in C, which was important for my build process. And the 68k decruncher is also quite small.

[Photon]

Nibbler wasn't made for small files that decrunch fast even with a slow cruncher, although it follows lha in size on them too.

phx: Who in the name of Fabio compresses a 310b file separately?? And if you also split files and compress them separately, you probably load them separately as well, so you might not share the goal of the Nibbler - fast loading and decompression. And fitting more on disk than legacy crunchers, which the Nibbler kicks the crap out of so far.

I've crunched a bunch of files now, if you have a candidate for the great-ratio legacy cruncher in your experience, I could add its results to the chart.

I guess by "legacy" I mean one that was and perhaps is commonly used when a game, demo or such is finalized. I know the names probably, but haven't used them enough to know which ones are good or highly regarded.

[Lonewolf10]

I took a quick look at it. Assuming I didn't overlook it, there are no minimum specs regarding OS and CPU - will it work on a 68K CPU and OS 2??

[Galahad/FLT]

Originally Posted by Photon

Nibbler wasn't made for small files that decrunch fast even with a slow cruncher, although it follows lha in size on them too.

phx: Who in the name of Fabio compresses a 310b file separately?? And if you also split files and compress them separately, you probably load them separately as well, so you might not share the goal of the Nibbler - fast loading and decompression. And fitting more on disk than legacy crunchers, which the Nibbler kicks the crap out of so far.

I've crunched a bunch of files now, if you have a candidate for the great-ratio legacy cruncher in your experience, I could add its results to the chart.

I guess by "legacy" I mean one that was and perhaps is commonly used when a game, demo or such is finalized. I know the names probably, but haven't used them enough to know which ones are good or highly regarded.

RNC Propack, no question. Was probably used nearly as much as Bytekiller for commercial games packing.

That would be a good test, Stone cracker and Propack produce similar results.

[Photon]

Originally Posted by Galahad/FLT

RNC Propack, no question. Was probably used nearly as much as Bytekiller for commercial games packing.

That would be a good test, Stone cracker and Propack produce similar results.

I have an idea what's going on with that file of yours. Could you send it, so I can run the analysis on it to confirm?

If it's what I think it is, I may actually be able to hoover up some more bytes.

Bytekiller behaves like the rest of the famous name crunchers I've looked at, i.e. mine is within 1% from LHA and the others are much farther from LHA, Bytekiller average from just 3 files is 12.8% from LHA.

Either way it's about not judging from just a few files, but some random large and small graphics and music files again follows LHA and compress fine.

Got some average decompression speeds, and no surprise (since SCX-Crunch had already beaten all of them) it's confirmed Nibbler absolutely kills all the others.

LZX comes closest at 5.5x slower and Propack is worst - in fact the figures were so bad I'm going to have an extra look that I didn't do it the wrong way.

(Or if you could give me an average decompression speed Galahad I would know that the figures are in the right ballpark.)

I'll put up some comparisons and statistics on bitbrain.se tonight if I have the oomph. Just want to try to find some 'problematic' ones and add them to the charts as well, once I've added the hoover mode.

Originally Posted by Lonewolf10

I took a quick look at it. Assuming I didn't overlook it, there are no minimum specs regarding OS and CPU - will it work on a 68K CPU and OS 2??

Yes. Like most crunchers, it will just need some work memory and a CPU, and is independent of the OS used. And mine come as command line tools with zero dependencies and should cause no problems whatsoever.

[Photon]

Read it and weep, gentlemen

I was most impressed with Propack, although it's quite far from Nibbler's ratio and speed even in mode 2.

Wish you'd send the "11-18% dream ratio" files!! But I have a couple similar files that I hope will do to make a Nibbler 1.03 with improved ratios and speeds on the low-entropy end.

[StingRay]

Originally Posted by phx

Hmm... my tests didn't confirm that. A simple RLE over a bytepacked file always created a slightly bigger result.

RLE is the first pass, 2nd pass is ByteKiller.

Originally Posted by Photon

Read it and weep, gentlemen

I was most impressed with Propack, although it's quite far from Nibbler's ratio and speed even in mode 2.

Wish you'd send the "11-18% dream ratio" files!! But I have a couple similar files that I hope will do to make a Nibbler 1.03 with improved ratios and speeds on the low-entropy end.

I have tested your cruncher with a few reasonable large files and there was no case where it beat CrunchMania, here are my stats:

ASM-One (288464 bytes) Nibbler: 151336 CrM: 140280
Asm-Pro (206692 bytes) Nibbler: 110064 CrM: 104288
CrM (42108 bytes) Nibbler: 24238 CrM: 23524
FLM (127620 bytes) Nibbler: 58534 CrM: 52160
SysSpeed (150676 bytes) Nibbler: 75968 CrM: 57276

Also, it seemed to be very slow and seemingly doing nothing (as there was no process indicator at all), this should be changed so that it is possible to know what's going on, when I first tried it I actually thought it crashed and used a debugger to check what was happening. Other than that it looks promising already, I like that it is is simple CLI executable, very handy!

[Photon]

Originally Posted by StingRay

I have tested your cruncher with a few reasonable large files and there was no case where it beat CrunchMania

Thanks, figures look OK for the current version at first glance, I've added the ones I could get hold of (matching file size) to the charts. I might take a second look at CrunchMania after I finish V1.03, since it comes close to LHA like Propack.

Originally Posted by StingRay

Also, it seemed to be very slow and seemingly doing nothing (as there was no process indicator at all), this should be changed

No need:

Originally Posted by readme.txt

"Each command has options, to see them run them from the command line without parameters."

(I also explained this earlier in the thread.)

The compressor is quite optimized, actually. But scan time increases exponentially with the size of the sliding window, and Nibbler's window is huge. Nibbler is made for producing "final release files", but I should make it a parameter. It makes it better for 'general use'.

Originally Posted by StingRay

I like that it is is simple CLI executable, very handy!

Well, it's more open and flexible that way I think, for scripting or menus.

[StingRay]

Originally Posted by Photon

No need:

That might be your opinion, I wholeheartedly disagree though! It should be the default behaviour that the cruncher outputs at least a rudimentary process indication instead of silently doing its work as you have absolutely no information how long it still takes until crunching has finished or if the cruncher is even still working at all. This is not user-friendly at all and even for a tool mostly intended for coders this isn't nice in my opinion. And I have tested the boing option of course but it doesn't help as it just outputs the final size after the cruncher has finished its work. There should be an option to disable any output if someone doesn't need it but this definitely shouldn't be the default behaviour.

Originally Posted by Photon

The compressor is quite optimized, actually. But scan time increases exponentially with the size of the sliding window, and Nibbler's window is huge. Nibbler is made for producing "final release files", but I should make it a parameter. It makes it better for 'general use'.

I have used it from standard shell window and it took considerably longer than for example CrunchMania, especially for larger files. There might be room for improvement here.

Originally Posted by Photon

Well, it's more open and flexible that way I think, for scripting or menus.

Definitely.

[Photon]

But your quote

Originally Posted by Photon

The compressor is quite optimized, actually. But scan time increases exponentially with the size of the sliding window, and Nibbler's window is huge. Nibbler is made for producing "final release files", but I should make it a parameter. It makes it better for 'general use'.

and your follow-up

Originally Posted by StingRay

[too slow]There might be room for improvement here.

are mutually exclusive!

If you're comparing apples and pears, why not use LHA? It compresses even faster and better, while of course staying in the same decompression speed league as all the others.

That said, I've run tests and stats all yesterday and today, and there will be a Nibbler 1.03 which will feature substantial compression speed gains for low-entropy files along with the compression gains.

[StingRay]

Originally Posted by Photon

But your quote


and your follow-up



are mutually exclusive!

I see, looks like your cruncher is using the most optimised code in the world already, excellent!

Originally Posted by Photon

If you're comparing apples and pears, why not use LHA? It compresses even faster and better, while of course staying in the same decompression speed league as all the others.

For one, LHA has a completely different goal than a file cruncher, I am sure you know that too. Besides, I have compared your cruncher with already available ones, I have no idea how that is comparing apples and pears! I guess, seeing your reply here, honest feedback is not really what you're after though so I'll happily continue to use some other crunchers and will refrain from commenting here in the future.

[Photon]

Originally Posted by StingRay

I see, looks like your cruncher is using the most optimised code in the world already, excellent!

Nibbler 1.03 will be faster for many files, but only due to adding to the algorithm to improve ratio. Now, if you think every algorithm is possible to use with half an order of magnitude faster decompression speeds, you're mistaken. Else it would be possible to just tell all the other cruncher programmers how their decompression speeds suck and that going many times faster just needs some code optimization.

Originally Posted by StingRay

For one, LHA has a completely different goal than a file cruncher, I am sure you know that too. Besides, I have compared your cruncher with already available ones, I have no idea how that is comparing apples and pears! I guess, seeing your reply here, honest feedback is not really what you're after though so I'll happily continue to use some other crunchers and will refrain from commenting here in the future.

No, I appreciate the feedback from everyone. You just seem to "ignite" sometimes and put accusations and disrespect in your comments.

[Lonewolf10]

Originally Posted by Photon

Note that crunching is quite slow due to the large "sliding window".

That's a major understatement. I did my own tests today and am not impressed (ok, I haven't done any compression work yet - but man these compression times are sooooo slowwwww):

Nibbler Compression Tests
--------------------------
MIXED_tiles.DAT   - 2400;   Nibbler = 585   (time: ~1 second w/ 68000 CPU @7Mhz, OS2)
                            LHA     = 212   (time: ~5 seconds w/ 68000 CPU @7Mhz, OS2)

platformTEST.AMOS - 107130; Nibbler = 90210 (time: ~4 hours  w/ 68000 CPU @7Mhz, OS2)
                            LHA     = 82438 (time: ~15 seconds w/ 68000 CPU @7Mhz, OS2)

DD_AllSquare      - 798372; Nibbler = 449696 (time: ~5.5 hours w/ 68040 CPU @14Mhz, OS2),
                            LHA     = 382788 (time: ~2 mins w/ 68040 CPU @14Mhz, OS2)

I tried with BOING, for example:

ram:Nibbler MIXED_tiles.DAT Output.NIB BOING


Do the colour changes (dark blue -> green) indicate anything,
or are they just simply random to show it is still running?

I appreciate that the decompression time is really fast, but with the compression time being on the ridiculous side in my tests (for largish files) I personally wouldn't use it in it's current form.

I also agree with StingRay that there should be some sort of progress display - a bar or perhaps like LHA does, so the user knows how far along it is. The BOING feature only shows us that it's working, and in the case of one file I compressed today I could have been sat there for over 5 hours waiting for it to finish (and that was on a faster CPU than I normally use!) - luckily, there was stuff on TV I could watch whilst I was waiting.