Anyone up for an ASM coding competition?

[Thorham]

That's pretty short

Can't you make it simpler by packing 8 mode bits into a byte, and then having 8 literals (8 bits) and offsets (16 bits) after that, or does that decrease packing efficiency?

[ross]

Originally Posted by Thorham

That's pretty short

Can't you make it simpler by packing 8 mode bits into a byte, and then having 8 literals (8 bits) and offsets (16 bits) after that, or does that decrease packing efficiency?

Yes, you can have the select bit packed in byte (is more or less what standard LZSS do: a select bit for single literal or match/lenght pair 4:12).
But you can try my encoder: practically beat LZSS everytime.

The problem in pure LZSS was the single literal run (initially the file grow significantly), and too many bit for pair.
In my version initially we have little waste for the run of literal (1byte/run vs 1bit/byte, if run>8 we gain); anyhow after the first stages of depacking, incompressible data tend to be less and less... so bad for big files. Same situation for match/lenght (1+15 vs 1+16 bit): if we stay in 'offset' we gain a bit everytime and with longer match!
So, well balanced for small data, bytecode access and no bit count/carry replenish.

[When you wrote 8 literals (8bits) do you mean 256 literals (8bits) or 8 literals (3bits)?]
Your suggestion is just my version with more bits available, so for large files work better. The breakeven could be around 16KB of data.

We could try

[ross]

First try for Thorham version, got 42byte depacker
I need to think better...

[ross]

38byte, i don't know if I can do better


EDIT: 36! (34 with 15 bit offset)

[meynaf]

Changing the encoding to pack more literals and longer offsets will give results that depend heavily on the kind of data to be compressed and its size. So if it's about changing the format, i would like trying to switch to another algorithm.
Can we have access to the typical data you intend to crunch ?

[ross]

Originally Posted by meynaf

Changing the encoding to pack more literals and longer offsets will give results that depend heavily on the kind of data to be compressed and its size. So if it's about changing the format, i would like trying to switch to another algorithm.
Can we have access to the typical data you intend to crunch ?

Meynaf sorry I did not understand if the question is directed to me or to Thorham.
If it's for me, I don't have a data target, it's all for fun and mental exercise like everything on Amiga.
Since returning I rediscovered why I loved this wonderfull machine.

Regarding other types of compressors algorithms I know them well enough, but it is always a pleasure to be able to discuss it with someone!

[meynaf]

It was directed to you. If you had some typical data to compress, we could have tried to get to the smallest code+data final size, more fun imo.

[ross]

Originally Posted by meynaf

It was directed to you. If you had some typical data to compress, we could have tried to get to the smallest code+data final size, more fun imo.

smallest code+data final size, mmh, maybe a depacker for lzx, or already exists?

But the real problem here is the encoder part.
All this lz+huff(/ari) packers (zip, lzx, lha, lzma..), aside optimal parsing and great encoding, is facing another well defined problem: the split point for the entropy table rebuild.
The myriad of ZIP 'deflate' alternative varies in this matter: some euristic vs brute force... (kzip, zopfli, the same 7zip..).
You can have the best algorithms but is like the chicken and the egg causality dilemma: which is the best split point for the best entropy result for the best encoding , if encoding modify entropy which in turn modifies the split point?
And then, as you said, it all depends on the type of data

Back on the road: exists a decompressor for lzx (in optimized 68k code of course!) or similar powerfull compressor?

[meynaf]

Originally Posted by ross

smallest code+data final size, mmh, maybe a depacker for lzx, or already exists?

Again it depends on the kind and size of data, but in some cases lzx can be easy to beat

Originally Posted by ross

But the real problem here is the encoder part.
All this lz+huff(/ari) packers (zip, lzx, lha, lzma..), aside optimal parsing and great encoding, is facing another well defined problem: the split point for the entropy table rebuild.
The myriad of ZIP 'deflate' alternative varies in this matter: some euristic vs brute force... (kzip, zopfli, the same 7zip..).
You can have the best algorithms but is like the chicken and the egg causality dilemma: which is the best split point for the best entropy result for the best encoding , if encoding modify entropy which in turn modifies the split point?

There is little choice but to try several points and keep the best...

Originally Posted by ross

And then, as you said, it all depends on the type of data

This is why we can't get far without actual data sets to compress.
A small bootblock isn't the same as a 30MB wave file.

Originally Posted by ross

Back on the road: exists a decompressor for lzx (in optimized 68k code of course!) or similar powerfull compressor?

None that I know of.
I have code for something powerful but it's not general purpose.

[ross]

Originally Posted by meynaf

..Again it depends on the kind and size of data, but in some cases lzx can be easy to beat
..A small bootblock isn't the same as a 30MB wave file.
..I have code for something powerful but it's not general purpose.

Do You have something for audio data?

[meynaf]

Originally Posted by ross

Do You have something for audio data?

I may have something in this area, yes. Why, do you have nasty ideas to pack audio like crazy ?

[ross]

Originally Posted by meynaf

I may have something in this area, yes. Why, do you have nasty ideas to pack audio like crazy ?

Not nasty nor crazy (can be a language misunderstanding, is this a negative sentence?).
Simply some idea, as I have in so many other fields (only for fun and selflearning).
Nothing of lossless, but something about Adaptive DPCM (not IMA related), with quantisation table based on Amiga audio characteristic (mostly on PWM volume and not-DMA driving).
Until a few days ago I did not even know what it was the ADPCM...

Ciao

[meynaf]

Originally Posted by ross

Not nasty nor crazy (can be a language misunderstanding, is this a negative sentence?).

This didn't have a negative meaning. I just asked if you were into something big, unusual.

Originally Posted by ross

Simply some idea, as I have in so many other fields (only for fun and selflearning).
Nothing of lossless, but something about Adaptive DPCM (not IMA related), with quantisation table based on Amiga audio characteristic (mostly on PWM volume and not-DMA driving).
Until a few days ago I did not even know what it was the ADPCM...

Frankly i prefer lossless compression so I will not follow you there. Sorry.

[ross]

Originally Posted by meynaf

This didn't have a negative meaning. I just asked if you were into something big, unusual.

If I can do something I'll let you know

[Don_Adan]

Originally Posted by ross

smallest code+data final size, mmh, maybe a depacker for lzx, or already exists?

But the real problem here is the encoder part.
All this lz+huff(/ari) packers (zip, lzx, lha, lzma..), aside optimal parsing and great encoding, is facing another well defined problem: the split point for the entropy table rebuild.
The myriad of ZIP 'deflate' alternative varies in this matter: some euristic vs brute force... (kzip, zopfli, the same 7zip..).
You can have the best algorithms but is like the chicken and the egg causality dilemma: which is the best split point for the best entropy result for the best encoding , if encoding modify entropy which in turn modifies the split point?
And then, as you said, it all depends on the type of data

Back on the road: exists a decompressor for lzx (in optimized 68k code of course!) or similar powerfull compressor?

One of the best known me 68k is Arj mode 7 (better than LZX). Better is only hitchhikr's PackFire compressor, but has slow decompression, due mulu.http://www.pouet.net/prod.php?which=54840
Optimised PackFire depacker was used in one or two WT releases.

[ross]

Originally Posted by Don_Adan

One of the best known me 68k is Arj mode 7 (better than LZX). Better is only hitchhikr's PackFire compressor, but has slow decompression, due mulu.http://www.pouet.net/prod.php?which=54840
Optimised PackFire depacker was used in one or two WT releases.

Thanks for the info!

PackFire is LZMA based (entropy coder is arithmetic, so the mul).
Slow for 68k

Some test on ARJm7: beat original LZX practically everytime and is relatively fast!. I try to better understand how the encoding is.
But the 'new' LZX (https://wimlib.net/compression.html#LZX) beat ARJm7 (only tested a small sample of files).

We urge unpacker for the new LZX to test speed on 68k

[meynaf]

If you really like high compression ratios I suggest you try WinRK.

[ross]

Originally Posted by meynaf

If you really like high compression ratios I suggest you try WinRK.

Tks!

[meynaf]

Another idea for short code competition...

It's about a routine which takes some value as input (in a register).

It must output 4 flags in the 4 lower bits of another register, giving the following information :
- value fits in signed word (range FFFF8000-00007FFF)
- value fits in unsigned word (range 00000000-0000FFFF)
- value fits in signed byte (range FFFFFF80-0000007F)
- value fits in unsigned byte (range 00000000-000000FF)

Original value must not be altered. Other bits in the flags register are unimportant.
020+ is allowed, total size (including rts and eventual data) has precedence over speed.

If you need motivation, goal to beat is 42 bytes.

[meynaf]

Next try, did it in 32 bytes
I'm starting to believe it can't be done shorter...