English Amiga Board - View Single Post

introspec · 26 November 2021, 15:21

Hi guys, I did a lot of work on fast decompression for Z80 and I think some of our tech could be useful for you as well. ZX0 is an amazing format with a couple of very modern tricks, so it can compete (almost!) with zip-style compressors, even though it does not come with an entropy coder. The compressor is terrily slow, but if you can afford to lose, maybe, 0.3-0.5% compression (at most), there is a new packer written by Emmanuel Marty that is orders of magnitude faster: https://github.com/emmanuel-marty/salvador
On some files it beats the original compressor too.

Since you mentioned LZ4 several times in this thread, I think that you may also want to have a look at LZSA compressor, also written by Emmanuel Marty: https://github.com/emmanuel-marty/lzsa
It comes with 2 compression formats: LZSA1 and LZSA2. LZSA1 was designed intentionally to compete with LZ4 and it does beat LZ4 both in compression ratio and in decompression speed (basically, it requires less work in the main loop, so correctly designed decompressor should always beat LZ4). LZSA2 is somewhere in between LZSA1 and ZX0 - compresses approximately at the level of NRV2B, but can be decompressed much faster, due to the active use of bytes and nibbles in the compressed stream.

Last but not least, I created the compression ratio/decompression speed diagrams listed earlier in this thread. It is important to understand that they were generated for (mostly) my decompressors and for Z80. I have no way of testing 68k decompressors, esp in terms of the decompression speed, so your mileage may vary depending how well optimized your decompressors are. I have examples of Z80 decompressors that I could make twice as fast as they were originally, so please be careful when comparing the compressors with each other. Just any decompressor will not automatically get you onto the Pareto frontier.

26 November 2021, 15:21	#64
introspec Registered User Join Date: Jun 2020 Location: London Posts: 4	Hi guys, I did a lot of work on fast decompression for Z80 and I think some of our tech could be useful for you as well. ZX0 is an amazing format with a couple of very modern tricks, so it can compete (almost!) with zip-style compressors, even though it does not come with an entropy coder. The compressor is terrily slow, but if you can afford to lose, maybe, 0.3-0.5% compression (at most), there is a new packer written by Emmanuel Marty that is orders of magnitude faster: https://github.com/emmanuel-marty/salvador On some files it beats the original compressor too. Since you mentioned LZ4 several times in this thread, I think that you may also want to have a look at LZSA compressor, also written by Emmanuel Marty: https://github.com/emmanuel-marty/lzsa It comes with 2 compression formats: LZSA1 and LZSA2. LZSA1 was designed intentionally to compete with LZ4 and it does beat LZ4 both in compression ratio and in decompression speed (basically, it requires less work in the main loop, so correctly designed decompressor should always beat LZ4). LZSA2 is somewhere in between LZSA1 and ZX0 - compresses approximately at the level of NRV2B, but can be decompressed much faster, due to the active use of bytes and nibbles in the compressed stream. Last but not least, I created the compression ratio/decompression speed diagrams listed earlier in this thread. It is important to understand that they were generated for (mostly) my decompressors and for Z80. I have no way of testing 68k decompressors, esp in terms of the decompression speed, so your mileage may vary depending how well optimized your decompressors are. I have examples of Z80 decompressors that I could make twice as fast as they were originally, so please be careful when comparing the compressors with each other. Just any decompressor will not automatically get you onto the Pareto frontier.