22 September 2021, 22:37 | #41 |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
WOW.
First test with enhanced offset. This forcibily require my unpacker, the official one doesn't/can't support offset >32KiB. I used an offset of 100KB, but theoretically in can work until 16MiB (but better to have a quantum computer in case...) Zombies.SHP compressed from 245720 to 44918 bytes! execution time : 3018.983 s (you read that right.. 50 minutes..) For reference: 65535 RNC (LZ, -m 2) 54966 Crunch-Mania (LZH) 53677 RNC (LZH, -m 1) 52415 gzip (-9) 50553 official zx0 (hardwired offset 32740) 48908 arjm7 (this is a special unfortunate case for arm7 because the file is highly redundant) 46779 aplibx (apultra like performance) 46238 nrv2r (my special version) 46126 rar (max compression) Yes, beaten all packers using LZ or LZH formats. And now the speed, standard A500 7Mhz, code and data in chip-ram, Zombies.SHP: 1239ms, 198.32 KB/s! It's very fast! Of course I have tested the unpacked data, so the decoder works fine Maybe I'll edit this post after doing a test with an offset of 256KB (it probably takes half a day to finish ) Now I stop flooding this thread.. |
22 September 2021, 23:12 | #42 |
Ex nihilo nihil
Join Date: Oct 2017
Location: CH
Posts: 4,995
|
|
23 September 2021, 08:30 | #43 |
son of 68k
Join Date: Nov 2007
Location: Lyon / France
Age: 51
Posts: 5,352
|
|
23 September 2021, 10:23 | #44 | ||
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Quote:
(while in other compressors it is a secondary factor). I realized this following the choices of regular match offsets in the decoder, which for a human seem to make no sense, but which in the long term give great results. The problem with using rep-matches is, to my knowledge so I could be wrong, the lack of a good heuristic algorithm for general use. While various accelerators exist for LZ parsers, the rep-match problem remains NP-hard (or similar) and therefore the solution in ZX0 is to use brute force. And since rep-match is in one of the primary paths in the decompression code it has a huge impact on the compression factor (despite the simplicity of the encoding). Quote:
You've a PM with (preliminary) code so do no spread But try it to confirm my decompression speed report and zombie.shp content. Last edited by ross; 26 September 2021 at 18:31. Reason: removed file, soon anyone will be able to generate their own :) |
||
23 September 2021, 11:47 | #45 | |
Registered User
Join Date: Aug 2018
Location: Untergrund/Germany
Posts: 410
|
Quote:
Super impressive depack speed and ratio Ross! Next of my games won't use Shrinkler anymore i guess (given you plan to this release it as an exe packer). For only packing data it would be a bit too slow for me |
|
23 September 2021, 13:21 | #46 | |
son of 68k
Join Date: Nov 2007
Location: Lyon / France
Age: 51
Posts: 5,352
|
Quote:
But content confirmed. Same zombie.shp than mine (as taken from zombie apocalypse 2 game). To be able to beat the size i had to go to PPM range of compressors. No hope for beating that i think, but perhaps it would be interesting to add a few complexity for faster machines. However this data is highly compressible. I'd like to see how this compressor performs on different data. If it's able to repetitively beat LZX (~46k here) then it can have interesting applications. |
|
23 September 2021, 14:09 | #47 |
Registered User
Join Date: Jun 2008
Location: somewhere else
Posts: 523
|
35872 bytes with packfire (but decrunching is a lot slower).
|
23 September 2021, 14:36 | #48 |
son of 68k
Join Date: Nov 2007
Location: Lyon / France
Age: 51
Posts: 5,352
|
Record to break is 28280 (WinRK).
|
23 September 2021, 19:29 | #49 | ||
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Quote:
So for compression factors such as shrinkler (or even for 'quality' LZH) it is not comparable. Having an entropic stage in light pre-compressed files (LZ4 or similar) or with fewer match relationships helps a lot this different category of compressors. Let alone if the entropy compression is arithmetic or even with CM. But the very interesting thing is the compression factor in relation to the decompression speed, it could replace MANY of existing packers And during compression you can Quote:
The decompression speed target is too much different. There are some very promising algorithms that use ANS or advanced forms of LZ on the compression forums and some can be ported to 68k, but the problem is the decompression speed. EDIT: and yes, for 'extreme' compression and slow decrunching you can use LZMA (packfire ), or the aforementioned shrinkler on 68k (the decoders with a 'mul'). Last edited by ross; 23 September 2021 at 19:56. |
||
24 September 2021, 19:18 | #50 |
Registered User
Join Date: Jul 2005
Location: -
Posts: 1,698
|
Impressive results. Haven't touched compression for a long time, but still find this topic interesting.
|
24 September 2021, 20:52 | #51 |
Ex nihilo nihil
Join Date: Oct 2017
Location: CH
Posts: 4,995
|
|
29 September 2021, 15:21 | #52 |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Ok, time for a significative test with the 'alternative' ZX0:
Code:
ZX0 v2.1a: Optimal data compressor by Einar Saukas alternative encoding by ross, beta version Usage: ZX0.exe [-f] [-c] [-b] [-a] [-mn] [-q] input [output.zx0] -f Force overwrite of output file -c Classic file format (v1.*) -b Compress backwards -a Alternative bitcoding -mn Max offset (n = 0 to 9) -q Quick non-optimal compression ZX0.exe -a -m9 file Code:
unpacked arjm7 azx0 dirkernel.tmp 9180 6720 6025 logo_fade.bin 131160 27284 27248 main.bin 331380 122194 121677 ym7Pack0.bin 18672 12402 12479 ym7Pack1.bin 202162 113136 113483 ym7Pack2.bin 200448 112806 112660 ym7Pack3.bin 199646 107798 108418 ym7Pack4.bin 203764 106350 105815 ym7Pack5.bin 174760 91400 91068 ym7Pack6.bin 128800 58362 58054 CosoPackLz4.bin 218912 142048 146061 Basically it is now on par with ARJm7, excluding the pre-LZ4-packed Coso_file. In this case, unfortunately, it is inevitable that many LZ sequences are 'stolen' by the prepack and the entropic part of ARJ wins. I'd like to try the un-prepacked version, or a version where LZ4 is replaced internally by ZX0. (I don't tell you how long it took me to pack all the files, even though I made several cores work at the same time with different files..) |
29 September 2021, 23:49 | #53 |
OCS forever!
Join Date: Mar 2019
Location: Birmingham, UK
Posts: 418
|
This is all cool stuff. I wrote a basic lz77 packer once and I never want to go near that again. Keep sharing
|
02 October 2021, 19:14 | #54 | |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Quote:
And as I imagined the situation is different. Not only by replacing LZ4 with ZX0 the original pre-compressed file is much smaller, but also the final file, and with a large margin Original file 153031 vs 218912. Packed file 134212 vs 142048! So for AmigAtari we have a possible substitute --- I experimented a bit with an entropy stage. The only point where it seems probable to insert it is for pure literals, because in all other cases I would destroy the properties of the encoder stream or I would just lose speed. The fastest and most effective seems to be using a prefix-coder constructed using a package-merge algorithm (basically a tabled length-limited canonical huffman) with an adaptive (tree) block subdivider. I squeezed another ~1700bytes from Zombies.SHP. Very rough tests point out to me that ARJm7 is systematically beaten. But I don't know, the decoder becomes much slower and complex. I don't have a decompressor for 68k yet, but the feeling is that it should unpack faster than ARJm7. For now I leave it on stand-by, I'm not too convinced. The beauty of this decompressor is the extreme speed and the good compression, I would not want it to lose its qualities. Only problem is the very slow compression, so to use only if you are aware of it |
|
02 October 2021, 20:07 | #55 | ||
Registered User
Join Date: Jun 2020
Location: Brno
Posts: 90
|
Quote:
Quote:
Thank you ross for your great work. |
||
02 October 2021, 20:49 | #56 | |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Maybe..
Quote:
I'll give you some examples. CosoPackxxx.bin is made up of more than a hundred small files, ranging in size from 1 to 15KiB. Pre-compression (which is used to keep the indexed files compressed directly in memory) took less than a couple of minutes for all files. But the "merged" file (153031 bytes) took about 15/20 minutes. And from an old message you will have read that Zombies.SHP (245720 bytes) took 50 minutes. The biggest file I compressed was main.bin (331380 bytes) from AmigaAtari demo: it took like.. hours (3, 4 maybe?) And in this case I also had another problem: I compiled a 32-bit version and for a moment was afraid that the compression would fail because the memory usage exploded to 3.5GiB! So maybe it's better if I also compile a 64-bit version.. In fact it was created for 8-bit systems, with files of 64KiB at max, in this cases the times are relatively acceptable. Yes, I used an old i7 (and alway with maximum compression) but hmm, it's a bit slow At least you can compress file in batches and using different cores. |
|
03 October 2021, 03:12 | #57 |
Registered User
Join Date: Jan 2008
Location: Warsaw/Poland
Age: 56
Posts: 2,030
|
Header ZX01 Ross Packer mode 1
Header ZX02 Ross Packer mode 2 |
06 October 2021, 14:41 | #58 |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Einar pointed me out to his new experimental ZX family compressor, ZX5.
Thanks, Basically it extend the rep-match concept to the last three offset (like LZX). "..ZX5 compressor is very much slower.." .. this sentence scares me a bit .. When I have some time I try to play around with it a bit, I think it is not complicated to write a suitable unpacker for 68k starting from my sources (and with acceptable speed losses). It might be useful to squeeze out a few more bytes It remains to be seen whether any gain in compression is adequately compensated for by the loss of speed. |
06 October 2021, 17:07 | #59 | |
Registered User
Join Date: Jun 2016
Location: europe
Posts: 1,054
|
Quote:
For zx0 it didn't take long to figure out I better modify the compressor to produce a script so I can then just bypass optimize(), and compress() the whole thing repeatedly in 1 second instead of waiting 40+minutes each time. So as long as there's no need to mess around with the optimizer it should be fine, one and done, but still... Very much slower . Anyway, I'm "slightly" behind on this, real life work catching up. How relevant are those numbers of yours above? I have several versions with different sizes and speeds, some with original bitcode and some with modified. Just screwing around for now, still not happy with depacker, trying to combine a few not really compatible optimizations. Testing with my stuff and zombies.shr, don't have any of the other files you mentioned. Zombies compressed with 256k max offsets down to ~44870 bytes and decompressed in 1.228 seconds with a 150ish bytes depacker is kind of best what I have at the moment... |
|
06 October 2021, 18:10 | #60 | |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,488
|
Quote:
You have a PM |
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Zip packer corrupting files ? | Retroplay | support.Apps | 13 | 23 July 2011 12:17 |
old soundeditors and pt-packer | Promax | request.Apps | 7 | 14 July 2010 13:21 |
Pierre Adane Packer | Muerto | request.Modules | 15 | 21 October 2009 18:03 |
Power Packer PP Files HELP | W4r3DeV1L | support.Apps | 2 | 30 September 2008 06:20 |
Cryptoburners graphics packer | Ziaxx | request.Apps | 1 | 06 March 2007 10:30 |
|
|