Fastest depacker - Page 2

mcgeezer · 18 June 2019, 20:56

Here's a comparison between Pro Pack and gzip (-9).

I did try cranker but the two largest files I packed didn't come out smaller than ProPack on my project files, I'm trying to squeeze everything on 1 880 kb disk... I guess there is sector overhead due to the large number of files.

Code:

UNPACK	GZIPPED	RNC	FILENAME	FILE TYPE
269364	118666	124117	ry_sprites_p1L	IFF
211168	75518	75448	rygar	EXE
189194	160362	160856	rygar_v12	MOD
164890	107786	113115	ry_tiles_p9	IFF
120092	18833	19579	ry_bg_lev9	IFF
110516	5761	6195	ry_bg_lev5	IFF
97920	26129	27154	rygar_hwconv	IFF
67304	13506	14043	ry_bg_lev1	IFF
67304	28365	28320	ry_bg_lev2	IFF
67304	13225	13278	ry_bg_lev21	IFF
67304	20585	20986	ry_bg_lev25	IFF
64304	14186	14786	ry_bg_lev23	IFF
57704	18046	18822	ry_bg_lev19	IFF
44504	15685	16632	ry_bg_lev13	IFF
36116	8367	9084	ry_bg_lev10	IFF
29616	8793	9082	RyIntro	IFF
28916	8231	8403	ry_bg_lev6	IFF
22896	6675	6751	lower_panel2	iFF
21120	727	632	ry_map9	DAT
21104	590	522	ry_bg_lev27	IFF
18020	14114	15196	rygar008	SAMPLE
16192	721	674	ry_map5	DAT
16020	11987	12957	rygar014	SAMPLE
15892	6484	6938	rygar043	SAMPLE
15360	706	659	ry_map11	DAT
15360	697	645	ry_map23	DAT
14776	9968	10845	rygar005	SAMPLE
12384	924	885	ry_map24	DAT
10184	736	732	rygar_sunset320_bg	IFF
9990	5717	6080	rygar040	SAMPLE
8924	8579	8645	rygar006	SAMPLE
8568	6068	6884	rygar022	SAMPLE
8086	994	977	rygar_mountains40_bg	IFF
6804	4616	5136	rygar023	SAMPLE
6272	4160	4403	RyLogoC	IFF
5536	3943	4446	rygar017	SAMPLE
5262	3731	4294	rygar031	SAMPLE
4224	991	1014	ry_map1	DAT
4224	742	751	ry_map10	DAT
4224	975	972	ry_map12	DAT
4224	856	866	ry_map13	DAT
4224	925	921	ry_map14	DAT
4224	1027	1040	ry_map15	DAT
4224	572	549	ry_map16	DAT
4224	913	931	ry_map17	DAT
4224	930	933	ry_map18	DAT
4224	740	739	ry_map19	DAT
4224	550	540	ry_map2	DAT
4224	691	685	ry_map20	DAT
4224	546	525	ry_map21	DAT
4224	708	693	ry_map22	DAT
4224	576	558	ry_map25	DAT
4224	980	994	ry_map26	DAT
4224	215	193	ry_map27	DAT
4224	552	527	ry_map3	DAT
4224	579	562	ry_map4	DAT
4224	919	917	ry_map6	DAT
4224	670	663	ry_map7	DAT
4224	567	545	ry_map8	DAT
3616	2540	2736	rygar003	SAMPLE
3064	2561	2867	rygar002	SAMPLE
2736	1167	1242	Ranks	iFF
2736	1138	1167	RanksMain	iFF
2586	1875	1849	rygar015	SAMPLE
2544	685	638	ry_config2	DAT
2448	683	669	ry_config4	DAT
2362	654	613	ry_config7	DAT
2320	607	556	ry_config3	DAT
2292	701	643	ry_config1	DAT
2266	604	578	ry_config6	DAT
2186	555	535	ry_config10	DAT
2156	639	590	ry_config11	DAT
2146	640	590	ry_config13	DAT
2074	596	535	ry_config17	DAT
2074	632	611	ry_config8	DAT
2072	633	614	ry_config12	DAT
2072	595	534	ry_config18	DAT
1946	564	546	ry_config24	DAT
1942	1881	1842	rygar012	SAMPLE
1936	663	628	letsfight32col	iFF
1936	317	285	letsfight32mask	iFF
1876	551	534	ry_config16	DAT
1876	550	535	ry_config20	DAT
1862	620	547	ry_config21	DAT
1860	620	548	ry_config22	DAT
1800	510	455	ry_config14	DAT
1798	503	463	ry_config15	DAT
1684	401	354	ry_config9	DAT
1680	444	411	ry_config19	DAT
1606	1565	1563	rygar016	SAMPLE
1548	410	364	ry_config5	DAT
1456	1275	1185	AACWaterAnim	iFF
1168	195	169	ry_config23	DAT
1080	196	169	ry_config27	DAT
1070	191	161	ry_config26	DAT
1068	191	163	ry_config25	DAT
710	524	531	RyFont8x8Time1	iFF
				
2120850	800081	823339

Gzip wins by roughly 20Kb... which is very significant!

I'll need to figure out how to decompress the data files though.

saimon69 · 18 June 2019, 21:27

Quote:

Originally Posted by DrBong

Yeh, what Phill said. Shrinkler will usually get the best bang for your buck for compression, but it's very very slow (e.g. took ~3 minutes IIRC to decompress the 1.4MB Crazy Priest game exec on a standard A500 when I one-disked the game). Not surprising as Shrinkler was likely designed to compress demos.

I admit that was partially my fault, in 2015 was unable to make the mod file sound good with less than 200k at the time; now with some more experience should be at least one third smaller

DrBong · 18 June 2019, 21:47

Quote:

Originally Posted by mcgeezer

Here's a comparison between Pro Pack and gzip (-9).

I did try cranker but the two largest files I packed didn't come out smaller than ProPack on my project files, I'm trying to squeeze everything on 1 880 kb disk... I guess there is sector overhead due to the large number of files.

Gzip wins by roughly 20Kb... which is very significant!

I'll need to figure out how to decompress the data files though.

That's quite an interesting outcome.....but, as you said, decompressing the Gzip packed data files might offer a challenge (I wonder if Lx will detect the format??!!).

BTW I forgot to mention another handy little gem called CrunchyDat. It's based on the ProPack algorithm and was used by the Bitmap Brothers and Team 17 for their games. Might be worth a looksy (along with XPK) to see how it compares to ProPack and Gzip.

DamienD · 18 June 2019, 22:07

Indeed CrunchyDat is awesome, I use it all the time

...but it's only for program files; not data files.

Don_Adan · 18 June 2019, 23:46

Quote:

Originally Posted by mcgeezer

Here's a comparison between Pro Pack and gzip (-9).

I did try cranker but the two largest files I packed didn't come out smaller than ProPack on my project files, I'm trying to squeeze everything on 1 880 kb disk... I guess there is sector overhead due to the large number of files.

Code:

UNPACK	GZIPPED	RNC	FILENAME	FILE TYPE
269364	118666	124117	ry_sprites_p1L	IFF
211168	75518	75448	rygar	EXE
189194	160362	160856	rygar_v12	MOD
164890	107786	113115	ry_tiles_p9	IFF
120092	18833	19579	ry_bg_lev9	IFF
110516	5761	6195	ry_bg_lev5	IFF
97920	26129	27154	rygar_hwconv	IFF
67304	13506	14043	ry_bg_lev1	IFF
67304	28365	28320	ry_bg_lev2	IFF
67304	13225	13278	ry_bg_lev21	IFF
67304	20585	20986	ry_bg_lev25	IFF
64304	14186	14786	ry_bg_lev23	IFF
57704	18046	18822	ry_bg_lev19	IFF
44504	15685	16632	ry_bg_lev13	IFF
36116	8367	9084	ry_bg_lev10	IFF
29616	8793	9082	RyIntro	IFF
28916	8231	8403	ry_bg_lev6	IFF
22896	6675	6751	lower_panel2	iFF
21120	727	632	ry_map9	DAT
21104	590	522	ry_bg_lev27	IFF
18020	14114	15196	rygar008	SAMPLE
16192	721	674	ry_map5	DAT
16020	11987	12957	rygar014	SAMPLE
15892	6484	6938	rygar043	SAMPLE
15360	706	659	ry_map11	DAT
15360	697	645	ry_map23	DAT
14776	9968	10845	rygar005	SAMPLE
12384	924	885	ry_map24	DAT
10184	736	732	rygar_sunset320_bg	IFF
9990	5717	6080	rygar040	SAMPLE
8924	8579	8645	rygar006	SAMPLE
8568	6068	6884	rygar022	SAMPLE
8086	994	977	rygar_mountains40_bg	IFF
6804	4616	5136	rygar023	SAMPLE
6272	4160	4403	RyLogoC	IFF
5536	3943	4446	rygar017	SAMPLE
5262	3731	4294	rygar031	SAMPLE
4224	991	1014	ry_map1	DAT
4224	742	751	ry_map10	DAT
4224	975	972	ry_map12	DAT
4224	856	866	ry_map13	DAT
4224	925	921	ry_map14	DAT
4224	1027	1040	ry_map15	DAT
4224	572	549	ry_map16	DAT
4224	913	931	ry_map17	DAT
4224	930	933	ry_map18	DAT
4224	740	739	ry_map19	DAT
4224	550	540	ry_map2	DAT
4224	691	685	ry_map20	DAT
4224	546	525	ry_map21	DAT
4224	708	693	ry_map22	DAT
4224	576	558	ry_map25	DAT
4224	980	994	ry_map26	DAT
4224	215	193	ry_map27	DAT
4224	552	527	ry_map3	DAT
4224	579	562	ry_map4	DAT
4224	919	917	ry_map6	DAT
4224	670	663	ry_map7	DAT
4224	567	545	ry_map8	DAT
3616	2540	2736	rygar003	SAMPLE
3064	2561	2867	rygar002	SAMPLE
2736	1167	1242	Ranks	iFF
2736	1138	1167	RanksMain	iFF
2586	1875	1849	rygar015	SAMPLE
2544	685	638	ry_config2	DAT
2448	683	669	ry_config4	DAT
2362	654	613	ry_config7	DAT
2320	607	556	ry_config3	DAT
2292	701	643	ry_config1	DAT
2266	604	578	ry_config6	DAT
2186	555	535	ry_config10	DAT
2156	639	590	ry_config11	DAT
2146	640	590	ry_config13	DAT
2074	596	535	ry_config17	DAT
2074	632	611	ry_config8	DAT
2072	633	614	ry_config12	DAT
2072	595	534	ry_config18	DAT
1946	564	546	ry_config24	DAT
1942	1881	1842	rygar012	SAMPLE
1936	663	628	letsfight32col	iFF
1936	317	285	letsfight32mask	iFF
1876	551	534	ry_config16	DAT
1876	550	535	ry_config20	DAT
1862	620	547	ry_config21	DAT
1860	620	548	ry_config22	DAT
1800	510	455	ry_config14	DAT
1798	503	463	ry_config15	DAT
1684	401	354	ry_config9	DAT
1680	444	411	ry_config19	DAT
1606	1565	1563	rygar016	SAMPLE
1548	410	364	ry_config5	DAT
1456	1275	1185	AACWaterAnim	iFF
1168	195	169	ry_config23	DAT
1080	196	169	ry_config27	DAT
1070	191	161	ry_config26	DAT
1068	191	163	ry_config25	DAT
710	524	531	RyFont8x8Time1	iFF
				
2120850	800081	823339

Gzip wins by roughly 20Kb... which is very significant!

I'll need to figure out how to decompress the data files though.

If you need very good packing and fast depacking, nothing can beat ARJ mode 7 from Atari . GZIP is only average packer, like RNC too. If you need the best packing and slow (for 68000, not tested speed for 68020) depacking, you can use PackFire (LZM mode).

SteveMoody · 19 June 2019, 09:43

I don't know if this will be of any use to you. FC8 has been designed for fast decompression.

https://www.bigmessowires.com/2016/0...decompression/

Galahad/FLT · 19 June 2019, 12:55

I cant help feeling that packing iff files isnt getting the best out of Propack as iff is alread RLE packed.

Have you tried converting an iff file to raw and then using propack on the raw file to see if you get a better result?

mcgeezer · 19 June 2019, 15:39

Quote:

Originally Posted by Galahad/FLT

I cant help feeling that packing iff files isnt getting the best out of Propack as iff is alread RLE packed.

Have you tried converting an iff file to raw and then using propack on the raw file to see if you get a better result?

All them IFF files are saved uncompressed.

ross · 08 July 2019, 22:12

Quote:

Originally Posted by ross

Speaking of UPX.. I'm working in a new toolchain for nrv2b

There is still something to squeeze out from it..

I quote myself.

Beta version for the new nrv2b 68k data compression handler ready.
Only a beta because my idea is to add some facilities for in-place decompression.
But would require a modification of the bit-coding..

I'm sure that WayneK would like to try it

roondar · 08 July 2019, 22:17

On the topic of the many files and filesystem overhead, IIRC the Amiga fast file system uses blocks of 512 bytes. So a file that is 513 bytes should take up the same space on disk as one that is 1024 bytes.

This ended up being a problem for me as my most recent project had a bunch of really small files that were on the order of 64 bytes in size. So, I ended up I bundling all the small files that logically 'belonged together' into a bigger one and simply use offsets into the loaded data. This has actually saved me both disk space due to lower filesystem overhead, but also due to RNC performing somewhat better on bigger files.

ross · 08 July 2019, 22:22

Quote:

Originally Posted by roondar

On the topic of the many files and filesystem overhead, IIRC the Amiga fast file system uses blocks of 512 bytes. So a file that is 513 bytes should take up the same space on disk as one that is 1024 bytes.

This ended up being a problem for me as my most recent project had a bunch of really small files that were on the order of 64 bytes in size. So, I ended up I bundling all the small files that logically 'belonged together' into a bigger one and simply use offsets into the loaded data. This has actually saved me both disk space due to lower filesystem overhead, but also due to RNC performing somewhat better on bigger files.

And if you even want to overdo it you can also make a filesystem that doesn't use blocks but indexes files directly by offset

roondar · 08 July 2019, 22:23

That would also work. But it's a lot of effort for a rather peculiar use case

ross · 08 July 2019, 22:24

Quote:

Originally Posted by roondar

That would also work. But it's a lot of effort for a rather peculiar use case

mcgeezer · 08 July 2019, 22:30

Quote:

Originally Posted by roondar

On the topic of the many files and filesystem overhead, IIRC the Amiga fast file system uses blocks of 512 bytes. So a file that is 513 bytes should take up the same space on disk as one that is 1024 bytes.

This ended up being a problem for me as my most recent project had a bunch of really small files that were on the order of 64 bytes in size. So, I ended up I bundling all the small files that logically 'belonged together' into a bigger one and simply use offsets into the loaded data. This has actually saved me both disk space due to lower filesystem overhead, but also due to RNC performing somewhat better on bigger files.

Rygar also has this problem in that I have 54 small level files, the problem is I can't box them up as they are loaded after each round is completed. All I can do is endeavour to keep them close to 512 byte boundaries after being packed with ar7.

ar7 by the way is an awesome packer.... it has beat RNC do death for Rygar....now I'm worried I have too much space left on disk for a really nice FlashTro.

ross · 08 July 2019, 22:39

Quote:

Originally Posted by mcgeezer

ar7 by the way is an awesome packer.... it has beat RNC do death for Rygar....now I'm worried I have too much space left on disk for a really nice FlashTro.

What did I tell you?

Galahad/FLT · 08 July 2019, 23:13

Quote:

Originally Posted by mcgeezer

Rygar also has this problem in that I have 54 small level files, the problem is I can't box them up as they are loaded after each round is completed. All I can do is endeavour to keep them close to 512 byte boundaries after being packed with ar7.

ar7 by the way is an awesome packer.... it has beat RNC do death for Rygar....now I'm worried I have too much space left on disk for a really nice FlashTro.

Me personally, i'd create a single file with all the map files in that single file, and use a longword index to point to the headers of each file.

AmigaDOS loading small files is a ballache, lots of grinding and stepping on the disk, if you have enough spare memory, just load the file up to be permanently in memory.

Galahad/FLT · 08 July 2019, 23:16

Well unfortunately, I never could use a depacker for realtime depacking of individual tracks within a certain game i've cracked, it would slow the game down on 68000 too much.

So I had to tackle the game in question in an entirely different way which made realtime depacking rather moot really.

All going well, this poxy game will be released next week, just need to do a conversion routine for it and then its done.

I have to say, its been a hell of a lot of work to get this game to operate because of the way the programmer made it as difficult as possible to crack in anything like a normal way, so kudos to him for trying, just a shame the game doesn't really warrant that kind of effort, but back in the day, this one would have been quite irritating to crack quickly because of the way its structured, but i guess that was entirely the point!

roondar · 08 July 2019, 23:19

Wasn't there a thread somewhere talking about using standard ZIP were someone managed to create a decruncher for ZIP files that was fast enough to keep up with the floppy drive even on a 68000?

Anyway, I like the 'make big file, use offsets' approach. Easier than making your own file system, but still lets you avoid block size issues.

Galahad/FLT · 08 July 2019, 23:29

Quote:

Originally Posted by roondar

Wasn't there a thread somewhere talking about using standard ZIP were someone managed to create a decruncher for ZIP files that was fast enough to keep up with the floppy drive even on a 68000?

Anyway, I like the 'make big file, use offsets' approach. Easier than making your own file system, but still lets you avoid block size issues.

Not in the case of this game as it uses a custom tracker loader which is tied to the interrupts (like SWIV and NinJa Warriors and Saint Dragon, Final Fight), and there just isn't enough processor time left to load a track, depack and not have it drop a frame, but my solution to that solved it, so it just preloads the data it needs and does a straight copy from extra memory to where it needs to be.

One off protections are always the best

ross · 09 July 2019, 00:35

Quote:

Originally Posted by Galahad/FLT

.. so it just preloads the data it needs and does a straight copy from extra memory to where it needs to be.

You're cheating! Nah, I'm kidding

If there is no other way this is the right one.
I look forward for the release

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What depacker to use? Premier Manager II	pepsimaxman	Coders. General	13	15 July 2019 11:17
Fastest Amiga Games	Djay	Nostalgia & memories	73	09 June 2016 21:42
Fastest unZIP on 030?	Amiga1992	support.Apps	7	04 October 2010 01:15
fastest hardfile or directory ?	turrican3	New to Emulation or Amiga scene	10	06 June 2007 19:08
RNC Data File Depacker v2.1	Nico	New to Emulation or Amiga scene	8	05 May 2002 18:05

18 June 2019, 22:07	#24
DamienD Banned Join Date: Aug 2005 Location: London / Sydney Age: 47 Posts: 20,420	Indeed CrunchyDat is awesome, I use it all the time ...but it's only for program files; not data files.

19 June 2019, 09:43	#26
SteveMoody Registered User Join Date: Jan 2017 Location: South Wales Age: 51 Posts: 42	I don't know if this will be of any use to you. FC8 has been designed for fast decompression. https://www.bigmessowires.com/2016/0...decompression/

19 June 2019, 12:55	#27
Galahad/FLT Going nowhere Join Date: Oct 2001 Location: United Kingdom Age: 50 Posts: 8,986	I cant help feeling that packing iff files isnt getting the best out of Propack as iff is alread RLE packed. Have you tried converting an iff file to raw and then using propack on the raw file to see if you get a better result?

08 July 2019, 22:17	#30
roondar Registered User Join Date: Jul 2015 Location: The Netherlands Posts: 3,409	On the topic of the many files and filesystem overhead, IIRC the Amiga fast file system uses blocks of 512 bytes. So a file that is 513 bytes should take up the same space on disk as one that is 1024 bytes. This ended up being a problem for me as my most recent project had a bunch of really small files that were on the order of 64 bytes in size. So, I ended up I bundling all the small files that logically 'belonged together' into a bigger one and simply use offsets into the loaded data. This has actually saved me both disk space due to lower filesystem overhead, but also due to RNC performing somewhat better on bigger files.

08 July 2019, 22:23	#32
roondar Registered User Join Date: Jul 2015 Location: The Netherlands Posts: 3,409	That would also work. But it's a lot of effort for a rather peculiar use case

08 July 2019, 23:16	#37
Galahad/FLT Going nowhere Join Date: Oct 2001 Location: United Kingdom Age: 50 Posts: 8,986	Well unfortunately, I never could use a depacker for realtime depacking of individual tracks within a certain game i've cracked, it would slow the game down on 68000 too much. So I had to tackle the game in question in an entirely different way which made realtime depacking rather moot really. All going well, this poxy game will be released next week, just need to do a conversion routine for it and then its done. I have to say, its been a hell of a lot of work to get this game to operate because of the way the programmer made it as difficult as possible to crack in anything like a normal way, so kudos to him for trying, just a shame the game doesn't really warrant that kind of effort, but back in the day, this one would have been quite irritating to crack quickly because of the way its structured, but i guess that was entirely the point!

08 July 2019, 23:19	#38
roondar Registered User Join Date: Jul 2015 Location: The Netherlands Posts: 3,409	Wasn't there a thread somewhere talking about using standard ZIP were someone managed to create a decruncher for ZIP files that was fast enough to keep up with the floppy drive even on a 68000? Anyway, I like the 'make big file, use offsets' approach. Easier than making your own file system, but still lets you avoid block size issues.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)