Open source decompressors for many "old" amiga compressors

[temisu]

Hi,

I wanted to access/decompress some old Amiga compressed files (on a PC) and I noticed that many of the compressed files had compression formats that do not have proper cross-platforms tools. Some things I found (dernc, defimp, unsqsh) did not look pretty, were missing rudimentary error checking, exploded on 64-bit, were pure 68k to C translations or had shaky licensing terms. Some of the stuff did not have anything available on non-amiga.

So what to do? Wel it took me three weeks but I wrote my own (hopefully better) ones. Currently it contains CrM!, CrM2, Crm!, Crm2, IMP (and variants), RNC1 (old and new), RNC2, TPWM, XPK with MASH, NONE, SQSH modes. (This was the stuff I needed)

After checking on the files what I have (~10k of them) I'm convinced that what I have is at least as good as what is available otherwise. Some files I suspect are broken and nothing helps. In some cases I'm missing the reference what the uncompressed file should be. And in some case I believe my decompressor is actually doing better job than the tools I found. I'm sure there are still real bugs as well

I tagged the license as BSD 2-clause and uploaded it to github:

https://github.com/temisu/ancient_format_decompressor

Enjoy. (If you have feedback, I'd love to hear that as well)

[emufan]

welcome, cool project.

on my cygwin 32 bit installation i had to "fix" something:

diff -u -p1 ancient-orig/HuffmanDecoder.hpp ancient-new/HuffmanDecoder.hpp
--- ancient-orig/HuffmanDecoder.hpp
+++ ancient-new/HuffmanDecoder.hpp
@@ -11,2 +11,4 @@

+#define ssize_t int
+
 template<typename T>
Only in ancient-new: IMPDecompressor.o
diff -u -p1 ancient-orig/main.cpp ancient-new/main.cpp
--- ancient-orig/main.cpp
+++ ancient-new/main.cpp
@@ -76,3 +76,3 @@ Buffer *readFile(const std::string &file
                file.seekg(0,std::ios::end);
-               size_t length=file.tellg();
+               size_t length = (size_t) file.tellg();
                file.seekg(0,std::ios::beg);

tested the compiled binary with an Imploder (fimp) compressed data file and it does work

[WayneK]

There was (is) an existing RNC1/2 depacker in C from 199x, done around the PSX/N64 days, which was widely used by various cracking groups on those platforms.

But cool work, might come in useful sometimes, thanks for releasing it!

[pants]

Awesome stuff! This was on my list of 'things I must find or write' - will try it out this weekend for sure.

Originally Posted by temisu

Enjoy. (If you have feedback, I'd love to hear that as well)

Because you asked...

It would be great if there was an option to pass in a binary blob (scan from start until it hits a signature, decompress, dump, continue until blob ended - one input, potentially many outputs). Not that I would feed it an entire .adf. Oh, no. Of course not. That would be cheating... >.> <.<

Thanks again, this already is a big timesaver...

[temisu]

Originally Posted by emufan

on my cygwin 32 bit installation i had to "fix" something:

Good catch. Last minute brainfart on my side. I should not use ssize_t and I did not test the latest on 32bit...

Originally Posted by WayneK

There was (is) an existing RNC1/2 depacker in C from 199x, done around the PSX/N64 days, which was widely used by various cracking groups on those platforms.

I could not find such thing. Only thing I could find were essentially a same dernc utility modified and with different copyrights (completely missing old RNC1 and RNC2 as well). I filed them under "shaky license". Anyway, does not matter anymore, now I made a clean one already

Originally Posted by pants

It would be great if there was an option to pass in a binary blob (scan from start until it hits a signature, decompress, dump, continue until blob ended - one input, potentially many outputs). Not that I would feed it an entire .adf. Oh, no. Of course not. That would be cheating... >.> <.<

Well, if you look the code it works by buffer abstraction (instead of files or wild pointers) so scanning should be rather easy to add. One thing is that probably then you would need catch-it-all sub-decompressor for XPK since the current implementation does not like if it encounters unimplemented compressor

[emufan]

Originally Posted by temisu

Good catch. Last minute brainfart on my side. I should not use ssize_t and I did not test the latest on 32bit...

tested the new version. does now compile without errors

[temisu]

Added now a few more things for XPK: GZIP and wired CrunchMania through XPK as well (CMR2, CRMS)

I still plan to add the trivial ones for XPK: CBR0, CBR1, and RLEN, as well as wire the IMPL.

(I did not finish binary scanner yet though, although I have it under work)

If you have some feedback/wishlist let me know

[temisu]

Well, I'll continue a bit more my monologue here.

Cholok provided me with a excellent references of XPK-sublibs with sources. Now I know the details that I had to guess earlier. I'm thankful

This means I have been able to implement support for a many algorithms, and there is still many more to come.

Also now I finished a scanner for embedded streams in files that can be applied to whole directories...

[ascp]

Cool project temisu!

[emufan]

now it got its correct filename

if you dont mind, you can add some diskimage decruncher from uaeunp. DMS, WRP and DSQ.

amiga-stuff.com lists alot of amiga crunchers and archivers - most without sources.
some have sourcecode, for example: zoosrc2-10.lha
maybe they will help.

[temisu]

I have reached a big milestone finally which is that I surpassed xpk_User when it comes to supporting different decompressors. Now there are just 24 XPK-decompressors supported (+some non-XPKs as well)

Final piece of that puzzle was SHRI which made me almost cry, the horror

There are a few popular comps left like LHLB, BLZW and BZP2 and rest is just esoterics. Oh well, now when I started I can add them all as well

Originally Posted by emufan

if you dont mind, you can add some diskimage decruncher from uaeunp. DMS, WRP and DSQ.

DMS I need myself in one form or another so I have to check what is the current quality of the undms, whether it makes sense to reimplement it properly. Others? not sure yet

[acd2001]

The d_type member is not portable, MinGW in Windows bombs errors:

main.cpp: In lambda function:
main.cpp:227:14: error: 'struct dirent' has no member named 'd_type'; did you me
an 'd_name'?
if (de->d_type==DT_DIR)
^~~~~~
main.cpp:227:22: error: 'DT_DIR' was not declared in this scope
if (de->d_type==DT_DIR)
^~~~~~
main.cpp:230:21: error: 'struct dirent' has no member named 'd_type'; did you me
an 'd_name'?
} else if (de->d_type==DT_REG) {
^~~~~~
main.cpp:230:29: error: 'DT_REG' was not declared in this scope
} else if (de->d_type==DT_REG) {
^~~~~~
Makefile:21: recipe for target 'main.o' failed
mingw32-make: *** [main.o] Error 1

[temisu]

Originally Posted by acd2001

The d_type member is not portable, MinGW in Windows bombs errors:

Should be fixed now. I don't have mingw myself, so I cant be sure.

However, since mingw does not have lstat either now it will follow symlinks, thus be careful...

[Leffmann]

Thanks, this is pretty useful to have.

[pants]

Originally Posted by temisu

I have reached a big milestone

Really appreciate your work on this temisu, thanks again!

[emufan]

Originally Posted by temisu

DMS I need myself in one form or another so I have to check what is the current quality of the undms, whether it makes sense to reimplement it properly. Others? not sure yet

build here on cygwin x86 on 64 bit win - no problem.
"Others" - support for dsq and wrp would be of good use - "when" doesnt matter

#2) regarding DMS, there are some encrypted DMS files, you have to use some sort of brute-force.
uaeunp should have that method already - to get an idea.

[Frog]

i don't know if it can be useful but i have used in the past http://wiki.xentax.com/index.php/Jaeder_Naub on pc, i'm not sure if it allow amiga packers

[Frog]

some packers/depackers on amiga and PC : ftp://ftp.spaceballs.no/public/Proje...er_Test_Suite/

[temisu]

Originally Posted by Frog

some packers/depackers on amiga and PC : ftp://ftp.spaceballs.no/public/Proje...er_Test_Suite/

Well, I have most of those packers already. but there are example files. I really like that especially when I do not have exe packed files yet (or support for those) for any format.

Funny thing though when I run scan with the current tool:

Found compressed stream at 548, size 129968 in file Amiga_Packed_File_Examples/Amiga_UnknownCruncher01 with type 'RNC1: Rob Northen RNC1 Compressor (old)'

That is not not so unknown anymore...

[temisu]

I added DMS support now. I think the implementation is as good or better than anything else out there for DMS

It is disturbing to see that there are corner cases which are handled differently between xdms and uae port. Unfortunately *files what I have do not exercise those corner cases. But I found some others that were not handled correctly in neither and tried to handle them properly. Please try it.

And then there is files like these: http://aminet.net/package/demo/disk/Eradication

It is obviously broken by the compressor already. But I think it could be fixed in decompression stage. However, I would need more than one datapoint to support my theory

Also I added password bypassing in the code. Already 16-bit key is ridiculously weak but I even added plaintext attack to it: if it is bootable image i.e. starting with 'DOS' the password bypassing is lightning fast. (if not, it is just fast)

Banner and file_id.diz will be thrown away. Who cares

Anyway. I need more files to test. Especially password protected and currently files that cant be decompressed. If someone has those, I'd love if you zone em...

*I have currently 1670 dms files. Basically I downloaded everything from aminet (I'm glad they did not IP-block me) as well as some other sources I could find. Googling password protected dms-files lead to me for searching amiga format subscriber disks. But I only found 1 of those