English Amiga Board


Go Back   English Amiga Board > Coders > Coders. General

 
 
Thread Tools
Old 02 March 2017, 22:35   #1
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
Open source decompressors for many "old" amiga compressors

Hi,

I wanted to access/decompress some old Amiga compressed files (on a PC) and I noticed that many of the compressed files had compression formats that do not have proper cross-platforms tools. Some things I found (dernc, defimp, unsqsh) did not look pretty, were missing rudimentary error checking, exploded on 64-bit, were pure 68k to C translations or had shaky licensing terms. Some of the stuff did not have anything available on non-amiga.

So what to do? Wel it took me three weeks but I wrote my own (hopefully better) ones. Currently it contains CrM!, CrM2, Crm!, Crm2, IMP (and variants), RNC1 (old and new), RNC2, TPWM, XPK with MASH, NONE, SQSH modes. (This was the stuff I needed)

After checking on the files what I have (~10k of them) I'm convinced that what I have is at least as good as what is available otherwise. Some files I suspect are broken and nothing helps. In some cases I'm missing the reference what the uncompressed file should be. And in some case I believe my decompressor is actually doing better job than the tools I found. I'm sure there are still real bugs as well

I tagged the license as BSD 2-clause and uploaded it to github:

https://github.com/temisu/ancient_format_decompressor

Enjoy. (If you have feedback, I'd love to hear that as well)
temisu is offline  
Old 02 March 2017, 23:20   #2
emufan
Registered User
 
Join Date: Feb 2012
Location: #DrainTheSwamp
Posts: 4,546
welcome, cool project.

on my cygwin 32 bit installation i had to "fix" something:
Code:
diff -u -p1 ancient-orig/HuffmanDecoder.hpp ancient-new/HuffmanDecoder.hpp
--- ancient-orig/HuffmanDecoder.hpp
+++ ancient-new/HuffmanDecoder.hpp
@@ -11,2 +11,4 @@

+#define ssize_t int
+
 template<typename T>
Only in ancient-new: IMPDecompressor.o
diff -u -p1 ancient-orig/main.cpp ancient-new/main.cpp
--- ancient-orig/main.cpp
+++ ancient-new/main.cpp
@@ -76,3 +76,3 @@ Buffer *readFile(const std::string &file
                file.seekg(0,std::ios::end);
-               size_t length=file.tellg();
+               size_t length = (size_t) file.tellg();
                file.seekg(0,std::ios::beg);
tested the compiled binary with an Imploder (fimp) compressed data file and it does work
emufan is offline  
Old 03 March 2017, 13:14   #3
WayneK
Registered User
 
Join Date: May 2004
Location: Somewhere secret
Age: 45
Posts: 294
There was (is) an existing RNC1/2 depacker in C from 199x, done around the PSX/N64 days, which was widely used by various cracking groups on those platforms.

But cool work, might come in useful sometimes, thanks for releasing it!
WayneK is offline  
Old 03 March 2017, 17:27   #4
pants
Registered User

 
Join Date: Feb 2017
Location: fastmem
Posts: 53
Awesome stuff! This was on my list of 'things I must find or write' - will try it out this weekend for sure.

Quote:
Originally Posted by temisu View Post
Enjoy. (If you have feedback, I'd love to hear that as well)
Because you asked...

It would be great if there was an option to pass in a binary blob (scan from start until it hits a signature, decompress, dump, continue until blob ended - one input, potentially many outputs). Not that I would feed it an entire .adf. Oh, no. Of course not. That would be cheating... >.> <.<

Thanks again, this already is a big timesaver...
pants is offline  
Old 03 March 2017, 18:28   #5
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
Quote:
Originally Posted by emufan View Post
on my cygwin 32 bit installation i had to "fix" something:
Good catch. Last minute brainfart on my side. I should not use ssize_t and I did not test the latest on 32bit...

Quote:
Originally Posted by WayneK View Post
There was (is) an existing RNC1/2 depacker in C from 199x, done around the PSX/N64 days, which was widely used by various cracking groups on those platforms.
I could not find such thing. Only thing I could find were essentially a same dernc utility modified and with different copyrights (completely missing old RNC1 and RNC2 as well). I filed them under "shaky license". Anyway, does not matter anymore, now I made a clean one already

Quote:
Originally Posted by pants View Post
It would be great if there was an option to pass in a binary blob (scan from start until it hits a signature, decompress, dump, continue until blob ended - one input, potentially many outputs). Not that I would feed it an entire .adf. Oh, no. Of course not. That would be cheating... >.> <.<
Well, if you look the code it works by buffer abstraction (instead of files or wild pointers) so scanning should be rather easy to add. One thing is that probably then you would need catch-it-all sub-decompressor for XPK since the current implementation does not like if it encounters unimplemented compressor
temisu is offline  
Old 03 March 2017, 21:34   #6
emufan
Registered User
 
Join Date: Feb 2012
Location: #DrainTheSwamp
Posts: 4,546
Quote:
Originally Posted by temisu View Post
Good catch. Last minute brainfart on my side. I should not use ssize_t and I did not test the latest on 32bit...
tested the new version. does now compile without errors
emufan is offline  
Old 13 March 2017, 19:52   #7
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
Added now a few more things for XPK: GZIP and wired CrunchMania through XPK as well (CMR2, CRMS)

I still plan to add the trivial ones for XPK: CBR0, CBR1, and RLEN, as well as wire the IMPL.

(I did not finish binary scanner yet though, although I have it under work)

If you have some feedback/wishlist let me know
temisu is offline  
Old 17 March 2017, 22:03   #8
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
Well, I'll continue a bit more my monologue here.

Cholok provided me with a excellent references of XPK-sublibs with sources. Now I know the details that I had to guess earlier. I'm thankful

This means I have been able to implement support for a many algorithms, and there is still many more to come.

Also now I finished a scanner for embedded streams in files that can be applied to whole directories...
temisu is offline  
Old 17 March 2017, 22:06   #9
ascp
Pastafarian

 
Join Date: Sep 2010
Location: Uppsala/Sweden
Posts: 290
Cool project temisu!
ascp is offline  
Old 17 March 2017, 23:19   #10
emufan
Registered User
 
Join Date: Feb 2012
Location: #DrainTheSwamp
Posts: 4,546
now it got its correct filename

if you dont mind, you can add some diskimage decruncher from uaeunp. DMS, WRP and DSQ.

amiga-stuff.com lists alot of amiga crunchers and archivers - most without sources.
some have sourcecode, for example: zoosrc2-10.lha
maybe they will help.

Last edited by emufan; 17 March 2017 at 23:24.
emufan is offline  
Old 08 April 2017, 07:54   #11
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
I have reached a big milestone finally which is that I surpassed xpk_User when it comes to supporting different decompressors. Now there are just 24 XPK-decompressors supported (+some non-XPKs as well)

Final piece of that puzzle was SHRI which made me almost cry, the horror

There are a few popular comps left like LHLB, BLZW and BZP2 and rest is just esoterics. Oh well, now when I started I can add them all as well

Quote:
Originally Posted by emufan View Post
if you dont mind, you can add some diskimage decruncher from uaeunp. DMS, WRP and DSQ.
DMS I need myself in one form or another so I have to check what is the current quality of the undms, whether it makes sense to reimplement it properly. Others? not sure yet
temisu is offline  
Old 08 April 2017, 17:03   #12
acd2001
Zone Friend
acd2001's Avatar
 
Join Date: Nov 2005
Location: Italy
Posts: 41
Use stat, not d_type

The d_type member is not portable, MinGW in Windows bombs errors:

main.cpp: In lambda function:
main.cpp:227:14: error: 'struct dirent' has no member named 'd_type'; did you me
an 'd_name'?
if (de->d_type==DT_DIR)
^~~~~~
main.cpp:227:22: error: 'DT_DIR' was not declared in this scope
if (de->d_type==DT_DIR)
^~~~~~
main.cpp:230:21: error: 'struct dirent' has no member named 'd_type'; did you me
an 'd_name'?
} else if (de->d_type==DT_REG) {
^~~~~~
main.cpp:230:29: error: 'DT_REG' was not declared in this scope
} else if (de->d_type==DT_REG) {
^~~~~~
Makefile:21: recipe for target 'main.o' failed
mingw32-make: *** [main.o] Error 1
acd2001 is offline  
Old 08 April 2017, 19:13   #13
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
Quote:
Originally Posted by acd2001 View Post
The d_type member is not portable, MinGW in Windows bombs errors:
Should be fixed now. I don't have mingw myself, so I cant be sure.

However, since mingw does not have lstat either now it will follow symlinks, thus be careful...
temisu is offline  
Old 08 April 2017, 22:01   #14
Leffmann
 
Join Date: Jul 2008
Location: Sweden
Posts: 2,265
Thanks, this is pretty useful to have.
Leffmann is offline  
Old 09 April 2017, 13:50   #15
pants
Registered User

 
Join Date: Feb 2017
Location: fastmem
Posts: 53
Quote:
Originally Posted by temisu View Post
I have reached a big milestone
Really appreciate your work on this temisu, thanks again!
pants is offline  
Old 09 April 2017, 15:20   #16
emufan
Registered User
 
Join Date: Feb 2012
Location: #DrainTheSwamp
Posts: 4,546
Quote:
Originally Posted by temisu View Post
DMS I need myself in one form or another so I have to check what is the current quality of the undms, whether it makes sense to reimplement it properly. Others? not sure yet
build here on cygwin x86 on 64 bit win - no problem.
"Others" - support for dsq and wrp would be of good use - "when" doesnt matter

#2) regarding DMS, there are some encrypted DMS files, you have to use some sort of brute-force.
uaeunp should have that method already - to get an idea.
emufan is offline  
Old 09 April 2017, 21:11   #17
Frog
Junior Member

Frog's Avatar
 
Join Date: Aug 2001
Location: France
Posts: 1,300
i don't know if it can be useful but i have used in the past http://wiki.xentax.com/index.php/Jaeder_Naub on pc, i'm not sure if it allow amiga packers
Frog is offline  
Old 01 October 2017, 14:44   #18
Frog
Junior Member

Frog's Avatar
 
Join Date: Aug 2001
Location: France
Posts: 1,300
some packers/depackers on amiga and PC : ftp://ftp.spaceballs.no/public/Proje...er_Test_Suite/
Frog is offline  
Old 01 October 2017, 16:23   #19
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
Quote:
Originally Posted by Frog View Post
some packers/depackers on amiga and PC : ftp://ftp.spaceballs.no/public/Proje...er_Test_Suite/
Well, I have most of those packers already. but there are example files. I really like that especially when I do not have exe packed files yet (or support for those) for any format.

Funny thing though when I run scan with the current tool:

Code:
Found compressed stream at 548, size 129968 in file Amiga_Packed_File_Examples/Amiga_UnknownCruncher01 with type 'RNC1: Rob Northen RNC1 Compressor (old)'
That is not not so unknown anymore...
temisu is offline  
Old 07 November 2017, 23:36   #20
temisu
Registered User

 
Join Date: Mar 2017
Location: Tallinn / Estonia
Posts: 31
I added DMS support now. I think the implementation is as good or better than anything else out there for DMS

It is disturbing to see that there are corner cases which are handled differently between xdms and uae port. Unfortunately *files what I have do not exercise those corner cases. But I found some others that were not handled correctly in neither and tried to handle them properly. Please try it.

And then there is files like these: http://aminet.net/package/demo/disk/Eradication

It is obviously broken by the compressor already. But I think it could be fixed in decompression stage. However, I would need more than one datapoint to support my theory

Also I added password bypassing in the code. Already 16-bit key is ridiculously weak but I even added plaintext attack to it: if it is bootable image i.e. starting with 'DOS' the password bypassing is lightning fast. (if not, it is just fast)

Banner and file_id.diz will be thrown away. Who cares

Anyway. I need more files to test. Especially password protected and currently files that cant be decompressed. If someone has those, I'd love if you zone em...

*I have currently 1670 dms files. Basically I downloaded everything from aminet (I'm glad they did not IP-block me) as well as some other sources I could find. Googling password protected dms-files lead to me for searching amiga format subscriber disks. But I only found 1 of those
temisu is offline  
 


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Similar Threads
Thread Thread Starter Forum Replies Last Post
"Voices8" 8 Channel Soundtracker "DemoSongI" song - "This is the Amiga with 8 Voices" DemosongIHunter request.Music 44 27 October 2019 20:04
"Reminder "Lincs Amiga User Group aka "LAG" Meet Sat 5th of January 2013" rockape News 4 30 January 2013 01:06
After creating OS 3.9 Emergency Disk: Cannot open "cd.device" unit 2 Snowwie support.Other 2 31 March 2012 15:44
A source of cheap LCD for "small Amiga" projects... mabus support.Hardware 2 14 February 2008 01:25
Who hides behind the handle "Source / The Source"? andreas Retrogaming General Discussion 15 04 January 2005 17:02

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +2. The time now is 02:08.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Page generated in 0.09253 seconds with 13 queries