Ambermoon open source remake in progress

[Don_Adan]

Originally Posted by dlfrsilver

If i use F2 inside LOB packer 3.7, it encodes the file to LOB format.

However, you talk about CRYPTING the new packed file. How was it done ?

and sorry to say this, but the LOB packer v3.7 doesn't allow automatic packing of multiple files. This means that for the file of archive 2Map_texts.amb, you need to pack each file individually (what a nightmare ! )

If you want to compare original packed file with LOB 3.7 packed use Track2File, choose right decoding and saving without depacking.
I dont remember exactly, it was many years ago and im still dead to check this. But perhaps packed data after 8 or 12 bytes was eored using value from 4 or 8 offset (it was length). Find decoding code in Track2File source.

http://aminet.net/package/disk/misc/Track2File_src

[alexh]

Interestingly XFDdecrunch crashes when unpacking Pyradcor's LOB packed files using 68000 CPU but works with 68020 CPU.

[dlfrsilver]

Originally Posted by alexh

Interestingly XFDdecrunch crashes when unpacking Pyradcor's LOB packed files using 68000 CPU but works with 68020 CPU.

I have discovered that the files where incorrectly encoded for some. And the LOB header was missing from the 1st subfile.....

I sent AlexH the file 1Map_texts.amb fully repacked from the v1.10 done by pyrdacor. Let's see if it crash.

[Pyrdacor]

Thanks to Nico Bendlin we found the reason. The pre-68020 CPUs don't allow odd (non word-aligned) data addresses. And as Ambermoon reads the data as long-words it crashes on such CPUs. This happens when either the uncompressed or compressed data size is odd.

I will adjust my packer tomorrow to fix this and then we try if it works with the older CPUs.

[Pyrdacor]

Originally Posted by dlfrsilver

I have discovered that the files where incorrectly encoded for some. And the LOB header was missing from the 1st subfile.....

I sent AlexH the file 1Map_texts.amb fully repacked from the v1.10 done by pyrdacor. Let's see if it crash.

This is just not true. Otherwise more recent CPUs could not load the data correctly.

Every compressor can work and compress differently. There is no "correct" compression output or encoding. Important is only that the output can be decompressed by the decompressor. When the LOB header is missing, this is not an error as well. The AMNP container format has optional LOB compression so it is totally valid to not LOB compress some sub-files. For small files the compression often makes no sense. My packer only uses LOB if it is necessary (some containers require it) or reduces the size in relation to uncompressed data. For example the packer won't compress files smaller than 20 bytes. But as I said this is totally valid for AMNP files.

[Neil79]

Ambermoon.net and my love of the Amiga game Ambermoon - Article dev remake special by Pyrdacor

http://www.indieretronews.com/2021/0...miga-game.html

[Don_Adan]

Originally Posted by Pyrdacor

This is just not true. Otherwise more recent CPUs could not load the data correctly.

Every compressor can work and compress differently. There is no "correct" compression output or encoding. Important is only that the output can be decompressed by the decompressor. When the LOB header is missing, this is not an error as well. The AMNP container format has optional LOB compression so it is totally valid to not LOB compress some sub-files. For small files the compression often makes no sense. My packer only uses LOB if it is necessary (some containers require it) or reduces the size in relation to uncompressed data. For example the packer won't compress files smaller than 20 bytes. But as I said this is totally valid for AMNP files.

I dont remember if Ambermoon used standard or modified LOB depacker. But for decrunch routine LOB header can be important. Because 1st byte is number of passes. And 5th byte is type of used packer. So 1LOB5 is equal to use 5th type of depacker in one pass. But exist f.e 2LOB6 which used 6th depacker in two passes. Second pass can be f.e swap word (some files can be better packed if data are swapped).

And yes all word/longword data must be stored at even addreses, if no special read/write word/longword routine is used on 68000. Odd lenghts/offsets must be padded.

[Pyrdacor]

Originally Posted by Don_Adan

I dont remember if Ambermoon used standard or modified LOB depacker. But for decrunch routine LOB header can be important. Because 1st byte is number of passes. And 5th byte is type of used packer. So 1LOB5 is equal to use 5th type of depacker in one pass. But exist f.e 2LOB6 which used 6th depacker in two passes. Second pass can be f.e swap word (some files can be better packed if data are swapped).

You got me wrong. If the header is not present for a sub-file this of course also means that the file is not compressed. Of course you need the header if the sub-file is compressed. Otherwise you wouldn't even know that the file is compressed.

The AMNP container only forces encryption (JH) but the compression (LOB) is optional for each sub-file. So either you have LOB+JH files or just JH files. In the latter case, there is no LOB header as the file is not compressed at all.

[dlfrsilver]

Originally Posted by Pyrdacor

Thanks to Nico Bendlin we found the reason. The pre-68020 CPUs don't allow odd (non word-aligned) data addresses. And as Ambermoon reads the data as long-words it crashes on such CPUs. This happens when either the uncompressed or compressed data size is odd.

I will adjust my packer tomorrow to fix this and then we try if it works with the older CPUs.

I discovered that the LOB packer just bombs over some of your files.

My test was simple :

1) the original tools must be able to process your files
2) I used Metallic AMB extractor, and after checking the files => i found errors.
3) I use the original LOB packer (running on Atari ST) to decrunch your files => it crashed.
4) I compared as asked by Don Adan v1.07 files crunched/decrunched with LOB packer v3.70 to see if the recrunch was identical.
When i did the same between v1.07 and v1.10 on the same files, the crunched content was not matching.

In the end, it only worked on 68020 by luck.

The LOB packer compression on Ambermoon was made to work on 68000 & 680x0.

I have just finished to build a new set of ADF of your patched version v1.10 with all files crunched with LOB packer type 6 (MSS scheme).

As a result, the game fully works with A500 1Mb + 2 drives Yay !

For those who want to try : https://we.tl/t-2OAXPwKhkO

[SquawkBox]

Works here with my setup, congrats guys! Maybe I'll wait until Pyrdacor reprocesses the files himself and updates the ADF set from his repository. Having said that, it seems you nailed it dlfr.

[Pyrdacor]

@dlfrsilver: It doesn't work by luck. My implementation is fine beside the word alignment which is basically adding a 0 byte to the end of files with odd size. It has nothing to do with the algorithm but is rather caused by a hardware limitation and the way Ambermoon implemented the decruncher/data copy routine.

I explained this to you at several places and Nico did too. So if you would have been patient for 1 day it would have worked with my packer too. But if you managed to do it the hard way, fine.

And again: Crunched content has not to be identical to be valid.

I will create a 1.10 version today as well with my fixed packer. Then we can check if it works.

Anyway, thanks for your effort.

[Pyrdacor]

To show that crunched content can differ, here an example:

To compress the data 44 44 44 44 you can do so with

f0 44 44 44 44

or

80 44 03 01.

The first one just uses 4 literals without an encoded match which is totally valid. The decompressor can process this and will output 44 44 44 44.

The second one uses 1 literal 44 and then a match which has a length of 3 and an offset of 1 (go back this amount of bytes and read from there).

So even this tiny example can be encoded in two ways and both are totally valid. It is a matter of how the compressor can find and encode matches.

For example the sequence 11 11 11 11 22 33 can be encoded in a way that the first 4 bytes are expressed as a literal and a match of 3 like in the previous example. However if the whole sequence was present before inside the data it would be even better to express the whole sequence as one single match (back reference).

So dependent on the compressor, there are multiple ways to compress or encode data. Some are better and some are not. And some are only better in specific scenarios.

[dlfrsilver]

Yes of course. The thing is that ambermoon was made to run on both cpus.

I have applied all the fixes you made referenced in the excel sheet, minor 2, for which I had no file number.

[Pyrdacor]

Originally Posted by dlfrsilver

Yes of course. The thing is that ambermoon was made to run on both cpus.

The compression has nothing to do with the CPU. Every CPU can handle all kind of compressions. The 68000 can handle other compressions as well. But it can't handle data with odd data addresses. But this is out of scope of the compression algorithm.

[dlfrsilver]

Originally Posted by Pyrdacor

The compression has nothing to do with the CPU. Every CPU can handle all kind of compressions. The 68000 can handle other compressions as well. But it can't handle data with odd data addresses. But this is out of scope of the compression algorithm.

Of course any CPU can handle all kind of compressions, we have tons of them on Amiga.

The MSS compression use a form of structure. If this structure is absent, it's not a problem on A1200, but on A500 it leads to odd data addresses problems.

And what matters is that the A500 programm recognize the pattern of the compression as devised by Jurie Hornemann. It's a sort of 'safety', otherwise you met the odd data addresses problem.

I talked with CFOU about the whole problem, and he confirmed me that it was a problem with the LOB compression.

[alexh]

Originally Posted by dlfrsilver

what matters is that the A500 programm recognize the pattern of the compression as devised by Jurie Hornemann.

Lothar Becks (LOB) not Jurie Horneman. And it's nothing to do with "recognising" it's maths and microprocessor architecture.

Originally Posted by dlfrsilver

It's a sort of 'safety',

If you mean by safety that the files need to be compressed with word alignment then they decompress on both CPU architectures then yes it's a "safety" ;-)

[Pyrdacor]

As expected the simple word-align fix did the job. It now runs on 68000 as well. I uploaded some test files to the bugfix folder here: https://github.com/Pyrdacor/Ambermoo...ish/Amberfiles

I did not change anything on the compression just adding a padding byte for odd data sizes.


I will pack 1.10 english and 1.09 german today (ADFs too).

[Pyrdacor]

Originally Posted by dlfrsilver

Of course any CPU can handle all kind of compressions, we have tons of them on Amiga.

The MSS compression use a form of structure. If this structure is absent, it's not a problem on A1200, but on A500 it leads to odd data addresses problems.

And what matters is that the A500 programm recognize the pattern of the compression as devised by Jurie Hornemann. It's a sort of 'safety', otherwise you met the odd data addresses problem.

I talked with CFOU about the whole problem, and he confirmed me that it was a problem with the LOB compression.

This is also not true. The MSS compression structure has nothing to do with the word-alignment. The compression algorithm itself works fine even with odd data addresses as you can see on the A1200. The (de)compression works byte by byte so word-alignment has no impact there.

The problem is that the Ambermoon executable (and maybe other loaders as well) copy the data as long-words before the decompression. This is the problem and it has nothing to do with the compression itself. You can also copy the data byte by byte and it would also work on a 68000.

Nico Bendlin documented it pretty good and he also posted the link before. Maybe you should finally read it.
https://gitlab.com/ambermoon/researc...is/compression

[dlfrsilver]

Originally Posted by Pyrdacor

The problem is that the Ambermoon executable (and maybe other loaders as well) copy the data as long-words before the decompression.

Exactly. This is what i found when i crunched the first file with the LOB packer.

This is the problem and it has nothing to do with the compression itself. You can also copy the data byte by byte and it would also work on a 68000.

Everything is tied to the decrunching routine. Of course, if you modify it to read byte per byte, the problem will be fixed.

Nico Bendlin documented it pretty good and he also posted the link before. Maybe you should finally read it.
https://gitlab.com/ambermoon/researc...is/compression[/QUOTE]

I've read the document yesterday.

[dlfrsilver]

Originally Posted by alexh

If you mean by safety that the files need to be compressed with word alignment then they decompress on both CPU architectures then yes it's a "safety" ;-)

This was exactly the sense of what i wrote. Without this safety, it crash on 68000, and nothing else