Correct executable hunk parser for 1.3/3.1?

[Photon]

@Don_Adan: I'm looking for a single piece of code that when run on OS1.3-3.1 would load and execute any executable object file as if typed by the user in the CLI on the respective OS version.

@Thomas: This means I can't use a piece of code that just works on 2.04+

I say experienced coders just because of a few checks that would need to be performed when executed on <2.0.

(Now, I mention compressors only because they would have had to deal with many such executable files from different OS versions, and they would have had these checks so the executable still works after being compressed. There's no hunk optimization goal for my question.)

@Spiv: Chopping off the file end is a hack, but since all OS versions still run it, my piece of code must run it.

So the question really is unchanged from my original post. Code (or flowchart) to load any exe, run it correctly, works on 1.3-3.1.

If it really isn't solved before, maybe it could be a nice coding effort. Or almost solved, but we discover that we must do a few extra checks.

[Thomas Richter]

Originally Posted by Photon

@Thomas: This means I can't use a piece of code that just works on 2.04+

Correct, you cannot run a code that was designed and written for >2.04 in mind on a 1.3 system. If you want to run such code, get 2.04.


I wonder why that surprises you.

Originally Posted by Photon

So the question really is unchanged from my original post. Code (or flowchart) to load any exe, run it correctly, works on 1.3-3.1.

And my answer is unchanged from my previous answer. Use LoadSeg(). You cannot run code designed for 2.04 on 1.3. Thus, if running a program is part of your problem, why does loading it (without being able to run it) solve the problem?

Originally Posted by Photon

If it really isn't solved before, maybe it could be a nice coding effort. Or almost solved, but we discover that we must do a few extra checks.

There is no solution to your problem. If you want to load and run a 2.04 program, you need to get 2.04, and then you also get support for all hunk types 2.04 support.


I'm sorry, either you explain very bad, or your request makes no sense.

[Bruce Abbott]

Back to the OP,

Originally Posted by Photon

DOS LoadSeg works fine, but I'm looking out for pitfalls from experience.

If DOS LoadSeg works fine then there is no reason not to use it.

Are there any pitfalls? Nobody here seems to know of any.

Some executable files have multiple hunks, decompression etc. The last hunk might have the last 0 longword (IIRC) removed to save 4 bytes, and they still work.

They still work because if they didn't it would be a fatal bug, which the coder would be forced to fix. What's left after that are latent bugs that don't prevent the OS from loading them, but might be a problem for other implementations. Therefore your best choice for compatibility is to use the same code as the OS, ie. LoadSeg.

So what you are really asking is does anyone have experience of programs that don't load in a particular OS due to bad hunk formatting. I can't think of any, but there probably are a few. There is a lot of bad stuff out there, but unless it is something really important to me I don't generally bother trying to find out why it fails (life's too short to spend it trying to fix other people's crappy code!).

having said that, a discussion on differences between hunk formats and possible issues with them could be quite informative (I found a few issues with my own implementation - required for my disassembler - which have actually been touched on here). So if anyone does have something to share please tell us, because it might be useful to know.

[mr.spiv]

Originally Posted by Photon

(Now, I mention compressors only because they would have had to deal with many such executable files from different OS versions, and they would have had these checks so the executable still works after being compressed. There's no hunk optimization goal for my question.)

I would challenge that. At least quite few relocators were able to handle very basic hunk types. Anything newer/exotic would crash'n'burn. The same goes with variety of broken cli pointer patchers etc They are typically a snapshot what was available at that time without too many worries about forward compatibility.
Anyway, today when we have all info from 3.x to 1.x (and Thomas with all bugs and quirks history) such loader is doable. Whether running such relocated program is possible is another issue (e.g. after loading 3.1.4 targeted exe in 1.2 system).

[phx]

Originally Posted by Photon

I'm looking for a single piece of code that when run on OS1.3-3.1 would load and execute any executable object file as if typed by the user in the CLI on the respective OS version.

I read that sentence and I think I understand what you mean now, but I don't see the goal of your effort. Sure, if you want to run executables which have 2.0 or 3.0 hunks, then you cannot use LoadSeg() under 1.3 and have to build a hunk parser yourself.

But a program which normally doesn't load under 1.3 is quite likely to be incompatible anyway, or do I miss something?

[Don_Adan]

Yes, you can load any exe (some need loading fixes for 3.0+ relocs) on every Amiga kick, but you can not run (without crash or quit) any exe on every Amiga kick or every Amiga config. Some exe files needs kick 2.0 or 3.0+ routines and has no extra checks. Some has big bss sections like f.e 16MB. Too many dependencies exists for loading and running all files. I used LoadSeg many times in my players, and it works very good for all tested kicks (1.3 ,2.0, 3.0, 3.1). But i never load and run word style relocs. First because ASMone or Devpac never created these hunks. Second, because i dont know single Amiga file from 1985-1996 years which used these hunks. Perhaps I found first file within word style relocs in 1998 year or maybe even 2000 year, it was DeliTracker2 external player created by Sunbeam/Digital Corruption. DT2 needs kick 2.0+, and works under kick 2.0. But this player simple can not be loading under kick 2.0. For me you can use LoadSeg routine, but before calling you can scan hunks for unsupported relocs for kick 1.3/2.0 and refuse to call LoadSeg on kicks 1.3/2.0. Or you can patch LoadSeg if you want to run exes with non standard Amiga relocs.

[Photon]

This code must load the exe into memory correctly under OS1.3-3.1 and then call the first instruction.

Whatever happens after that is of no concern whatever.

If specialized hunk types were added after 1.3, they will either be absolutely required for the correct functioning of the program or not. If not, they must still be parsed correctly. If so, they can return an error message on an OS version not supporting it, but detection must determine support. At a future time, code from that OS version can be added to the code.

This is the spec.

[Bruce Abbott]

Originally Posted by Photon

If specialized hunk types were added after 1.3, they will either be absolutely required for the correct functioning of the program or not. If not, they must still be parsed correctly. If so, they can return an error message on an OS version not supporting it, but detection must determine support.

So you want to be able to load a program under eg. KS1.3 which has a hunk format the OS doesn't recognize, but which might run if it could be loaded. If it can't be loaded then an error should be returned that identifies why.

IOW, you want a replacement for LoadSeg which supports hunk formats that the OS it's running under may not, thus allowing users to attempt to run programs that were not designed for that OS (or at least have a incompatible hunk format and so have never been tested on that OS).

I guess there could be a use for that, but I suspect the vast majority of programs that don't have a compatible hunk format won't be compatible for other reasons too.

I faced a sort of similar situation with my disassembler, which I wanted to be compatible with KS1.3 to permit disassembling any program on older machines. I needed a custom LoadSeg anyway to retain the relocation information, so I just took some published code and modified it to suit. Then I discovered that a few programs use drel32 and reloc32short, so I added code to handle them. I also added a read cache to speed up parsing symbol hunks (which can slow down loading a lot because every entry has to be parsed individually to get to the end of the hunk). To date I have not found any program that won't load, though I don't load overlay hunks because my disassembler doesn't know what to do with them (one day, maybe...).

My code is no good for your application, but this might:-

https://sintonen.fi/src/loadseg/LS.asm

[Thomas Richter]

Originally Posted by Bruce Abbott

My code is no good for your application, but this might:-

https://sintonen.fi/src/loadseg/LS.asm

Nope. Masks bits incorrectly and will accept hunk types that aren't proper, and does not handle overlays, which do exist, in particular under 1.3.

[Bruce Abbott]

Originally Posted by Thomas Richter

Nope. Masks bits incorrectly and will accept hunk types that aren't proper, and does not handle overlays, which do exist, in particular under 1.3.

I fully expected that code might need modifications. Please tell us more about the incorrect mask bits and 'improper' hunk types.

Yes, that code doesn't handle overlays, but how could it? I presume the overlay manager in the loaded code would call OS functions to switch in overlays, right? Would you have to patch the system to redirect those calls to your own code?

But Photon's spec is:-

This code must load the exe into memory... and then call the first instruction.

Whatever happens after that is of no concern whatever.

If the overlay manager can't handle it, that's its fault!

[jotd]

"Piru" isn't exactly a noob. Besides if this code needs fixing, let's improve it instead of criticizing.

[Photon]

Originally Posted by Bruce Abbott

So you want to be able to load a program under eg. KS1.3 which has a hunk format the OS doesn't recognize

The idea is that this new code can recognize all hunk types from 1.3 to 3.1 and knows how to parse them safely according to the OS version the new code is run on. If parsed and detected as impossible, not execute & return error code.

Originally Posted by Bruce Abbott

IOW, you want a replacement for LoadSeg

KS1.3 parsed hunks to run 1.3 executables just fine. So IDK if it needs a LoadSeg replacement. The idea is that this new code does this, self-contained. I'd draw the line at calling absolute ROM addresses.

I don't want to limit myself needlessly here. I.e., if the exe uses hunks available on 1.3, it should parse and execute on 1.3-3.1. And if code inside the hunks relies on 1.4, 3.1, 3.9 (or RTG, Vampire, you name it) and crashes that's fine.

Imagine an exe compressor, or a disassembler, or heck, an IFF image loader, that doesn't require a certain OS version to do their job. That's the goal. Run an exe.

[Bruce Abbott]

Originally Posted by Photon

Imagine an exe compressor, or a disassembler, or heck, an IFF image loader, that doesn't require a certain OS version to do their job. That's the goal. Run an exe.

I can imagine an exe compressor or a disassembler needing a custom loadseg (obviously) but not an image loader or other application. As for loading and running standard exes in a CLI environment, I can see no reason not to use the OS.

But I am against exe compressers anyway, since they sometimes cause problems and we have plenty of storage space today.

If there are executables out there that have problems loading under the normal OS code then we should look at fixing them, not develop a custom loader to accommodate their flaws.

[Photon]

Originally Posted by Bruce Abbott

I can imagine an exe compressor or a disassembler needing a custom loadseg (obviously) but not an image loader or other application. As for loading and running standard exes in a CLI environment, I can see no reason not to use the OS.

The image loader would obviously support a different hunk format, I was hoping you'd see that it would support old images as well as new ones and see the similarity to exes which are just a file, too.

Originally Posted by Bruce Abbott

But I am against exe compressers anyway, since they sometimes cause problems and we have plenty of storage space today.

Not on floppy disks. They have a certain limit. Some exes enter competition and come compressed for this reason, not distribution media, Internet (or Internet connected PC) availability, etc. The new code must run every compressed exe, as long as the decompression code cannot be shown to contain an illegal instruction for the by the exe intended CPU. Again: Correct code.

Originally Posted by Bruce Abbott

If there are executables out there that have problems loading under the normal OS code then we should look at fixing them, not develop a custom loader to accommodate their flaws.

I actually described how this would be handled in my previous comment. All exes would have loaded under the OS version intended. Existing 40,000+ exes from 2018, 2008, 1998, or 1988 will not have to be fixed to be run under the same OS version, if the code is correct.