Correct executable hunk parser for 1.3/3.1?

[Don_Adan]

Unfortunelly hunk RELOC32SHORT doesnt works for kick 2.0 too.
Then using RELOC32SHORT hunk as DEFAULT is totally useless for vbcc.
Only OPTIONAL kick31 option has minimal sense for me. In general this is useless now (it generates only problems for new Amiga programmers/users).
Code optimisation has sense, hunk optimisation has maybe 1% sense.
With good code optimisation you have much less hunks too.
RELOC32SHORT has sense only if someone want to run executable on kick 3.0+ and dont want to use/add special checking routine for kick 3.0+.

[Photon]

Thomas: It's on a high level for sure, that's why a flow chart will suffice for the gotchas.

Originally Posted by phx

Implementation is trivial, but a bit of work to support everything.

This is a good summary of my original post.

Originally Posted by phx

What do you need?

Basically what Powerpacker needed through all those versions to finally get it right. Except I just have to load it into memory.

I envision a relatively short piece of code with some checks and some LoadSeg() calls, and possibly some handling of exes with the last hunk 0 longword removed to reduce filesize.

Currently I check for OS version and either LoadSeg() or use the Run command, but this new piece of code must be self-contained.

[mr.spiv]

Originally Posted by Photon

Basically what Powerpacker needed through all those versions to finally get it right. Except I just have to load it into memory.

Aha.. I might still have my old StoneCracker hunk parsers (up to OS 2.x) and hunk "packers" around Those were mostly based on "trial and error" approach and never worked with overlays. Also stupid stuff was added e.g. it was 'ok' to forget hunk ends (OS scatter loader still worked).

[Photon]

Originally Posted by mr.spiv

Aha.. I might still have my old StoneCracker hunk parsers (up to OS 2.x) and hunk "packers" around Those were mostly based on "trial and error" approach and never worked with overlays. Also stupid stuff was added e.g. it was 'ok' to forget hunk ends (OS scatter loader still worked).

I'm not sure if this is the same as removing the hunk end for intros to make them smaller, but since the OS started them fine with the last longword of the last hunk removed manually from the file, it's a hack I must support 100% (hopefully by tacking it onto an actually correct hunk parser )

[phx]

Originally Posted by Thomas Richter

LoadSeg() *is* the general solution

Either you or me fail to understand the intention of this thread.

Originally Posted by Don_Adan

Unfortunelly hunk RELOC32SHORT doesnt works for kick 2.0 too.

It definitely does! I'm not 100% sure about V36, but V37 (OS2.04) is guaranteed.

Code optimisation has sense, hunk optimisation has maybe 1% sense.

Any optimisation makes sense, especially when it comes for free.

[phx]

Originally Posted by Photon

Basically what Powerpacker needed through all those versions to finally get it right. Except I just have to load it into memory.

Ok. So it's a Powerpacker-like tool which has to process the hunk structure of executables and rewrite them? Then LoadSeg() is no solution, even if you run your tool on 3.1, because you probably want to keep or convert all hunks, like the relocations? I have the hunk-parsers in my linkers, but the code wouldn't be suitable for your project. I'm always glad to help, though.

Originally Posted by Photon

I'm not sure if this is the same as removing the hunk end for intros to make them smaller, but since the OS started them fine with the last longword of the last hunk removed manually from the file, it's a hack I must support 100%

I wonder which "removing the hunk end" feature you are constantly refering to?
Is it just the feature to have a larger hunk-size in the HUNK_HEADER than in the code/data hunk itself (data-bss hunks)? Or is the hunk really shorter than specified in both locations?

[Bruce Abbott]

Originally Posted by Photon

Trying to not reinvent the wheel, but apparently no-one has a hunk parser to share.

I have a hunk parser, but it may not be much good to you depending on what you want it for.

The goal is to load an executable into memory and execute it correctly...

Basically what Powerpacker needed through all those versions to finally get it right. Except I just have to load it into memory.

Which is exactly what the system Loadseg function does.

each OS has code that only has to work for that OS version, but my execution command must have code that works on 1.3-3.1 and be self-contained.

Why? Is the OS not available? Do you have to load random executables designed for higher OS's (that might not run in 1.3 anyway)?

and possibly some handling of exes with the last hunk 0 longword removed to reduce filesize.

Now there's a pointless hack.

[ross]

Originally Posted by phx

I wonder which "removing the hunk end" feature you are constantly refering to?

00000000   00 00 03 F3 00 00 00 00  00 00 00 05 00 00 00 00      ó            
00000010   00 00 00 04 00 00 00 01  00 00 00 01 40 00 00 01               @   
00000020   40 00 00 01 40 00 00 02  00 00 03 E9 00 00 00 01   @   @      é    
00000030   4E 71 4E 75 00 00 03 F2  00 00 03 EA 00 00 00 01   NqNu   ò   ê    
00000040   12 34 56 78 00 00 03 F2  00 00 03 E9 00 00 00 01    4Vx   ò   é    
00000050   4E 73 4E 73 00 00 03 F2  00 00 03 EA 00 00 00 01   NsNs   ò   ê    
00000060   9A BC DE F0 00 00 03 F2  00 00 03 EB 00 00 00 02   š¼Þð   ò   ë    
00000070   00 00 03 F2

to:

00000000   00 00 03 F3 00 00 00 00  00 00 00 05 00 00 00 00      ó            
00000010   00 00 00 04 00 00 00 01  00 00 00 01 40 00 00 01               @   
00000020   40 00 00 01 40 00 00 02  00 00 03 E9 00 00 00 01   @   @      é    
00000030   4E 71 4E 75 00 00 03 EA  00 00 00 01 12 34 56 78   NqNu   ê     4Vx
00000040   00 00 03 E9 00 00 00 01  4E 73 4E 73 00 00 03 EA      é    NsNs   ê
00000050   00 00 00 01 9A BC DE F0  00 00 03 EB 00 00 00 02       š¼Þð   ë    
00000060   00 00 03 F2

[Thomas Richter]

Originally Posted by phx

Either you or me fail to understand the intention of this thread.

If loading a binary to memory and performing the relocation is the problem, then LoadSeg() is the solution.

Originally Posted by phx

It definitely does! I'm not 100% sure about V36, but V37 (OS2.04) is guaranteed.

Actually, it doesn't strictly speaking. It works, but under the wrong hunk number. HUNK_RELOC32SHORT (under this very ID) only works from 3.0 up. Under 2.0, you need to use this HUNK_ABSRELOC16 or how it is called. This is due to a typo in LoadSeg(). Both hunk types work alike for LoadSeg, though they are intended for something different (or were).

[Thomas Richter]

Originally Posted by Photon

Basically what Powerpacker needed through all those versions to finally get it right. Except I just have to load it into memory.

Does the data come from file? Then LoadSeg() is the solution. If the data comes from memory, then InternalLoadSeg() is the right solution. Beware, however, that you cannot load overlays from memory (there is no file pointer) so that needs to be accounted for. For the purists: The most prominent overlayed program under 1.3 is DPaint, so no, I'm not making an academic argument. If you need to run under 1.3, then there is probably a solution for LoadSeg() as well, but I need to check, that works by manipulating the fle handle aka "SCB" (as Tripos puts it - the Stream Control Block). To the purists: The system "RUN" command works likewise, so it's not that I'm making up something new or creating a hacky solution - it's "Os approved". Whether that is any better or worse than writing LoadSeg yourself is probably a matter of taste, but it would seem easier to get right and there are less pitfalls.

Originally Posted by Photon

Currently I check for OS version and either LoadSeg() or use the Run command, but this new piece of code must be self-contained.

I'm sorry, but you seem confused. LoadSeg() is always existing and working, but does not load from memory (InternalLoadSeg() does), but Run does something completely different, and does certainly not work from memory, nor would it initiate workbench based programs correctly. I am sorry, but I guess before anyone can answer what the solution should look like, it requires a problem description. Such as where the hunk data is coming from, where it is going to.

[Thomas Richter]

Unfortunately, if the hunk binary comes from memory, LoadSeg() does not provide a solution under 1.3 and before (it does not go through the buffered I/O functions). You can loadseg from an open stream, though (i.e. loadseg can also operate on a file already open, and there is no need to provide a file name).

Anyhow, whether that matters or not is a matter of the problem statement.

[phx]

Originally Posted by ross

00000030   4E 71 4E 75 00 00 03 EA  00 00 00 01 12 34 56 78   NqNu   ê     4Vx

Oh, wow! So you weren't saying "removing the hunk end", but "removing the HUNK_END"!? I didn't even know about this dirty trick. Thanks (not that I would want to use it).

Originally Posted by Thomas Richter

If loading a binary to memory and performing the relocation is the problem, then LoadSeg() is the solution.

Sure. But I must say that Photon's intentions are still quite unclear to keep so many developers thinking in different directions...

Actually, it doesn't strictly speaking. It works, but under the wrong hunk number. HUNK_RELOC32SHORT (under this very ID) only works from 3.0 up.

You're right (and then also Don Adan ist right). Sorry. It's always good to be precise. When I wrote HUNK_RELOC32SHORT I was thinking about its function and not about the exact hunk-ID. It should be clear that I'm only using the ID $00003f7, which works since OS2.0.

Under 2.0, you need to use this HUNK_ABSRELOC16 or how it is called.

It's called HUNK_DREL32 (32-bit base-relative references), which can only occur in object files, so it is not a big problem.

But if you know the history of HUNK_ABSRELOC16 I would be interested (yes, off-topic). Who had the idea that 16-bit absolute addresses could ever make sense in an AmigaOS executable?

[ross]

Originally Posted by phx

Oh, wow! So you weren't saying "removing the hunk end", but "removing the HUNK_END"!? I didn't even know about this dirty trick. Thanks (not that I would want to use it).

, but I thought an example was more illustrative

[Thomas Richter]

Originally Posted by phx

Oh, wow! So you weren't saying "removing the hunk end", but "removing the HUNK_END"!? I didn't even know about this dirty trick. Thanks (not that I would want to use it).

I don't see how this could work. loadseg under 1.3 releases the segments if a hunk-end is not seen, it has a flag value for it. 3.1 forgot to set a secondary result code there, in 3.2 you'll get a ERROR_OBJECT_WRONG_TYPE as secondary result code. Edit: At least the last HUNK_END must be present. The ones in between actually do not do much difference.

Originally Posted by phx

It's called HUNK_DREL32 (32-bit base-relative references), which can only occur in object files, so it is not a big problem.

But if you know the history of HUNK_ABSRELOC16 I would be interested (yes, off-topic). Who had the idea that 16-bit absolute addresses could ever make sense in an AmigaOS executable?

Not much. The list of hunk types LoadSeg (as of 3.1 and also as of 3.2) supports is:

HunkName (which is ignored)
HunkCode
HunkData
HunkBss
HunkReloc32
HunkReloc16 (triggers an error)
HunkReloc8 (triggers an error)
HunkExt (triggers an error)
HunkSymbol (ignored)
HunkDebug (ignored)
HunkEnd
HunkHeader
HunkCont
HunkOverlay
HunkBreak
HunkDRel32 (wors as HunkReloc32short, on error)
HunkDRel16 (errors)
HunkDRel8 (errors)
HunkLib (errors)
HunkIndex (errors)
HunkReloc32Short (the real one)
HunkRelReloc32 (exotic nonsense for relocating bra.l and jmp (offset.l,pc)

Also all hunks derived from them with bits 30 or 31 (or both) set, to support the ATOM hunk modifiers. All unknown hunks create an error, unless bit 29 is set (HUNK_ADVISORY) which means that the hunk will also be skipped like a HUNK_DEBUG hunk.


Concerning HUNK_ABSRELOC16 I can only guess. The hunk type does not show up in the Os sources, and neither in the original tripos code, so it was probably reserved way before that. As you say, it does not generally make sense as even Tripos could not reserve memory exclusively in the first 32K (or last 32K).

[mr.spiv]

Originally Posted by phx

Oh, wow! So you weren't saying "removing the hunk end", but "removing the HUNK_END"!? I didn't even know about this dirty trick. Thanks (not that I would want to use it).

Yes, that kind of dirty tricks we practised How did I found out? By programming error afair..

[ross]

Originally Posted by mr.spiv

.. and hunk "packers" around ..

Hi mr.spiv
What do you mean by hunk packers? Hunk merging?
Sort of the relocation tables for subsequent delta compression?

Anyway picture me interested
I am convinced that I wrote dedicated code in the past, but probably lost by long time.

[mr.spiv]

Originally Posted by ross

Hi mr.spiv
What do you mean by hunk packers? Hunk merging?
Sort of the relocation tables for subsequent delta compression?

Anyway picture me interested
I am convinced that I wrote dedicated code in the past, but probably lost by long time.

Simple stuff (well then as a teenager it felt different).. afair my stuff did remove all possible "left over hunks", sort the reloc entries and then applied delta encoding to those resulting 1, 2 or 3 bytes reloc entries. The decruncher reloc code understood this proprietary hunk structure. This actually made a difference when crunching exe files. For a moment I thought it was unique lol

I recall I stated in some readme that there will a proper "hunk preprocessor" to merge hunks etc in the next release but that never materialized as many of my projects.

[Don_Adan]

Originally Posted by phx

Oh, wow! So you weren't saying "removing the hunk end", but "removing the HUNK_END"!? I didn't even know about this dirty trick. Thanks (not that I would want to use it).


Sure. But I must say that Photon's intentions are still quite unclear to keep so many developers thinking in different directions...

You're right (and then also Don Adan ist right). Sorry. It's always good to be precise. When I wrote HUNK_RELOC32SHORT I was thinking about its function and not about the exact hunk-ID. It should be clear that I'm only using the ID $00003f7, which works since OS2.0.

It's called HUNK_DREL32 (32-bit base-relative references), which can only occur in object files, so it is not a big problem.

But if you know the history of HUNK_ABSRELOC16 I would be interested (yes, off-topic). Who had the idea that 16-bit absolute addresses could ever make sense in an AmigaOS executable?

Sorry, but hunk $00003f7 doesnt works for kick 2.0. Works for kick 3.0+. Maybe can be supported by kick 2.0, but DOESNT works OK. When I was young and alive, i try to run something on my A2000 (kick 2.0). And it doesnt works, at begining i thinked that exe was corrupted or E bit is not set.Then i made investigation. It works on my A1200 (kick 3.0) and doesnt works on my A500 (kick 1.3) too. Later i try to dissasemble this file. And ReSource 5.0 can not load this file as exe, later i try ReSource 6.0 (from 1995 or 1996 year) and again i cant. Finally i used FileMaster 2.2 and View this file. And I seen where is/was problem, word style relocs, which i seen first time. Then i made pseudo converter from word relocs to normal/standard Amiga longword relocs. From my memory only Phxass and Barfly can created word style relocs on real Amiga. Yes, this reloc can be used on Amiga, but not as DEFAULT. Only as OPTIONAL. I never seen single file from Commodore which used this reloc.

[Don_Adan]

I dont know what is Photon target. Shortest exe or loading every or some Amiga files from memory.
If shortest exe, he can used other relocation method, f.e Atari ST (some Amiga games used this) or CodeWarrior.
If he want to load every Amiga file from memory then LoadSeg is the best solution.
But if he want to load only files created by own, then he can use own relocation routine. I used/created my relocation routine for some players, but I always created only one Hunk Amiga exe file. And it was easy to relocation. I can reloc module to chip or fast, dependent to user's choose.

[Thomas Richter]

Originally Posted by Don_Adan

Sorry, but hunk $00003f7 doesnt works for kick 2.0.

Ehem. I just tested, with the v37 ROM. Yes, it does. Note, however, that 0x3f7 is actually not the correct hunk number. It should be 0x3fc, and *that* does not work. This is by error, and v39 supports both, the incorrect, and the correct hunk number.

Originally Posted by Don_Adan

I never seen single file from Commodore which used this reloc.

That's because the Os build is based on SAS/C, h68x and SLink, and SLink does not support this hunk type. Actually, when building 3.1.4, I was mostly trying to avoid relocation hunks at all, and tried to create pure and relocatable binaries. That is of course not always possible, but the size of the reloc hunks should be rather small.