RKRM - AmigaDOS work in progress

[Thomas Richter]

Some news on this - I reached a new mile stone, so maybe people want to look into this again. The next chapter on the hunk structure is now (almost) finished. The highlights are (probably) the documentation of the overlay structure, and the documentation of the HUNK_SYMBOL and HUNK_DEBUG hunks, for the "Line-debug" option of SAS/C. Thanks goes to Steve Krueger for providing sufficient documentation on how this works.

There are (of course) already tools that make use of the line debug info, the latest SegTracker does this for you if you enable it.

There is also documenation on how the "runtime binder" aka "BCPL startup" for handlers and the shell work, which is a "structure within the hunk structure", including some example code. Even though this is a legacy mechanism, it is unfortunately still required if you want to write your own Port-Handler (which is mounted with GlobVec=0) or your own shell (also using the legacy BCPL startup). I do not plan to document the GlobVec in entity, this stuff is right away obsolete, only the minimal amount of information necessary to get such programs with "bcpl linkage" working (for the above applications).

Also, documentation for many additional dos.library calls is now present, though I'm still only half the way through. It is already >100 pages now, so I definitely need some help for cross-reading.

Please forgive me for the bad formatting. This is the very last thing I will fix after correcting technical errors and spelling errors. (That is, I outright admit that it is still quite rough at this point).

[phx]

Originally Posted by Thomas Richter

documentation of the HUNK_SYMBOL and HUNK_DEBUG hunks, for the "Line-debug" option of SAS/C. Thanks goes to Steve Krueger for providing sufficient documentation on how this works.

In case you are interested in more SAS/C specific Debug hunk interna, then there is this:
https://gitlab.com/boemann/codecraft...escription.txt

It is already >100 pages now, so I definitely need some help for cross-reading.

Don't know if you want to mention bugs in AmigaDOS, but here are some notes on the hunk chapter, which I remembered:

HUNK_DATA

: AmigaDOS (at least up to V40?) doesn't like data-bss sections, which are completely uninitialized (initialized size is 0).

HUNK_BSS

: Kickstart 1.x has a bug with BSS sections > 256K.

HUNK_SYMBOL

: LoadSeg doesn't allow symbol names with more than 31 longwords.

HUNK_RELOC32

: AmigaDOS rejects executables with more than 65536 relocations.

HUNK_RELOC32SHORT

: Does not support more than 65535 entries.

HUNK_RELRELOC32

: Due to a bug it uses 16-bit relocation offsets, like

HUNK_RELOC32SHORT

. In your chapter 9.1.7 the size is shown as 32.
And, BTW, unlike stated in the chapter from above,

HUNK_RELRELOC32

is completely supported by vasm and vlink.

[Olaf Barthel]

Originally Posted by phx

In case you are interested in more SAS/C specific Debug hunk interna, then there is this:
https://gitlab.com/boemann/codecraft...escription.txt

Don't know if you want to mention bugs in AmigaDOS, but here are some notes on the hunk chapter, which I remembered:

HUNK_DATA

: AmigaDOS (at least up to V40?) doesn't like data-bss sections, which are completely uninitialized (initialized size is 0).

HUNK_BSS

: Kickstart 1.x has a bug with BSS sections > 256K.

HUNK_SYMBOL

: LoadSeg doesn't allow symbol names with more than 31 longwords.

HUNK_RELOC32

: AmigaDOS rejects executables with more than 65536 relocations.

HUNK_RELOC32SHORT

: Does not support more than 65535 entries.

HUNK_RELRELOC32

: Due to a bug it uses 16-bit relocation offsets, like

HUNK_RELOC32SHORT

. In your chapter 9.1.7 the size is shown as 32.
And, BTW, unlike stated in the chapter from above,

HUNK_RELRELOC32

is completely supported by vasm and vlink.

I for one will never again suspect the Kickstart 1.x AmigaDOS of being not fully-baked, sort of. The limitations/shortcomings of features which used to work with more than 65535 entries in the Kickstart 1.x AmigaDOS are a by-product of reimplementing the load file loader in assembly language, with a focus on keeping the code size very small because it had to go into the A590 ROM.

That compact version eventually became the foundation for the Kickstart 2.0 dos.library load file loader, with overlay support, etc. getting added and new features such as HUNK_RELOC32SHORT, too.

Because of its compactness, different hunk types were now getting processed by shared code which made them subject to constraints (dbf-loops) which did not exist in the BCPL original.

Funny enough, the Kickstart 1.x AmigaDOS load file loader does not bother to check for constraints such as "too many relocation offsets" because it "knew" that two billion would be more than enough. But the reimplementation easily stumbles over 65535 entries which could occur in actual compiled and linked code produced by gcc 6.2 (with workarounds in place to avoid bumping against these constraints).

[Thomas Richter]

Originally Posted by phx

In case you are interested in more SAS/C specific Debug hunk interna, then there is this:
https://gitlab.com/boemann/codecraft...escription.txt

Thanks, though I probably check with Steve first.

Originally Posted by phx

Don't know if you want to mention bugs in AmigaDOS, but here are some notes on the hunk chapter, which I remembered:

HUNK_DATA

: AmigaDOS (at least up to V40?) doesn't like data-bss sections, which are completely uninitialized (initialized size is 0).

Thanks, but I do not quite understand. Do you mean that v34 and below does not clear the (implicit) BSS sections at the end of those hunks if the load size is smaller than the allocation size?

Originally Posted by phx

HUNK_BSS

: Kickstart 1.x has a bug with BSS sections > 256K.

Thanks, confirmed. That is a defect in clearvec().

Originally Posted by phx

HUNK_SYMBOL

: LoadSeg doesn't allow symbol names with more than 31 longwords.

Thanks, that is an off-by-one error on my end.

Originally Posted by phx

HUNK_RELOC32

: AmigaDOS rejects executables with more than 65536 relocations.

There, and for all other RELOC types, thanks.

Originally Posted by phx

HUNK_RELRELOC32

: Due to a bug it uses 16-bit relocation offsets, like

HUNK_RELOC32SHORT

. In your chapter 9.1.7 the size is shown as 32.

Yes, it should really be 32-bits it should read, but does not. This really limits the usefulness of this hunk.

Originally Posted by phx

And, BTW, unlike stated in the chapter from above,

HUNK_RELRELOC32

is completely supported by vasm and vlink.

Aha, thank you. Never thought any program would actually make use of this thing. It is rather exotic.


I updated the chapter on git, would be great if you could check again.

[phx]

Originally Posted by Thomas Richter

Thanks, though I probably check with Steve first.

If you can really get official information from Steve Kruger about the SAS/C SRC6 debug format, this would be extremely helpful for Camilla Boemann and the OS3.2 team, who already spent many months to reengineer it!

Do you mean that v34 and below does not clear the (implicit) BSS sections at the end of those hunks if the load size is smaller than the allocation size?

No, that should be well known and you already documented it.

I was refering to the situation when the given size in

HUNK_CODE

or

HUNK_DATA

is zero, although the

HUNK_HEADER

specifies a size > 0. In this case it looks more like a

BSS

section, without any initialized payload. The loader doesn't like that, though. You need at least 1 longword of initialized payload with these hunks.

There, and for all other RELOC types, thanks.

Although, forget my note about the maximum of 65535 for the short-reloc hunks, like

HUNK_RELOC32SHORT

. It should be clear to everybody that you cannot specify a bigger reloc-count in a 16-bit field. So you can also delete your last paragraph in 9.1.6 again.

Yes, it should really be 32-bits it should read, but does not. This really limits the usefulness of this hunk.

Indeed, that's quite unfortunate.

I updated the chapter on git, would be great if you could check again.

There are certainly a few spelling/syntax errors here and there (like "do not understand the hunk-type 0x3fc not properly" in 9.1.6, or "Luckely" in 9.1.7), but I wouldn't fix them until the final version is ready. And then you should better find a native speaker for proof reading.

Otherwise, I noticed in table 28 you still have size 32.

In 9.1.8 and 9.1.9 you write that symbol names may not be longer than 127 characters. Shouldn't that be 124 characters? Into a maximum size of 31 longwords I can put 124 characters (terminating zero is not required).
Also in 9.5.4 (

HUNK_EXT

) you mention the same limitation. I just noticed that vlink doesn't check this case. So if that defect can be confirmed (which is likely) I have to thank you as well!

Some notes on table 41 (

HUNK_EXT

):

EXT_REF8

and

EXT_REF16

were later (since some OS2.0 SDK?) redefined as

EXT_RELREFn

, to indicate that they are PC-relative references. In my experience old linkers may have used the type of the symbol definition to decide if it was an

EXT_RELREF

(symbol defined as

EXT_DEF

) or an

EXT_ABSREF

(symbol defined as

EXT_ABS

).
Not many linkers support common symbol references. I have heard the original alink linker could do that (and vlink, of course).

[Thomas Richter]

Originally Posted by phx

I was refering to the situation when the given size in

HUNK_CODE

or

HUNK_DATA

is zero, although the

HUNK_HEADER

specifies a size > 0. In this case it looks more like a

BSS

section, without any initialized payload. The loader doesn't like that, though. You need at least 1 longword of initialized payload with these hunks.

Actually, what I see is that the initialization is not done, but the code should correctly continue loading.

Originally Posted by phx

It should be clear to everybody that you cannot specify a bigger reloc-count in a 16-bit field. So you can also delete your last paragraph in 9.1.6 again.

Yes, thanks.

Originally Posted by phx

Otherwise, I noticed in table 28 you still have size 32.

It should really be 32, but due to a defect, it is only 16. Thus, all I can say is that this hunk cannot be safely used right now.

Originally Posted by phx

In 9.1.8 and 9.1.9 you write that symbol names may not be longer than 127 characters. Shouldn't that be 124 characters?

Thanks.

Originally Posted by phx

Also in 9.5.4 (

HUNK_EXT

) you mention the same limitation. I just noticed that vlink doesn't check this case. So if that defect can be confirmed (which is likely) I have to thank you as well!

No, my fault. This hunk cannot actually appear in a binary load file (and is therefore in the object file section, and not in the binary file section).


Maybe a final question. In the object code format, the topmost two bits of HUNK_CODE, HUNK_DATA and HUNK_BSS are also used to indicate the memory type, which is then moved to HUNK_HEADER. In the HUNK_HEADER, the bit combination 11 (MEMF_CHIP|MEMF_FAST) indicates that the next longword is the explicit memory type. Is there a similar mechanism for object files? I see that nowhere mentioned, I just wonder how one could possibly create such a custom memory type in HUNK_HEADER if not by linking.

[phx]

Originally Posted by Thomas Richter

Actually, what I see is that the initialization is not done, but the code should correctly continue loading.

Oh, indeed! Now I analyzed it again and the seglist seems ok. The loader just refused clearing the segment in this case.
My notes tell me that this is a problem up to V40. Can anybody confirm it was fixed in later versions?

It should really be 32, but due to a defect, it is only 16. Thus, all I can say is that this hunk cannot be safely used right now.

This severe limitation makes it only useful for small programs, but I wouldn't call it unsafe. The linker should show an error when offsets > 65535 are required, just like with small data.

No, my fault. This hunk cannot actually appear in a binary load file

Right. I didn't think about that either.

In the HUNK_HEADER, the bit combination 11 (MEMF_CHIP|MEMF_FAST) indicates that the next longword is the explicit memory type. Is there a similar mechanism for object files?

I asked myself the same question, some years ago. I didn't find an official definition anywhere. So I decided, unofficially, that the combined attributes/size word in a section hunk is followed by a word with the detailed memory flags, just like in the header.

If you agree that this approach is ok, you may document it in your AmigaDOS RKRM, which makes it a little bit more official.

I don't know any other assembler or linker supporting these extended memory attributes, but in vasm you can specify arbitrary memory flags in place of the

chip

or

fast

keywords:

section name,data,$12345678

I just wonder how one could possibly create such a custom memory type in HUNK_HEADER if not by linking.

Maybe a linker supports overwriting a section's memory flags. In vlink I have the option

-hunkattr secname=value

to define the memory flags for a section by name. Example, force .data into Chip-RAM:

-hunkattr .data=2

. When memory flags get more complicated, vlink creates the additional word in the header.

[Thomas Richter]

Thanks again - in the meantime, I completed the binary file chapter by adding a section on link libraries. Unfortunately, the information from the Bantam book is quite inaccurate and I tried a bit to make it accurate. But maybe you can have a look into this section as well.

Unfortunately, Steve's email is no longer working, he must have been retired I guess.

[phx]

Ok, here are some more notes:

• 9.1.1, table 22, row 10:

  ms[i]

  should be

  mt[i]

  (memory type is explicitely provided).
• 9.1.7 Not only the reloc offsets are 16 bits, due to a defect, but also the number of relocations and the relative hunk id. Exactly like

  HUNK_RELOC32SHORT

  .
• 9.1.10, table 32, row 4: This field shall not always be zero. It is used as a base-offset, added to all section offsets given in this hunk. Reason: When a linker merges debug sections from multiple object files, it only has to adapt the base offset and doesn't have to understand or process its contents in detail.
• Also 9.1.10, table 32: I would say that "Compiler- and configuration specific data" starts two rows later, i.e. it doesn't include the above mentioned base-offset and 4-character ID for the debug hunk type. Isn't that the same for all types of debug hunks?
• Table 32 LINE debug format: I learned from Olaf and Camilla that these dummy bytes may have a special meaning for the SRC6 debug format. But it's probably ok to define them as dummy here.
• 9.5.2: Theoretically the linker could check if there is a branch instruction as the potential source of an ALV. But I don't know if this is really done by all linkers.
• 9.5.4-9.5.6 (small data): You write that the addend at the relocation offset is adjusted by the start address of the combined

  __MERGED

  sections. This is not always true. In many cases you have an offset of 32766 into this section to be able to use the maximum of 64K. Some linkers select this offset automatically depending on total section size (blink, slink?), some have it fixed at -32766 or modifiable by an option (vlink). AmigaDOS linkers should generally export a linker symbol which points to this base address, called

  _LinkerDB

  , which can be used to initialize the base register.
• Small data: Here the n-bit addend at the relocation offset is unsigned (section offset), although usually addends are signed.
• Besides

  __MERGED

  , there are more special section names a linker should know. For example

  _NOMERGE

  prevents the merge of two sections, even when their type and name matches.
• 9.5.7, table 45: Your definition for common symbols says that an

  EXT_DEF

  or

  EXT_ABS

  with the same name is allowed, and will automatically resolve all common references (as if it would be weak symbol). I wasn't so sure about that and I generate an error in this case. Although I know that ELF linkers work as you say.
• 9.5.7 (Common Symbols): "This mechanism is mostly used by FORTRAN"? Most C compilers I have seen use common symbols, whenever you define an uninitialized variable. So you can for example have

  int bla;

  in multiple object files, which will merge into one common symbol.

  extern int bla;

  OTOH, would make sure it is externally defined (no common symbol). You are right that Amiga compilers (maybe besides gcc with a.out or ELF objects) usually did not create common symbols.

[Thomas Richter]

Thanks again for the very valuable feedback, I tried to address them in the update from today.

Originally Posted by phx

• 9.1.1, table 22, row 10:

  ms[i]

  should be

  mt[i]

  (memory type is explicitely provided).
• 9.1.7 Not only the reloc offsets are 16 bits, due to a defect, but also the number of relocations and the relative hunk id. Exactly like

  HUNK_RELOC32SHORT

  .

Thanks, I fixed that.

Originally Posted by phx

• 9.1.10, table 32, row 4: This field shall not always be zero. It is used as a base-offset, added to all section offsets given in this hunk. Reason: When a linker merges debug sections from multiple object files, it only has to adapt the base offset and doesn't have to understand or process its contents in detail.

Hmm. Interesting. The code snipped I received from Steve always writes this as zero. It can probably become non-zero when pre-linking.

Originally Posted by phx

• Also 9.1.10, table 32: I would say that "Compiler- and configuration specific data" starts two rows later, i.e. it doesn't include the above mentioned base-offset and 4-character ID for the debug hunk type. Isn't that the same for all types of debug hunks?

I do not know. If I look this up in the Bantam book, it only defines the hunk length as always present. It is probably a good idea to design all debug hunks in the same way, so I keep your comment in as a suggestion.

Originally Posted by phx

• Table 32 LINE debug format: I learned from Olaf and Camilla that these dummy bytes may have a special meaning for the SRC6 debug format. But it's probably ok to define them as dummy here.

I cannot really tell. Unfortunately, I do not seem to be able to reach Steve anymore.

Originally Posted by phx

• 9.5.2: Theoretically the linker could check if there is a branch instruction as the potential source of an ALV. But I don't know if this is really done by all linkers.

At least not by Slink, I have seen it creating junk ALVs in case. Actually, if you see what SAS/C does when generating symbols is just to avoid such situations: If you put a static const object into a function, the compiler places it in the text segment. If, however, you define a static const object outside of a function, it ends up in __MERGED as the compiler cannot ensure that it will always be in reach.

Originally Posted by phx

• 9.5.4-9.5.6 (small data): You write that the addend at the relocation offset is adjusted by the start address of the combined

  __MERGED

  sections. This is not always true. In many cases you have an offset of 32766 into this section to be able to use the maximum of 64K. Some linkers select this offset automatically depending on total section size (blink, slink?), some have it fixed at -32766 or modifiable by an option (vlink). AmigaDOS linkers should generally export a linker symbol which points to this base address, called

  _LinkerDB

  , which can be used to initialize the base register.

Thanks, corrected this. It is partially mentioned, but I should really include the additional offset in the relocation procedure to be performed.

Originally Posted by phx

• Small data: Here the n-bit addend at the relocation offset is unsigned (section offset), although usually addends are signed.
• Besides

  __MERGED

  , there are more special section names a linker should know. For example

  _NOMERGE

  prevents the merge of two sections, even when their type and name matches.

I'm not sure whether this is rather linker specific and a convention. The Bantam book does not say anything here (but it generally does not say much).

Originally Posted by phx

• 9.5.7, table 45: Your definition for common symbols says that an

  EXT_DEF

  or

  EXT_ABS

  with the same name is allowed, and will automatically resolve all common references (as if it would be weak symbol). I wasn't so sure about that and I generate an error in this case. Although I know that ELF linkers work as you say.

That is at least what the Bantam book has to say. This also follows from what FORTRAN "COMMON" works like, it likely comes from there.

Originally Posted by phx

• 9.5.7 (Common Symbols): "This mechanism is mostly used by FORTRAN"? Most C compilers I have seen use common symbols, whenever you define an uninitialized variable. So you can for example have

  int bla;

  in multiple object files, which will merge into one common symbol.

  extern int bla;

  OTOH, would make sure it is externally defined (no common symbol). You are right that Amiga compilers (maybe besides gcc with a.out or ELF objects) usually did not create common symbols.

As far as I remember, SAS/C can be configured to generate common symbols, but this is not how the C language works. It requires the "one definition rule". Thus, if you have an "extern int foo;" in all your translation units, the linker should warn you that "foo" is undefined (unlike FOTRAN COMMON). It requires at least one translation unit where you have an "int foo;", i.e. a definition.

[phx]

Originally Posted by Thomas Richter

Hmm. Interesting. The code snipped I received from Steve always writes this as zero.

Which is correct for object files. But not necessarily for executables.
Refer to 22.3.7 (HUNK_DEBUG) of The Amiga Guru Book. Last paragraph:

A convention established by BLink is that the first longword in the data area of a debugging hunk contains the byte offset (into a coalesced hunk) of the original hunk the debugging block was associated with in the originating object module

Debuggers are aware of this base offset.

At least not by Slink, I have seen it creating junk ALVs in case.

Then I hope that ALVs are not enabled by default. AFAIK this was the case for BLink. A development tool should never enable dangerous options by default.

Actually, if you see what SAS/C does when generating symbols is just to avoid such situations: If you put a static const object into a function, the compiler places it in the text segment.

Hmm. But a static const object is always local to the translation unit, so the assembler should easily find the optimal addressing mode and the linker is not involved.

I'm not sure whether this is rather linker specific and a convention. The Bantam book does not say anything here (but it generally does not say much).

It is probably SAS/C or BLink specific. But these tools, together with Devpac, have set some kind of standard, which I tend to follow whenever the official documentation is lacking.

As far as I remember, SAS/C can be configured to generate common symbols,

Using

EXT_COMMON

? I didn't know. This would mean that at least SLink supports common symbols, while I remember that BLink didn't...

but this is not how the C language works. It requires the "one definition rule". Thus, if you have an "extern int foo;" in all your translation units, the linker should warn you that "foo" is undefined (unlike FOTRAN COMMON). It requires at least one translation unit where you have an "int foo;", i.e. a definition.

Absolutely! I wrote nothing else.
I just wanted to mention that an uninitialized variable, like

int foo;

will generate a common symbol with most compilers (unlike Amiga compilers, which usually put those immediately into BSS; which may lead to linker errors when the symbol is defined more than once).

frank@nerthus cat tst.c
int foo;
frank@nerthus gcc -S tst.c
frank@nerthus cat tst.s
        .file   "tst.c"
        .comm   foo,4,4
        .ident  "GCC: (nb3 20180905) 5.5.0"

[ross]

I've a question for @Thomas.

EDIT: no response is necessary.
I hadn't opened the file on GitHub yet and it's all clearly specified in chapter 10.3.5 and following.

Really great document, thanks!

[my concern was related to the 'hidden' functionality LoadSeg(NULL,,) and that it could be deprecated and eliminated in future kickstarts, but if a trace of it remains in a document like this it certainly can't disappear ]

[Thomas Richter]

What remains in the Kickstart is not on me to decide, that's the Os team. What I would hope is that they address the defects in LoadSeg that have been identified so far (and they are unfortunately numerous), but do not remove functionality.

Overlays I consider rather practical, even though the standard overlay manager is a bit limited. (MuLink, for example, uses the overlay mechanism, but does not use the hierarchical overlay manager).

I suggest the mechanism for the following use case: What I find a bit "irritating" is that some applications come with their own "private" dynamic libraries that somehow clutter your LIBS: directory - without an API documentation, these libraries remain an artifact of the corresponding application, and it makes such applications hard to install or remove as some files remain or need to be copied, without clear reference to the depending program. Such "on demand" functions should be better put into an overlay and loaded from there, such that everything could stay in one file a user can move around freely.
DPaint V is an example for this - it has two libraries for blitting, one for the hardware blitter, one for CPU blitting. It would be better to have that as part of the core code and only load those functions that are required.

That LoadSeg has hidden parameters is probably something that needs to be addressed in the #pragma files by giving it a separate name. The way how it currently looks makes it hard to access from C (or any other higher language).

[Olaf Barthel]

Originally Posted by ross

I've a question for @Thomas.

EDIT: no response is necessary.
I hadn't opened the file on GitHub yet and it's all clearly specified in chapter 10.3.5 and following.

Really great document, thanks!

[my concern was related to the 'hidden' functionality LoadSeg(NULL,,) and that it could be deprecated and eliminated in future kickstarts, but if a trace of it remains in a document like this it certainly can't disappear ]

This is one of the hard-to-shake legacy features of dos.library. Some of the decisions which went into making what became the dos.library API in 1985 are quite strange. For example, there is another private LoadSeg() function which works just like the documented "regular" version does, except that it will not display an AmigaDOS requester window if the load operation should fail, and it does not support loading overlays either.

Getting rid of the undocumented behaviour of LoadSeg() is not an easy option. Those programs which make use of overlays expect it to behave as it did back in 1985 and beyond. The overlay manager code is part of the program itself, and there are several variants of that code in use. Trying to render the dos.library API more robust is sometimes hard to achieve and one should keep trying. But sometimes the gods are against you and they're rowdily drunk, too.

A more pressing issue would be to find a way to resolve the issues which tumble out of the Manx Aztec 'C' custom overlay manager, which resorts to self-modifying code. This is going to hurt, especially if you are using a JIT or a CPU which expects its data and code to have distinct characteristics. Now what? You can't just flush the data cache on every context Task switch as a preventive measure, for example. And for a JIT this might just be a disaster.

[Olaf Barthel]

Originally Posted by Thomas Richter

What remains in the Kickstart is not on me to decide, that's the Os team. What I would hope is that they address the defects in LoadSeg that have been identified so far (and they are unfortunately numerous), but do not remove functionality.

Some of the functionality can be replaced at runtime, through SetPatch, for example. Just not the historic BCPL mischief because it takes a different path through the library. But there is plenty of room for improvement, not just in the dark corners of the design & implementation. For example, LoadSeg() now always implies a wallop on the data and instruction caches, to make sure that data and code are consistent with the execution state. It does not have to be this way. For example, it would be possible, within reason, to selectively flush the contents of the hunks which LoadSeg() just loaded and processed to memory. Just not for executables which contain hundreds of small hunks

Overlays I consider rather practical, even though the standard overlay manager is a bit limited. (MuLink, for example, uses the overlay mechanism, but does not use the hierarchical overlay manager).

The overlay feature's power and limitations are also a product of the overlay manager and the specific linker involved. You can bypass this to some degree, as the Aztec 'C' custom overlay manager and the matching "ln" linker, but the LoadSeg() function expects the overlay manager to keep the hunk table consistent (which the Aztec 'C' overlay manager "hacks"). You don't have much room to work around these constraints.

[Thomas Richter]

Originally Posted by Olaf Barthel

A more pressing issue would be to find a way to resolve the issues which tumble out of the Manx Aztec 'C' custom overlay manager, which resorts to self-modifying code. This is going to hurt, especially if you are using a JIT or a CPU which expects its data and code to have distinct characteristics. Now what? You can't just flush the data cache on every context Task switch as a preventive measure, for example. And for a JIT this might just be a disaster.

I wonder - are there actually any Amiga applications that make use of the Atzec overlay manager? There are certainly numerous Aztec compiled programs (and there is already a workaround for its faulty FPU-init code in the 68060.library), but is there one that uses its overlay manager and would otherwise run on the 68040 upwards?


This being said, I'm still not clear where such a workaround (if necessary) should go. SetPatch? The CPU libraries? I had the same problem at 3.1.4 that I wasn't sure whether the workarounds from the CPU libraries should actually end up in SetPatch or not.

[Olaf Barthel]

Originally Posted by Thomas Richter

I wonder - are there actually any Amiga applications that make use of the Atzec overlay manager?

There's Deluxe Paint II and beyond, Deluxe Video, etc. Almost every large application published in 1987 and the following years was making use of the Aztec 'C' overlay manager. The use of overlays peaked by 1988/1989 when you could solve the problem of large applications not fitting into memory by having more RAM installed.

There are certainly numerous Aztec compiled programs (and there is already a workaround for its faulty FPU-init code in the 68060.library), but is there one that uses its overlay manager and would otherwise run on the 68040 upwards?

Hard to say how much of the 1988/1989 era software is still in use today, or was in use in the early 1990'ies on the 68040/68060. The Aztec 'C' custom overlay manager only ever called LoadSeg() first and would then patch the jump table afterwards (so no help from a CacheClearE() in dos.library). It did not call UnLoadSeg() either, but used FreeMem() instead, then patch the jump table (no CacheClearE() there either).

This being said, I'm still not clear where such a workaround (if necessary) should go. SetPatch? The CPU libraries? I had the same problem at 3.1.4 that I wasn't sure whether the workarounds from the CPU libraries should actually end up in SetPatch or not.

Hm...

A patch in LoadSeg() alone won't help because the jump table patching follows it. A patch in FreeMem() alone, with special code to identify hunks being freed like that, wouldn't help either because the jump table patching also happens after that.

What might be detectable is the segload()/freeseg() code which is being used. The text in the Aztec 'C' runtime library source code says "Copyright (C) 1986,1987 by Manx Software Systems, Inc." so maybe it never changed through Aztec 'C' 3.4a-5.0b.

Save for the use of _LVOAlert, _LVOFreeMem, _LVOOpenLibrary and _LVOSeek it's PC-relative code which could be easy to identify. When loading an overlay node, segload() opens dos.library first (never closes it; so much for professional work), calls Seek() and then LoadSeg(). freeseg() only ever calls FreeMem() but you can check the code which precedes the FreeMem() call, and what follows it.

First call to segload(), you could detect it and rewrite the segload() implementation to include a cache clear operation after the Aztec 'C' linker-built jump table entry has been rewritten. The same approach might work just as well in the freeseg() code. You could detect whether you patched an Aztec 'C' overlay executable already: the first hunk contains the magic overlay cookie and the rest you could find out by checking whether LoadSeg() was called with the known code patterns preceding or following it.

This might be doable. Hey, you might even be able to scan the root node code for the overlay cookie and then for the segload() and freeseg() code patterns and identify the culprit right at the first LoadSeg() performed by Workbench or the shell.

As a test program, I can recommend Deluxe Paint II and III (not the copy-protected version, mind you).

[Thomas Richter]

Some update on this - I reached the next milestone. The first draft of the next bigger chapter on handlers, filesystems and a list of DosPackets is available - on its project page:

https://github.com/thorfdbg/rkrm-dos

So, this provides now a section on "how does a handler work", an a list of dos packets AmigaDOS components use and issue. This is fairly more complete than what is found in dos/dosextens.h, and even include some exotic ones like "ACTION_QUEUE" the console supports at least since v40 (Kick 3.1) and likely even before. The official documentation from Bantam does not list them - and, upon reviewing this work - has numerous defects for the packets that came into the system as of 2.0. It looks like this chapter was written up in a rush. Ok, my writeup is at this point also very rough, but I hope I'll be able to improve it, and I also hope it has less factual errors (at least in its final stage).

As before, I would be glad if people would sketch through it and report issues. In particular, I do not know how readable the new "handler" chapter actually is.

Next, Olaf requested a chapter on the Aztec C overlay manager (just see above), so I'll work on it as little Christmas gift (though, if I may, Halloween might have been more appropriate for this announcement as it is a bit scary what MANX does here - self-modifying code. Sorry, did not make it for this deadline.)

[jman]

Hey Thomas,

I have prepared patch with some spellcheck fixes.

I cannot open a pull request because apparently the git repository is configured to only allow that to collaborators.

How can I submit this patch? I mean, if you prefer you can also apply it yourself, it's attached here.

I don't know your git workflow but you can apply this patch by first unzipping it and then running

git am 0001-Round-of-typo-fixes.patch

Thanks for your amazing project.

[mr.spiv]

Originally Posted by Thomas Richter

Some update on this - I reached the next milestone. The first draft of the next bigger chapter on handlers, filesystems and a list of DosPackets is available - on its project page:

https://github.com/thorfdbg/rkrm-dos

First, thanks for doing this. Excellent work and this will find a place as a printed booklet next to the Guru Book.
Having done just a quick overall look I would suggest adding a table somewhere in the intro that would have mapping between kickstarts (1.x, 2.x etc) to these v3x & v4x versions. Just for lazy people like me to have all essentials in one document