13 September 2023, 17:08 | #21 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Some news on this - I reached a new mile stone, so maybe people want to look into this again. The next chapter on the hunk structure is now (almost) finished. The highlights are (probably) the documentation of the overlay structure, and the documentation of the HUNK_SYMBOL and HUNK_DEBUG hunks, for the "Line-debug" option of SAS/C. Thanks goes to Steve Krueger for providing sufficient documentation on how this works.
There are (of course) already tools that make use of the line debug info, the latest SegTracker does this for you if you enable it. There is also documenation on how the "runtime binder" aka "BCPL startup" for handlers and the shell work, which is a "structure within the hunk structure", including some example code. Even though this is a legacy mechanism, it is unfortunately still required if you want to write your own Port-Handler (which is mounted with GlobVec=0) or your own shell (also using the legacy BCPL startup). I do not plan to document the GlobVec in entity, this stuff is right away obsolete, only the minimal amount of information necessary to get such programs with "bcpl linkage" working (for the above applications). Also, documentation for many additional dos.library calls is now present, though I'm still only half the way through. It is already >100 pages now, so I definitely need some help for cross-reading. Please forgive me for the bad formatting. This is the very last thing I will fix after correcting technical errors and spelling errors. (That is, I outright admit that it is still quite rough at this point). |
18 September 2023, 12:40 | #22 | ||
Natteravn
Join Date: Nov 2009
Location: Herford / Germany
Posts: 2,553
|
Quote:
https://gitlab.com/boemann/codecraft...escription.txt Quote:
HUNK_DATA: AmigaDOS (at least up to V40?) doesn't like data-bss sections, which are completely uninitialized (initialized size is 0). HUNK_BSS: Kickstart 1.x has a bug with BSS sections > 256K. HUNK_SYMBOL: LoadSeg doesn't allow symbol names with more than 31 longwords. HUNK_RELOC32: AmigaDOS rejects executables with more than 65536 relocations. HUNK_RELOC32SHORT: Does not support more than 65535 entries. HUNK_RELRELOC32: Due to a bug it uses 16-bit relocation offsets, like HUNK_RELOC32SHORT. In your chapter 9.1.7 the size is shown as 32. And, BTW, unlike stated in the chapter from above, HUNK_RELRELOC32is completely supported by vasm and vlink. |
||
18 September 2023, 13:25 | #23 | |
Registered User
Join Date: Aug 2010
Location: Germany
Posts: 537
|
Quote:
That compact version eventually became the foundation for the Kickstart 2.0 dos.library load file loader, with overlay support, etc. getting added and new features such as HUNK_RELOC32SHORT, too. Because of its compactness, different hunk types were now getting processed by shared code which made them subject to constraints (dbf-loops) which did not exist in the BCPL original. Funny enough, the Kickstart 1.x AmigaDOS load file loader does not bother to check for constraints such as "too many relocation offsets" because it "knew" that two billion would be more than enough. But the reimplementation easily stumbles over 65535 entries which could occur in actual compiled and linked code produced by gcc 6.2 (with workarounds in place to avoid bumping against these constraints). Last edited by Olaf Barthel; 18 September 2023 at 13:30. |
|
18 September 2023, 20:23 | #24 | ||||||
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Quote:
Quote:
Thanks, confirmed. That is a defect in clearvec(). Quote:
Quote:
Quote:
Quote:
I updated the chapter on git, would be great if you could check again. |
||||||
19 September 2023, 17:10 | #25 | ||||
Natteravn
Join Date: Nov 2009
Location: Herford / Germany
Posts: 2,553
|
If you can really get official information from Steve Kruger about the SAS/C SRC6 debug format, this would be extremely helpful for Camilla Boemann and the OS3.2 team, who already spent many months to reengineer it!
Quote:
I was refering to the situation when the given size in HUNK_CODEor HUNK_DATAis zero, although the HUNK_HEADERspecifies a size > 0. In this case it looks more like a BSSsection, without any initialized payload. The loader doesn't like that, though. You need at least 1 longword of initialized payload with these hunks. Quote:
HUNK_RELOC32SHORT. It should be clear to everybody that you cannot specify a bigger reloc-count in a 16-bit field. So you can also delete your last paragraph in 9.1.6 again. Quote:
Quote:
Otherwise, I noticed in table 28 you still have size 32. In 9.1.8 and 9.1.9 you write that symbol names may not be longer than 127 characters. Shouldn't that be 124 characters? Into a maximum size of 31 longwords I can put 124 characters (terminating zero is not required). Also in 9.5.4 ( HUNK_EXT) you mention the same limitation. I just noticed that vlink doesn't check this case. So if that defect can be confirmed (which is likely) I have to thank you as well! Some notes on table 41 ( HUNK_EXT): EXT_REF8and EXT_REF16were later (since some OS2.0 SDK?) redefined as EXT_RELREFn, to indicate that they are PC-relative references. In my experience old linkers may have used the type of the symbol definition to decide if it was an EXT_RELREF(symbol defined as EXT_DEF) or an EXT_ABSREF(symbol defined as EXT_ABS). Not many linkers support common symbol references. I have heard the original alink linker could do that (and vlink, of course). |
||||
20 September 2023, 19:05 | #26 | ||||
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Quote:
Quote:
It should really be 32, but due to a defect, it is only 16. Thus, all I can say is that this hunk cannot be safely used right now. Quote:
Quote:
Maybe a final question. In the object code format, the topmost two bits of HUNK_CODE, HUNK_DATA and HUNK_BSS are also used to indicate the memory type, which is then moved to HUNK_HEADER. In the HUNK_HEADER, the bit combination 11 (MEMF_CHIP|MEMF_FAST) indicates that the next longword is the explicit memory type. Is there a similar mechanism for object files? I see that nowhere mentioned, I just wonder how one could possibly create such a custom memory type in HUNK_HEADER if not by linking. |
||||
21 September 2023, 11:43 | #27 | |||||
Natteravn
Join Date: Nov 2009
Location: Herford / Germany
Posts: 2,553
|
Quote:
My notes tell me that this is a problem up to V40. Can anybody confirm it was fixed in later versions? Quote:
Quote:
Quote:
If you agree that this approach is ok, you may document it in your AmigaDOS RKRM, which makes it a little bit more official. I don't know any other assembler or linker supporting these extended memory attributes, but in vasm you can specify arbitrary memory flags in place of the chipor fastkeywords: section name,data,$12345678 Quote:
-hunkattr secname=valueto define the memory flags for a section by name. Example, force .data into Chip-RAM: -hunkattr .data=2. When memory flags get more complicated, vlink creates the additional word in the header. |
|||||
21 September 2023, 22:23 | #28 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Thanks again - in the meantime, I completed the binary file chapter by adding a section on link libraries. Unfortunately, the information from the Bantam book is quite inaccurate and I tried a bit to make it accurate. But maybe you can have a look into this section as well.
Unfortunately, Steve's email is no longer working, he must have been retired I guess. |
22 September 2023, 16:07 | #29 |
Natteravn
Join Date: Nov 2009
Location: Herford / Germany
Posts: 2,553
|
Ok, here are some more notes:
|
22 September 2023, 23:13 | #30 | |||||||||
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Thanks again for the very valuable feedback, I tried to address them in the update from today.
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
|
|||||||||
23 September 2023, 22:38 | #31 | |||||||
Natteravn
Join Date: Nov 2009
Location: Herford / Germany
Posts: 2,553
|
Quote:
Refer to 22.3.7 (HUNK_DEBUG) of The Amiga Guru Book. Last paragraph: Quote:
Quote:
Quote:
Quote:
Quote:
EXT_COMMON? I didn't know. This would mean that at least SLink supports common symbols, while I remember that BLink didn't... Quote:
I just wanted to mention that an uninitialized variable, like int foo;will generate a common symbol with most compilers (unlike Amiga compilers, which usually put those immediately into BSS; which may lead to linker errors when the symbol is defined more than once). Code:
frank@nerthus cat tst.c int foo; frank@nerthus gcc -S tst.c frank@nerthus cat tst.s .file "tst.c" .comm foo,4,4 .ident "GCC: (nb3 20180905) 5.5.0" |
|||||||
23 September 2023, 23:53 | #32 |
Defendit numerus
Join Date: Mar 2017
Location: Crossing the Rubicon
Age: 54
Posts: 4,501
|
I've a question for @Thomas.
EDIT: no response is necessary. I hadn't opened the file on GitHub yet and it's all clearly specified in chapter 10.3.5 and following. Really great document, thanks! [my concern was related to the 'hidden' functionality LoadSeg(NULL,,) and that it could be deprecated and eliminated in future kickstarts, but if a trace of it remains in a document like this it certainly can't disappear ] Last edited by ross; 24 September 2023 at 00:28. |
24 September 2023, 10:00 | #33 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
What remains in the Kickstart is not on me to decide, that's the Os team. What I would hope is that they address the defects in LoadSeg that have been identified so far (and they are unfortunately numerous), but do not remove functionality.
Overlays I consider rather practical, even though the standard overlay manager is a bit limited. (MuLink, for example, uses the overlay mechanism, but does not use the hierarchical overlay manager). I suggest the mechanism for the following use case: What I find a bit "irritating" is that some applications come with their own "private" dynamic libraries that somehow clutter your LIBS: directory - without an API documentation, these libraries remain an artifact of the corresponding application, and it makes such applications hard to install or remove as some files remain or need to be copied, without clear reference to the depending program. Such "on demand" functions should be better put into an overlay and loaded from there, such that everything could stay in one file a user can move around freely. DPaint V is an example for this - it has two libraries for blitting, one for the hardware blitter, one for CPU blitting. It would be better to have that as part of the core code and only load those functions that are required. That LoadSeg has hidden parameters is probably something that needs to be addressed in the #pragma files by giving it a separate name. The way how it currently looks makes it hard to access from C (or any other higher language). |
24 September 2023, 12:10 | #34 | |
Registered User
Join Date: Aug 2010
Location: Germany
Posts: 537
|
Quote:
Getting rid of the undocumented behaviour of LoadSeg() is not an easy option. Those programs which make use of overlays expect it to behave as it did back in 1985 and beyond. The overlay manager code is part of the program itself, and there are several variants of that code in use. Trying to render the dos.library API more robust is sometimes hard to achieve and one should keep trying. But sometimes the gods are against you and they're rowdily drunk, too. A more pressing issue would be to find a way to resolve the issues which tumble out of the Manx Aztec 'C' custom overlay manager, which resorts to self-modifying code. This is going to hurt, especially if you are using a JIT or a CPU which expects its data and code to have distinct characteristics. Now what? You can't just flush the data cache on every context Task switch as a preventive measure, for example. And for a JIT this might just be a disaster. |
|
24 September 2023, 12:27 | #35 | ||
Registered User
Join Date: Aug 2010
Location: Germany
Posts: 537
|
Quote:
Quote:
|
||
24 September 2023, 12:59 | #36 | |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Quote:
I wonder - are there actually any Amiga applications that make use of the Atzec overlay manager? There are certainly numerous Aztec compiled programs (and there is already a workaround for its faulty FPU-init code in the 68060.library), but is there one that uses its overlay manager and would otherwise run on the 68040 upwards? This being said, I'm still not clear where such a workaround (if necessary) should go. SetPatch? The CPU libraries? I had the same problem at 3.1.4 that I wasn't sure whether the workarounds from the CPU libraries should actually end up in SetPatch or not. |
|
24 September 2023, 17:34 | #37 | |||
Registered User
Join Date: Aug 2010
Location: Germany
Posts: 537
|
Quote:
Quote:
Quote:
A patch in LoadSeg() alone won't help because the jump table patching follows it. A patch in FreeMem() alone, with special code to identify hunks being freed like that, wouldn't help either because the jump table patching also happens after that. What might be detectable is the segload()/freeseg() code which is being used. The text in the Aztec 'C' runtime library source code says "Copyright (C) 1986,1987 by Manx Software Systems, Inc." so maybe it never changed through Aztec 'C' 3.4a-5.0b. Save for the use of _LVOAlert, _LVOFreeMem, _LVOOpenLibrary and _LVOSeek it's PC-relative code which could be easy to identify. When loading an overlay node, segload() opens dos.library first (never closes it; so much for professional work), calls Seek() and then LoadSeg(). freeseg() only ever calls FreeMem() but you can check the code which precedes the FreeMem() call, and what follows it. First call to segload(), you could detect it and rewrite the segload() implementation to include a cache clear operation after the Aztec 'C' linker-built jump table entry has been rewritten. The same approach might work just as well in the freeseg() code. You could detect whether you patched an Aztec 'C' overlay executable already: the first hunk contains the magic overlay cookie and the rest you could find out by checking whether LoadSeg() was called with the known code patterns preceding or following it. This might be doable. Hey, you might even be able to scan the root node code for the overlay cookie and then for the segload() and freeseg() code patterns and identify the culprit right at the first LoadSeg() performed by Workbench or the shell. As a test program, I can recommend Deluxe Paint II and III (not the copy-protected version, mind you). Last edited by Olaf Barthel; 25 September 2023 at 08:11. |
|||
02 November 2023, 18:18 | #38 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,331
|
Some update on this - I reached the next milestone. The first draft of the next bigger chapter on handlers, filesystems and a list of DosPackets is available - on its project page:
https://github.com/thorfdbg/rkrm-dos So, this provides now a section on "how does a handler work", an a list of dos packets AmigaDOS components use and issue. This is fairly more complete than what is found in dos/dosextens.h, and even include some exotic ones like "ACTION_QUEUE" the console supports at least since v40 (Kick 3.1) and likely even before. The official documentation from Bantam does not list them - and, upon reviewing this work - has numerous defects for the packets that came into the system as of 2.0. It looks like this chapter was written up in a rush. Ok, my writeup is at this point also very rough, but I hope I'll be able to improve it, and I also hope it has less factual errors (at least in its final stage). As before, I would be glad if people would sketch through it and report issues. In particular, I do not know how readable the new "handler" chapter actually is. Next, Olaf requested a chapter on the Aztec C overlay manager (just see above), so I'll work on it as little Christmas gift (though, if I may, Halloween might have been more appropriate for this announcement as it is a bit scary what MANX does here - self-modifying code. Sorry, did not make it for this deadline.) |
02 November 2023, 21:36 | #39 |
Registered User
Join Date: Nov 2010
Location: .
Posts: 386
|
Hey Thomas,
I have prepared patch with some spellcheck fixes. I cannot open a pull request because apparently the git repository is configured to only allow that to collaborators. How can I submit this patch? I mean, if you prefer you can also apply it yourself, it's attached here. I don't know your git workflow but you can apply this patch by first unzipping it and then running git am 0001-Round-of-typo-fixes.patch Thanks for your amazing project. Last edited by jman; 02 November 2023 at 21:43. |
03 November 2023, 10:44 | #40 | |
Registered User
Join Date: Aug 2006
Location: Finland
Age: 52
Posts: 244
|
Quote:
Having done just a quick overall look I would suggest adding a table somewhere in the intro that would have mapping between kickstarts (1.x, 2.x etc) to these v3x & v4x versions. Just for lazy people like me to have all essentials in one document |
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Uni-Launcher (Work in Progress) Ideas? | DisasterIncarna | support.Apps | 10 | 10 August 2022 11:36 |
PdfTools Suite - Work in Progress a New Programs | savior | News | 18 | 26 April 2019 19:31 |
Work in progress. | Cowcat | Coders. General | 7 | 18 February 2014 22:33 |
Work In Progress - Speedball 2 | Bitmap Brother | Nostalgia & memories | 3 | 21 September 2005 00:50 |
Trick Or Treat Remake - Work in Progress | ant512 | Amiga scene | 15 | 16 November 2004 17:25 |
|
|