RKRM - AmigaDOS work in progress

[Thomas Richter]

Hi folks,


years ago when I did not know better, I was always wondering (and a bit upset) why there was not an RKRM volume about the dos.library and AmigaDOS. Well, reasons are of course that AmigaDOS is nothing but Tripos in a disguise, and thus the AmigaDOS manual was published through a different channel than the RKRMs. In fact, if you look into the Tripos manual and compare it to the Bantam volume, you might be suprised about the similarities.


Anyhow, the AmigaDOS manual leaves a lot to be deserved - it does not document a lot, and it is incomplete and misses quite some information. Then, we have documentation such as Ralph's Guru Book, which is pretty much out of print and hard to get.


Hence, it is really about time to improve the situation, and so I started drafting a dos.library documentation in RKRM style:



https://github.com/thorfdbg/rkrm-dos/


Note well that this is all work in progress, at this point quite incomplete, and it will certainly take several months (if not years) to arrive at a final stage. But, at least, a start has been made.


Of course, for a project like this, I need some help. Proof-reading, fixing defecs, improving formulation, spell-checking, and much more. My github page therefore not only contains the (fairly incomplete) PDF, but also the LaTeX source. If you are interested, you can submit change requests, defect reports etc. there, and even patches to the LaTeX source.


Thanks,
Thomas

[attle]

Wow! This looks great. Well done.

[kamelito]

Nice, but existing RKRM stopped at version 2.0 too.

[hooverphonique]

Nice.. back in the day, the stuff that I felt was missing, was all the packet/handler/filesystem stuff, not so much the high level file functions of dos.library.

This just a nudge towards where to focus

[Thomas Richter]

Originally Posted by hooverphonique

Nice.. back in the day, the stuff that I felt was missing, was all the packet/handler/filesystem stuff, not so much the high level file functions of dos.library.

No worries, this will all be covered, I'm just not there yet. It is a lot of stuff.

[kamelito]

Is all this stuff necessary if you backport the new DOS from OS4?

[paraj]

Gave it a quick skim, and so far looks very good and well-written. Really like the tips+code snippets.

Noticed two small things: Indentation for the parameters for Open() in 3.4 is a bit off, and in 3.9.2 you write that it's important to use MEMF_PUBLIC, but I think that should be MEMF_CLEAR.

[Thomas Richter]

Thanks, can you please open up a bug report on the github so I don't forget to look it up? With a work of that size, it is easily forgotten.

Thanks,
Thomas

[arcanist]

It looks really useful.

I'm not sure why but the PDF is very slow to render in Chrome and Okular/Poppler on Linux. I installed Texlive and rebuilt the PDF. It's 4x smaller and renders almost instantly.

[Olaf Barthel]

Originally Posted by kamelito

Is all this stuff necessary if you backport the new DOS from OS4?

Um, backporting AmigaDOS from AmigaOS4 is likely not an option, it being the result of almost two decades of continuous development. The implementation and also the architecture are very different from the current state of AmigaDOS 3.x in the current development version. I keep fixing bugs which Colin Wenzel had already found and fixed by 2006

[Olaf Barthel]

Originally Posted by Thomas Richter

No worries, this will all be covered, I'm just not there yet. It is a lot of stuff.

Ain't that the truth, and that's before you get into the sharp & pointy sections which will need to cover what doesn't work and how you can work around it, if at all. AmigaDOS file systems and handlers share the same faults as and limitations as exec devices

[Thorham]

Thanks, very nice

Originally Posted by arcanist

I'm not sure why but the PDF is very slow to render in Chrome and Okular/Poppler on Linux.

It's slow in Firefox under Windows 10, too, while quite fast in SumatraPDF.

[mr.spiv]

This fabulous! Thanks for initiating it. (secretly waiting for an overlay file description for absolute dummies)

[redblade]

Originally Posted by Thorham

Thanks, very nice

It's slow in Firefox under Windows 10, too, while quite fast in SumatraPDF.

Slow under Firefox on Lubuntu too. Maybe a single page html?

**qpdf is a bit quicker on Lubuntu

[hooverphonique]

Originally Posted by arcanist

It looks really useful.

I'm not sure why but the PDF is very slow to render in Chrome and Okular/Poppler on Linux. I installed Texlive and rebuilt the PDF. It's 4x smaller and renders almost instantly.

Yeah, what's up with that? Is the conversion from tex to pdf generating too many points, perhaps?

[Olaf Barthel]

Originally Posted by mr.spiv

This fabulous! Thanks for initiating it. (secretly waiting for an overlay file description for absolute dummies)

The overlay file format is not a complex one and, in spite of the shortcomings of the documentation in the AmigaDOS manual's "technical reference" section, it is decently well-documented already. But that's like saying that 90% of an ice berg can be underwater

What is not covered quite so well is the role of the overlay manager code, the part which the undocumented bits of LoadSeg() play and, arguably, how to specify the overlay node layout so that both the linker and you understand what is supposed to happen. In short, the complex bits are complex and the obscure bits are obscure, but they are taken care of before the executable is created and later loaded by a combination of dos.library magic in LoadSeg() and the overlay manager code which is linked against that executable.

The use of overlays fell out of favour in the late 1980'ies because the Amigas made in that time had much more memory available or could be expanded (RAM prices were also not quite so high any more as they used to be in 1984-1987).

The design of the overlay feature also left something to be desired, specifically flexibility. You had to preplan which parts of the program had to be in memory at a time. For some applications, e.g. the original Deluxe Paint (which had to work with as little as 256 KBytes of system RAM), this must have been not much of a challenge, but Deluxe Paint II struggled. Also, you could not override the overlay manager and say "my Amiga has enough memory to load the entire program, just do it".

The Aztec 'C' compiler (ca. 1987, I think) offered an alternative overlay manager which sort of emulated what was the standard on the Apple Macintosh. You could actually override it and load the entire program into memory. You could also load and unload overlay portions on demand under program control, rather than having the overlay manager code do it all for you.

This worked so well, it became the go-to solution for use of overlays in large programs, e.g. Deluxe Paint III.

Of course, there was a catch. The Aztec 'C' custom overlay manager worked its magic by hacking how the dos.library overlay loader did its part of the job by providing information about the size of the overlay table which disabled clearing that table. This is important because the Aztec 'C' custom overlay manager needs to be able to load/unload each overlay node upon demand and has to know where these nodes are stored in memory.

Sounds harmless, but it gets more interesting. The load/unload operation performed was cleverly optimized to reuse the overlay node information associated with every function call in the program. If you invoked a function stub found in an overlay node which had not been loaded yet, it would call the custom overlay manager, which would see to getting it loaded and then rewrite the function stub to point to the code that was just loaded. Yup, self-modifying code in action and no "cache clear" operation involved. You can guess how much fun this was bound to be when the 68040 came around. To complement the load operation, the unload operation then restored the function stub by looking at what it found in the overlay function table which had been cleverly prevented from getting cleared. With self-modifying code, of course.

By comparison, the AmigaDOS default overlay manager does not use self-modifying code. It's just that the linker makes sure that every single function call passes through the overlay manager, even if the overlay node associated with that function is already in memory. So you have this built-in function call overhead (not so great for Sculpt-3D) and a significantly larger executable than might have been needed. To add insult to injury, the 1985-1987 era developer tools were prone to wrap every single object code module which was part of an overlay executable into a single hunk. This is how you you got large overlay programs consisting of hundreds (!) of hunks which would slow down loading further.

For the whole saga, buy the book

[nogginthenog]

The RKRM looks really interesting thanks Thomas. I know you already have a few guides on Aminet.
Probably it's easy to render a HTML or AmigaGuide from the LaTeX if you know how.

It's amazing to me how they managed to incorporate Tripos into AmigaOS.

Take a look at the code for the date command:
https://github.com/borb/tripos/blob/.../com/bcpl/date

It's completely alien! Notice the use of rdargs() though.

[kamelito]

OT : Was CAOS only a design document? How far was it for completion?

[Wepl]

Originally Posted by mr.spiv

This fabulous! Thanks for initiating it. (secretly waiting for an overlay file description for absolute dummies)

there is a good info on aminet: http://aminet.net/package/docs/misc/Overlay

[Thomas Richter]

Certainly. This is what I wrote up, it will become (in one way or another, probably in shorter form) be part of the above document.

This being said, some work was done on it last week such that there is now already something about processes and how to create them. There will be not much progress this week as I'm on a business trip.