AmigaOS 3.2 released!

[shades_aus]

Originally Posted by OlafSch

@shades_aus

as Olaf said AROS inherited the same advantages but also shortcomings of the old AmigaOS. You cannot overcome this without rewriting everything. And I would add... you still would have the same problems any niche OS has... not enough developer, driver and modern software. The idea to port important parts of Aros to a mainstream OS like Linux is much more promising (and that is what Deadwood is already doing).

Future will be in my view:
Aros 68k (including ApolloOS)
Aros X86 (VM and real Hardware X86)
Aros on Linux base (AMD64)

my personal favorites are the 68k branch for the amiga retro market and Aros on Linux

Haiku have branched and re-written most of the original OS into 2 flavours. The 32 bit BeOS program compatible version and the new, non-binary compatible 64 bit version. It's on their website.

[shades_aus]

Originally Posted by Olaf Barthel

BeOS is a creation of the 1990'ies, the Amiga operating system goes way back to around 1984/1985. The decisions which went into creating the respective operating system architectures were guided by what the hardware available at the time permitted, and at which cost.

For example, the contemporaries of the Amiga operating system were the operating systems used by the Apple Lisa and the Apple Macintosh. Both were, in terms of design, using established well-known concepts, such as that you would use a 68000 trap instruction to call an operating system function. RAM was expensive, so these machines had little to use (e.g. 64 KBytes, like a Commodore 64).

Also common at the time was the use of the first few "pages" of the adress space for operating system use, such as (I kid you not) was the case for the Commodore 64 (the "zero page"). The Macintosh operating system had a "zero page" and it took Apple years to wean developers off of its use.

Also "also common" was the use of a single shared address space for everything, code, data you name it. This is what the architecture flavour of the year was, as it was.

Fast forward some 10 years and you get BeOS, which did not use a shared memory space and offered POSIX APIs. It even had a Unix-style monolithic kernel to enable all of this. It could afford that because memory had become much cheaper during the past 10 years and the CPUs (yes: BeOS ran on the AT&T Hobbit, then the PowerPC, then the inevitable Intel offering of the day) had become more powerful and cheaper, too.

Unless a lot has changed since I last looked at AROS, it still shares the same early/mid 1980'ies advantages and limitations in terms of architecture which gave us the Amiga operating system. These lines are very hard to redraw unless you are prepared to abandon every bit of software ever written for the platform.

By comparison, BeOS had it "easier". Small bugs and errors do not bring down the operating system as swiftly as for Amiga OS, owing to memory protection and all those small but important foundational elements which Amiga OS lacks.

Sure, both AROS and Amiga OS still see development and incremental change, but it is much harder to build robust software. This is one of the major, major challenges to overcome in terms of "friction". You have to be exceptionally careful not to knock over something by accident, you have to be prudent to play within the narrow limitations of what is considered "safe" in terms of APIs and data structures.

This is 1980'ies style programming at its most sophisticated. Not everybody is cut out for that kind of task, to put it mildly. Passion will get you somewhere, but it will not get you everywhere and anywhere.

Of course not everyone is experienced enough to work through the challenge, I agree and it's part of the reasons I listed, however, it's not impossible to get there.

BeOS also has 2 different versions. The 32bit BeOS binary compatible version and the 64bit variant. Something "similar" should be considered with AMIGA OS if changes are needed to continue forward. As for memory management, some sort of functional MMU should be incorporated.
At least that way, you open up provision for sandboxing in legacy space if you choose to offer it.

I certainly am not advocating that AMIGA OS doesn't develop further. It needs to transition into something better/more modern and that opportunity, still exists, just like every other OS that's still worked on is doing. No one wants AMIGA OSx.x to NOT develop and stay in the dark ages. IBRowse and Miami were made as extensions to get on the web but now, it's too difficult to get more memory (as a shortcoming example) to remain functional on the Web? Amiga was about doing stuff no one said could be done. Like NewTek and the Toaster.
68k "may" have to be retired, like its parent company Motorolla did.

What was that famous saying about getting to the moon? We choose to do it because it is hard? The Amiga "experience" can be realised outside of emulation however, some tougher decisions are going to need to be made. Mac did, Microsoft did, BeOS/Haiku did...

[Samurai_Crow]

Quick note: the 2 versions of BeOS were PPC and x86. Both were 32-bit. Haiku is a BeOS replacement that spawned a 64-bit variant of x86_64 in addition to x86. I use the 64-bit variant on my tower and it's quite performant. I can't wait for the Radeon drivers.

[kamelito]

And AT&T Hobbit

[ma693541]

All this are Offtopic!!!! Stay Ontopic, AmigaOS 3.2.

[bubbob42]

Indeed. No one is going to fundamentally rewrite AmigaOS. It’s retro. Get over it.

[mfilos]

Quick question.

The GlowIcons that are shipped with OS3.2 are really nice work and while they look awesome on 64-256 colors, they seem a lot pixelated under higher resolutions and RTG screens.
Is there a compilation of these icons made without color degration?

[samo79]

Originally Posted by Olaf Barthel

Plenty

There are now some three months worth of bug fixes, changes and optimizations waiting to see a release. Some of the bugs are so old, they survived almost three decades of development work, performed by four developers, including yours truly. Putting your trust in the last guy to have reviewed the code which his changes were built upon (this includes a previous version of myself, of course) is dangerous.

The usual trouble of buffer overflows was to be expected, but there also was a disheartening number of integer overflows, misuse and misinterpretation of API parameters and not validating the parameters to begin with.

Basic functions such as Read(), Write() and Examine() were subtly broken. Not quite so basic functions such as ExNext(), ExAll(), ChangeMode(), MakeLink(), ReadLink(), SetFileSize(), SameLock(), LockRecord()/LockRecords() worked most of the time only because developers would follow the API documentation. Beyond these limitations, things only worked by a combination of luck and really dumb luck. If neither kind of luck was available, you'd be in for undefined behaviour very quickly.

My personal favourites among the (up until then) undiscovered old bugs involved the use of soft links with file notification, ChangeMode() not working consistently and closing a file always having the same "tragicomic" consequences.

mfilos is likely referring to the combination of soft links and file notification not playing together well, limiting their usefulness to exactly "none whatsoever". I found and fixed that one in November last year.

Because these and other bug fixes continued to move the total ROM space budget into a dangerous direction, size optimizations became necessary. Repeated review & rework of the code uncovered further issues, dead code, etc.

ram-handler is probably the operating system component which occupies the two top spots of importance in the operating system architecture and number of implementation issues.

Mmm these bugs are also present in and affect AmigaOS4?

[Olaf Barthel]

Originally Posted by samo79

Mmm these bugs are also present in and affect AmigaOS4?

I don't believe so. The ram-handler for AmigaOS4 had its own challenges, because, for example, virtual memory support in the operating system. What goes into RAM: should stay in RAM and not get paged out to disk. If you wanted that, you would have put it there already.

As with all the components which came from AmigaOS 68k, rewriting and reimplementation went along with bug fixing. I worked on ram-handler back then, for a long time, among other components.

What the AmigaOS 68k ram-handler did not receive was the kind of top to bottom and back, bottom to top review & rewriting it eventually got. Everybody who worked on ram-handler expected that the code was sufficiently robust as it was and changes were limited to getting things to work properly which were discovered as they appeared. Sadly, the code really needed much more work, but that could be said about the entire AmigaOS 68k operating system back when the work on AmigaOS 3.1.4 started.

[prometeo]

I'd like to ask a question to AmigaOS developers (at least the current ones).
Do they think that the actual status of AmigaOS code quality is fine or there is more space for improvements?
I'm not talikng about new futures but, for example, reimplementing some algorithms to be O(logn) instead of O(n2) or simplify some code without losing functionality.
I know that there are some assembly rewriting for a few parts in order to win back enough ROM space to fit new and fixed libraries and modules, but I'm curious...
TIA,
Giacomo.

[boemann]

Originally Posted by prometeo

I'd like to ask a question to AmigaOS developers (at least the current ones).
Do they think that the actual status of AmigaOS code quality is fine or there is more space for improvements?
I'm not talikng about new futures but, for example, reimplementing some algorithms to be O(logn) instead of O(n2) or simplify some code without losing functionality.
I know that there are some assembly rewriting for a few parts in order to win back enough ROM space to fit new and fixed libraries and modules, but I'm curious...
TIA,
Giacomo.

Not that many places where there are algorithms with high n. But yes I'm sure there are still places where such improvements can be done. Doing profiling to find such bottlenecks is not super easy on the Amiga though, and going through the code looking for slow algorithms is not the best approach either.

As for saving ROM we rarely if ever go toward asm - in fact we tend to go the other way, and choosing to focus on cleaning up algoritmically instead.

What I can say is that we are university educated developers so we are well aware of topics like this.

[Thomas Richter]

Originally Posted by prometeo

I'd like to ask a question to AmigaOS developers (at least the current ones).

I dare to answer, despite not being an AmigaOs developer.

Originally Posted by prometeo

Do they think that the actual status of AmigaOS code quality is fine or there is more space for improvements?

As with every larger software project, it is rather a pretty mixed bag. There are some parts that are in quite good shape (intuition, datatypes) and some parts that are in pretty desolate shape (graphics in particular, also the console.device).

Originally Posted by prometeo

I'm not talikng about new futures but, for example, reimplementing some algorithms to be O(logn) instead of O(n2) or simplify some code without losing functionality.

I wouldn't know off my head to which parts this applies. There were in the past such cases, layers is such an example.

Originally Posted by prometeo

I know that there are some assembly rewriting for a few parts in order to win back enough ROM space to fit new and fixed libraries and modules, but I'm curious....

To give you my two cents: This is the wrong direction. RAM is cheaper today than it was when AmigaOs was put together (avoiding the term "designed"), and SD-disks are faster today than disks were back then. ROMs are hard to update, but modules on disk are easy to replace. Thus, modules need to go out of ROM, not into ROM.


There are stil a couple of modules that are not essential for booting, and should rather go out. The audio.device, mathffp, mathieeesingbas, the RAM-Handler, ramdisk.device and probably some others I forgot - are not essential for booting the machine. These could likely go without much impact.

[Gorf]

Originally Posted by OlafSch

No. AROS was created to get it running on a better ISA (X86).

please define: "better ISA"

[shelter]

But there will be more updates? What's the plan/roadmap?

[Nbdy]

Is it possible to put 3.2 ROM in A500 512kb+512kb ???

Have the hyperion CD, would like to burn the rom to try, what for an eprom I need ???

[prometeo]

Originally Posted by boemann

As for saving ROM we rarely if ever go toward asm - in fact we tend to go the other way, and choosing to focus on cleaning up algoritmically instead.

Maybe then I simply misunderstood some comments read here and there about 3.2 development, my bad. And a much better choice, IMHO.


Thanks for the answer!
Giacomo.

[prometeo]

Originally Posted by Thomas Richter

I dare to answer, despite not being an AmigaOs developer.

Well, I guess not many people are more knowledgeable than you about AmigaOS.

Moving lots of modules out of ROM seems like a win-win situation.
Hopefully this will allow better versions of graphics and console.
Thanks,
Giacomo.

[Olaf Barthel]

Originally Posted by prometeo

I'd like to ask a question to AmigaOS developers (at least the current ones).
Do they think that the actual status of AmigaOS code quality is fine or there is more space for improvements?
I'm not talikng about new futures but, for example, reimplementing some algorithms to be O(logn) instead of O(n2) or simplify some code without losing functionality.
I know that there are some assembly rewriting for a few parts in order to win back enough ROM space to fit new and fixed libraries and modules, but I'm curious...
TIA,
Giacomo.

AmigaOS 68k code quality is decidedly mixed on account that the developers who created it took different approaches to solve problems over the decades. You have the slickest, clearest and the best in exec and Intution, for example, but not exactly the same degree of vision, care and pressure (!!!) went into building and maintaining hard disk partitioning software.

On the whole, I would say that the degree of quality is in line with and often exceeds what the industry as a whole could produce in the 1980'ies and early 1990'ies. After all, the Amiga is a microcomputer which would offer preemptive multitasking in 1985 when other competitors had just mastered the art of rendering black and white bitmap imagery on a small screen at high speed using only the 68000 CPU. Just saying

That said, there is only so far which a handful of data structures and algorithms will take you without bumping into limitations some 20-30 years on. The Amiga operating system uses linked lists (both kinds: doubly-linked and singly-linked), tables, bit maps and (very rarely) trees as its building blocks. Binary trees are uncommon. Hashing is used in the file system and also graphics.library. That is basically it. As far as I can tell, binary trees have been absent up until AmigaOS 3.5 came around.

The Amiga file systems, which includes RAM:, make use of tables, hashing and trees, but they scale poorly. You do not necessarily notice it, because the underlying storage hardware hierarchy is so good at slowing things down that the drawbacks of the scalability limitations will not kick in until you have almost filled up your multi-gigabyte volume. For RAM: it is a different story because it basically works like the disk file system does, using data structures which share the same limitations as the default Amiga file system.

Not much can be done about the default Amiga file system, but there is plenty of room for improvement in the RAM: file system on the condition that ROM space allows for it, which it currently (AmigaOS 3.2) does not. This is where the rubber meets the road: can you spend ROM space or RAM on making this scale better, i.e. massively save CPU time?

It can be done, actually. The AmigaOS4 RAM: file system makes use of red-black trees in order to make access to individual directory entries faster. If you want to open a file among thousands of other files stored in the same directory, the AmigaOS4 RAM: file system can do it in O(log(n)) time. But on AmigaOS 3.2 and its precursors it is still O(n), on a clear day with fair weather.

The use of singly-linked lists limits scalability in Intuition and, of course, AmigaDOS. You may not want to use thousands of AmigaDOS assignments, volumes and devices because they all share the same list, which is ordered by type. Opening hundreds of windows on a screen will get slower and slower still, not just because of clipping and buffering work. This could be remedied, but the real limitations lie within the APIs.

exec.library uses the doubly-linked list as its building block for everything. Because so many programs will retrieve the exec.library base pointer from address 4 (which in days of old would reside in non-cacheable chip memory) and then call an exec.library function, you had a built-in slow-down there. This did not necessarily matter much, except when it did. For example, the ARexx support code which is used for changing and retrieving the value of variables (Rexx Variables Interface: RVI) opens rexxsyslib.library for each access, retrieving the exec.library base from address 4. Big deal (one of the biggest users of the RVI functions is the workbench.library ARexx interface)? Well, this code actually becomes faster if rexxsyslib.library is at the beginning of the library list. I kid you not... This is where the use of doubly-linked lists in exec.library hits its limitations. The same data structure is used for the memory management subsystem: same problem, especially if memory fragmentation is high. The exec.library memory management implementation is as slick as it can be, but it could be considered naive, compared to other designs (e.g. the buddy allocator, which even in 1985 was already 20 years old). It scales poorly, but it was more than able to deal well with as little/much RAM as the Amiga 1000 shipped with.

Prominent areas in which scalability issues were identified and addressed include the asl.library file requester (AmigaOS 3.5), the workbench.library drawer text mode display (AmigaOS 3.5) and the catalog string lookup code in locale.library, plus the locale-enable formatted output conversion code (AmigaOS 3.2).

The asl.library file requester in Workbench 2.1 through 3.1 would use a doubly-linked list to store the names of directory entries as these were being read, using straight insertion (actual algorithm name!). While this made for a very clear and compact implementation it ran into its share of scalability issues as early as 1992. This was the age of the CD-ROM drive and the ISO 9660 file system format stored the contents of directories in alphabetically sorted order (which brings out the worst in the straight insertion algorithm). Now imagine you are reading from a double speed (300 KByte/s!) CD-ROM drive and happen to read a directory with a few hundred files. Your Amiga would crawl, then crawl more slowly, more slowly still, etc. while the CPU load would jump, then jump even more. This was one of the reasons why, at the time, there were so many helper tools which redirected the asl.library file requester to use reqtools.library, for example, or the little acorn which MUI grew out of: MagicFileRequester.

The same problem would exist in Workbench 2.0 through 3.1, if you switched a drawer to show the names of the files and directories within. The more data there were to display, the slower Workbench would be in displaying it.

I found a solution for these problems by changing the underlying data structure. My approach was to combine a red-black tree with the doubly-linked exec list. Insertion went from O(n^2) to O(log(n)) instantly. Note that it could have been possible to sort the respective list with mergesort, for example, but my approach has the advantage of not having to rebuild the entire list every time a new entry is added (or whenever enough changes have been made to warrant a resort). The list remains sorted at all times, and you do not notice that it is being sorted as it is being read (reading never stops and freezes to resort the list).

The locale.library is responsible for wrangling the catalog files which provide translations of the original program text into a different language. Each such text is identified by a numeric key, and there is a function which takes that key, looks up the corresponding text (if any) and returns it. How do you make this efficient? Firstly, you make sure that the data structures in the catalog file need no complex pre-processing in order to be suitable for text lookup. locale.library can load it and transform that data into a singly-linked list with very little effort, yay Singly-linked list, hm... we have been here before, haven't we? How do you look up the text? locale.library will walk down the list until it finds what you asked it to retrieve. What if that list contains hundreds of entries? You have a delay loop there, not a lookup function.

For AmigaOS 3.2 we reworked the lookup code so that it can be, optionally, transformed into a binary search (O(n) replaced by O(log(n)), available memory permitting. Also, this speed optimization does not kick in unless it seems likely to yield a benefit. Hence, you will not get it if the catalog contains just a handful of entries.

Because no good deed goes unpunished while we were taking in the sights, we also found something in locale.library which impacted the performance of the entire operating system and every program negatively. The exec.library RawDoFmt() function is used prominently and massively. And locale.library patches itself into it, so that it can enable parameter processing and reordering for output. Turns out that this involved a certain degree of preprocessing and setup work before even the first character was ready for output (also: requires a lot of free stack space!). locale.library basically assumed that parameter reordering would be called for and would spend the effort on making sure it was ready for anything. Sadly, it took much more time to be prepared for parameter reordering than producing the output, especially for something simple as "%lc" (which produces a single character). Parameter reordering is very rarely used, so we added custom code in AmigaOS 3.2 which figures out if it is indeed necessary, and if not, skips all that complex setup. Even with that initial test included in the budget, that code is now much, much faster than it had been in the past 30 years.

These are a few funny examples of what you can achieve if you stay curious and try to make a difference for the next operating system release to ship. That is, if compatibility, memory and CPU performance constraints are, just for once, manageable

[Honitos]

Thank you for this deep dive into some code areas of the OS, Olaf.
It was exciting to read!

[shelter]

^^^^^holy batmobile, Barthelman! That was the reply of the century.
Well done!