AmigaOS 3.2 performance vs 3.1

[ShK]

How to move AmigaOS 3.1 "expansion.library" into fast memory without MMU?

[Thomas Richter]

Originally Posted by DanyPPC

Is it possible to use MuMove4K as a BlizKick Module avoiding a second reboot ?

It's in the manual. Yes, really. LoadModule and MuMove4K both have a "NOREBOOT" option, so whatever comes first in your startup-sequence, add "NOREBOOT" to that command.

[Thomas Richter]

Originally Posted by ShK

How to move AmigaOS 3.1 "expansion.library" into fast memory without MMU?

Why? The expansion.library is only used to identify and configure the hardware, and not any time further.


To answer your question, you cannot. The purpose of the library is to find autoconfiguring RAM, and for that reason, it cannot be in the RAM it is supposed to configure. Exec is (re-)constructed after expansion has run, and thus, if autoconfig RAM is present, is moved to fast RAM.

[ShK]

Originally Posted by Thomas Richter

Why?

Just out of technical interest.

[Amiga68k]

Originally Posted by Thomas Richter

It's in the manual. Yes, really. LoadModule and MuMove4K both have a "NOREBOOT" option, so whatever comes first in your startup-sequence, add "NOREBOOT" to that command.

That it's stated in the manual, doesn't make it necessarily true ;-) You are correct when using MuMove4K in a LoadModule combination, then only one reboot is needed. However, in this specific scenario, we were discussing a BlizKick combination, then it doesn't work (trial & error).

Unfortunately the specific modules that are provided with the BlizKick package stopped working with 3.1.4+ roms.

P.S. because the "LoadModule method" only needs one reboot and there is more freedom to exchange various "modules", I prefer the LoadModule method. This comes with the extra benefit of being able to also use MuProtectModules for instance.

When using LoadModule the startup of my system looks like this (roughly):
1. MuMove4K NOREBOOT
2. LoadModule L:System-Startup NOMEMFKICK REVERSE ROMUPDATE
3. MuFastZero ON FASTEXEC MOVESSP
4. MuFastRom ON PROTECT
5. MuProtectModules ON REMAP

When using BlizKick it looks like:
1. BlizKick ROM
2. MuMove4K
3. MuFastZero ON FASTEXEC MOVESSP

[meynaf]

Originally Posted by Thomas Richter

Why?

To free a small chunk of precious chipmem ?

Originally Posted by Thomas Richter

The expansion.library is only used to identify and configure the hardware, and not any time further.

Then why does it stay in the system after that ? I've seen open count of 4 under WB.

[Thomas Richter]

Originally Posted by meynaf

To free a small chunk of precious chipmem ?

About 1K? Serious?

Originally Posted by meynaf

Then why does it stay in the system after that ? I've seen open count of 4 under WB.

Because "after that" is "after all hardware expansions have been configured", and depending on your hardware, this can be rather late. BindDrivers is one program that is relatively late, RTG hardware is also pretty late when the RTG stack is loaded.

[Don_Adan]

Originally Posted by Thomas Richter

About 1K? Serious?

Dependent where this chunk is allocated in chip mem.
It can waste only 1 KB of chip memory, but in wrong place it can waste much more chip memory f.e 5KB.
Freeing later this if is unnecessary is good idea.
Or simple make mirror (copy) in fast memory, and free later.

[meynaf]

Originally Posted by Thomas Richter

About 1K? Serious?

Yes, serious. Remember that the waste of one individual, though not much, is added to everyone else's waste. Little things add together. There are no small savings.

Originally Posted by Thomas Richter

Because "after that" is "after all hardware expansions have been configured", and depending on your hardware, this can be rather late. BindDrivers is one program that is relatively late, RTG hardware is also pretty late when the RTG stack is loaded.

This doesn't explain why the library stays open after everything has been configured.

[Thomas Richter]

Originally Posted by meynaf

Yes, serious. Remember that the waste of one individual, though not much, is added to everyone else's waste. Little things add together. There are no small savings.

Then let's avoid wasting everybodies time in such a pointless discussion.

Originally Posted by meynaf

This doesn't explain why the library stays open after everything has been configured.

It doesn't stay open, or at least not unless every user closes it correctly after having opened it. It stays in memory, and the reason is quite simple: Because it is unclear at which time everything is configured. Despite, it doesn't cost a lot of memory in first place.

[meynaf]

Originally Posted by Thomas Richter

Then let's avoid wasting everybodies time in such a pointless discussion.

It is easy to say a discussion is pointless when you're wrong.

Originally Posted by Thomas Richter

It doesn't stay open, or at least not unless every user closes it correctly after having opened it. It stays in memory, and the reason is quite simple: Because it is unclear at which time everything is configured. Despite, it doesn't cost a lot of memory in first place.

Well, the problem is that precisely : it does stay open. So every user does not close it correctly, apparently. Who are these users anyway, I don't know.
Else it would be fine, it is open first time in chipmem, then closed, then flushed if necessary, then eventually reopened and able to go in previously configured fastmem.
Remember that even though 1k isn't much, it can still turn an allocation that works into an allocation that fails. Sometimes every kilobyte counts, especially for chipmem.

[Thomas Richter]

Originally Posted by meynaf

It is easy to say a discussion is pointless when you're wrong.

It is pointless to discuss with ignorant people lacking the background. Here is the background for educating you.

Originally Posted by meynaf

Well, the problem is that precisely : it does stay open.

A library is "open" if its open count is non-zero. Otherwise, it is not open. That is something *different* from "resident in RAM". A library that is resident in RAM and closed can *decide* by itself to be no longer resident in RAM by unloading itself when flushed. Once it is needed again, it is loaded or recreated by ramlib. However, ROM based libraries usually do not implement this function, though in principle some could. dos, exec, graphics and many many other cannot be flushed simply because it is assumed that everybody will need them all the time anyhow. That intuition can be flushed is relatively recent and was added in v47 to be able to update it even if already loaded, and it is a very rare exception of a ROM-based system library implementing this feature.



The expansion library neither can be flushed since it *also* carries the database of the system expansions it identified, and these system expansions cannot be re-detected once the system is running, thus its data base and library base need to remain in RAM until the system is shut down or rebooted.

Originally Posted by meynaf

So every user does not close it correctly, apparently. Who are these users anyway, I don't know.

I already told you who typical users of expansions are. Dare to read? The typical users such as BindDrivers and RTG functions (though possibly others) will *close* the library (unless forgetting this step by mistake), but that is not related to "removing it and its database from RAM". That is an independent step, which, however, would be a really bad idea since the expansion database cannot be rebuild once autoconfiguration has finished and its config nodes may be still in use.

Originally Posted by meynaf

Else it would be fine, it is open first time in chipmem, then closed, then flushed if necessary, then eventually reopened and able to go in previously configured fastmem.

Which is not possible has hardware redetection is not possible once the system is up, and it is also fairly pointless as the amount of data the autoconf database takes is less than the number of bytes of this post. That hardware redetection is not possible is due to the way how the autoconf hardware works, by a serial config-chain that is only reset by a hardware reset, which - apparently - also de-configures autoconfiguring RAM.


The database can neither be relocated since there is no "obtain and release" interface to it. You only get the ConfigDev corresponding to a hardware, and then hold on it. A program has no means to "release" an already obtained configuration, thus it cannot be removed under the feed of a program either.

Originally Posted by meynaf

Remember that even though 1k isn't much, it can still turn an allocation that works into an allocation that fails. Sometimes every kilobyte counts, especially for chipmem.

I remember that <1K is really not much memory indeed. it's making a rumble for nothing probably for the purpose of making a rumble and nothing else, especially without knowing the details and the background.

[Firestone]

Originally Posted by Amiga68k

Unfortunately the specific modules that are provided with the BlizKick package stopped working with 3.1.4+ roms.

P.S. because the "LoadModule method" only needs one reboot and there is more freedom to exchange various "modules", I prefer the LoadModule method. This comes with the extra benefit of being able to also use MuProtectModules for instance.

When using LoadModule the startup of my system looks like this (roughly):
1. MuMove4K NOREBOOT
2. LoadModule L:System-Startup NOMEMFKICK REVERSE ROMUPDATE
3. MuFastZero ON FASTEXEC MOVESSP
4. MuFastRom ON PROTECT
5. MuProtectModules ON REMAP

When using BlizKick it looks like:
1. BlizKick ROM
2. MuMove4K
3. MuFastZero ON FASTEXEC MOVESSP

Thanks for this reply @Amiga68k !
I've been doing experiments with combinations of using Blizkick and LoadModule to get the performance back, and it seems like the puzzle is solved now thanks to all you guys.

I tried both the Blizkick and LoadModule-approach, and it seems like the LoadModule-approach is the one that seems to give me most performance and the cleanest installation of AmigaOS3.2

I tried with and without MMULib, MuMove4k, MuFastZero and so on, and the setup that revived the old speed (and even surpass 3.1 speeds in some cases) was the LoadModule-approach from the post above.

It will only need one reboot which also was one of my goals. Intuition feels snappy again and disk loads are as fast as I can remember them.

The only thing I'm wondering now is how the procedure is to migrate the system from a ROM-based installation to a disk-based installation.
How does the installer detect this, and is it possible to run the installer again when the system is loaded with LoadModule? Is it then aware that the system is diskbased, or will it still detect the system as a ROMbased-system like it will when ROM is kicked using Blizkick?

And one last question: Why is MuMove4k not a part of the MMULib disk for AmigaOS3.2? It's present in the Aminet archive.

Originally Posted by DanyPPC

Is it possible to use MuMove4K as a BlizKick Module avoiding a second reboot ?
I see a MuMove4K module on Blizkick archive, but I don't know if it's obsoleted.
EDIT: for me the old MuMove4K BK module works, exec is moved in fast, the system runs better and the IDE speed get 2,0 MB/s vs 1,5MB/s. And only one Reboot !

For me Blizkick with MuMove4k didn't work. Exec was still in chip.

[Don_Adan]

Sorry, but which problem is updating expansion.library for extra/missing calls for new kick 3.3?
If something can be made better than for old kickstarts then can be done.
Same for support for files over 2GB. And over 2GB RAM limit.
Only good idea is necessary.

[Thomas Richter]

Originally Posted by Firestone

How does the installer detect this, and is it possible to run the installer again when the system is loaded with LoadModule?

it probably does not detect it. The startup sequence left by the installer makes a version check and only runs into LoadModule if it is needed, so whatever is installed as startup-sequence *by default* does the right thing regardless of the ROM present.

Originally Posted by Firestone

And one last question: Why is MuMove4k not a part of the MMULib disk for AmigaOS3.2? It's present in the Aminet archive.

Because the goal of an operating system distribution is a little bit different. The purpose of the distribution is to make it easily accessible to the user, and have a running and stable system once it is installed. This goal was obtained. Optimizing and tweaking the system is up to the user if it is even necessary. The system runs and it stable with the default startup-sequence. Every tweaking beyond this point is a bit hardware specific and not so easy.

[Thomas Richter]

Originally Posted by Don_Adan

Sorry, but which problem is updating expansion.library for extra/missing calls for new kick 3.3?

Because just because you update a library does not mean that you update its users. Once again: FindConfigDev() of expansion *passes out* a pointer to an expansion library database structure (a ConfigDev). There is no interface to *release this pointer again*. Even if a future release would provide such a function, the library cannot assume that every user calls this function, and thus, the design is stuck. Or, to put it differently:


"It's as in real life: Once you make a mistake, you have to support it for your lifetime."



If expansion is ever supposed to be updated, then rather into the direction of detecting autoconfiguring hardware on other bus systems and provide them to the user, but transparently with existing interfaces. That is a potential and useful development direction, and not to worry about its minimal RAM footprint (and it is really minimal compared to other system libraries).

Originally Posted by Don_Adan

Same for support for files over 2GB. And over 2GB RAM limit.
Only good idea is necessary.

That is exactly the same question and has exactly the same reasons. It does not help to update a library if you do not update its users. See RKRM AmigaDOS. How to represent a file size >2GB in a FileInfoBlock where the file size is indicated as LONG? What happens if the file size in the FileInfoBlock does not correspond to its true file size? What does a Seek() with a OFFSET_BEGINNING and a negative argument mean? Can we depend that this is actually a seek beyond 2GB, or is it actually an error by attempting to seek backwards?



At this point, my usual canon starts: The problem is that the *interface* of the dos.library is not properly defined to handle such cases, and it is too late to change an interface if we have implementations in the wild that instead depend on the actual implementation details since the documentation of this interface was so lousy (and in case of the dos.library, that was unfortunately really the case).

[Firestone]

Yeah the idea with the goal makes sense, Thomas.

But is the installation the same regardless of rom or disk based installations? I mean, will all the same files be copied to the hard drive, or will something be excluded because it’s in the rom “anyway” ? For example classes, libs, handlers and so on?

[patrik]

Originally Posted by Amiga68k

When using BlizKick it looks like:
1. BlizKick ROM
2. MuMove4K
3. MuFastZero ON FASTEXEC MOVESSP

For the Blizzard1230, given that it is a 030, the highest performing system would be achieved by:
1. BlizKick ROM
2. FastExec
3. Disable mmu.library

FastExex does the same as MuMove4K and MuFastZero - move the library base of exec.library to fastmem and optionally move SSP too - this is what gives the snappy feel, but without using the MMU. Not enabling the MMU on the 030 (with a fairly complicated config) will give you anywhere from 10-20% extra performance when doing things that uses some memory bandwidth, like networking, extracting archives etc.

[meynaf]

Originally Posted by Thomas Richter

It is pointless to discuss with ignorant people lacking the background. Here is the background for educating you.

If you want to "educate" me in one way or another, you're welcome but start by reading what I post. I explicitly wrote that in my case the observed open count was nonzero.

Originally Posted by Thomas Richter

A library is "open" if its open count is non-zero. Otherwise, it is not open. That is something *different* from "resident in RAM". A library that is resident in RAM and closed can *decide* by itself to be no longer resident in RAM by unloading itself when flushed. Once it is needed again, it is loaded or recreated by ramlib. However, ROM based libraries usually do not implement this function, though in principle some could. dos, exec, graphics and many many other cannot be flushed simply because it is assumed that everybody will need them all the time anyhow. That intuition can be flushed is relatively recent and was added in v47 to be able to update it even if already loaded, and it is a very rare exception of a ROM-based system library implementing this feature.

Guess what, I already knew all this ! (except that v47 detail which i don't care about anyway)
However, as stated above (several times now) the open count of the library (as shown by ARTM) is nonzero.

Originally Posted by Thomas Richter

The expansion library neither can be flushed since it *also* carries the database of the system expansions it identified, and these system expansions cannot be re-detected once the system is running, thus its data base and library base need to remain in RAM until the system is shut down or rebooted.

This contradicts what you wrote in post #23.

Originally Posted by Thomas Richter

I already told you who typical users of expansions are. Dare to read? The typical users such as BindDrivers and RTG functions (though possibly others) will *close* the library (unless forgetting this step by mistake), but that is not related to "removing it and its database from RAM". That is an independent step, which, however, would be a really bad idea since the expansion database cannot be rebuild once autoconfiguration has finished and its config nodes may be still in use.

Dare to read ?? Yourself first !
I still observe an open count of 4 under WB (how many times i wrote this already ?) - and this is without either BindDrivers being run nor RTG as none of them is used in my system. So who are other clients ?

Originally Posted by Thomas Richter

Which is not possible has hardware redetection is not possible once the system is up, and it is also fairly pointless as the amount of data the autoconf database takes is less than the number of bytes of this post. That hardware redetection is not possible is due to the way how the autoconf hardware works, by a serial config-chain that is only reset by a hardware reset, which - apparently - also de-configures autoconfiguring RAM.

Ok but it does not mean we can't put that data to fastmem. We do not need a hardware redetection to be able to move data around.

Originally Posted by Thomas Richter

The database can neither be relocated since there is no "obtain and release" interface to it. You only get the ConfigDev corresponding to a hardware, and then hold on it. A program has no means to "release" an already obtained configuration, thus it cannot be removed under the feed of a program either.

The database no, but the library yes. So the library itself can in theory be moved as it only contains a pointer to the database.

Originally Posted by Thomas Richter

I remember that <1K is really not much memory indeed. it's making a rumble for nothing probably for the purpose of making a rumble and nothing else, especially without knowing the details and the background.

Well, no. I just don't want to waste chip memory to design mistakes (that can be fixed).

Originally Posted by Thomas Richter

Because just because you update a library does not mean that you update its users. Once again: FindConfigDev() of expansion *passes out* a pointer to an expansion library database structure (a ConfigDev). There is no interface to *release this pointer again*. Even if a future release would provide such a function, the library cannot assume that every user calls this function, and thus, the design is stuck.

And what about being a little clever, for once ?
Perhaps the design isn't really stuck.
The library itself has nothing that can not be moved (once it is no longer open).
The database can be put in fastmem if said fastmem is found before database is allocated.
Is there a reason why expansion can't find memory first, then write the config down ?
If yes, expansion can still reallocate current data right when fastmem is finally found.

Originally Posted by Thomas Richter

"It's as in real life: Once you make a mistake, you have to support it for your lifetime."

Not really. Not fixing a mistake is making another mistake.

Originally Posted by Thomas Richter

If expansion is ever supposed to be updated, then rather into the direction of detecting autoconfiguring hardware on other bus systems and provide them to the user, but transparently with existing interfaces. That is a potential and useful development direction, and not to worry about its minimal RAM footprint (and it is really minimal compared to other system libraries).

RAM footprint isn't minimal as chipmem is severely limited in comparison to fastmem (which is where other libraries reside).

Originally Posted by Thomas Richter

That is exactly the same question and has exactly the same reasons. It does not help to update a library if you do not update its users.

In our current case it does not seem updating the users is necessary.

Originally Posted by Thomas Richter

How to represent a file size >2GB in a FileInfoBlock where the file size is indicated as LONG?

By changing that definition to ULONG ? That wouldn't break old programs, you know.

Originally Posted by Thomas Richter

What happens if the file size in the FileInfoBlock does not correspond to its true file size?

Bad things. But in case of 2 to 4GB, it *does* correspond.

Originally Posted by Thomas Richter

What does a Seek() with a OFFSET_BEGINNING and a negative argument mean? Can we depend that this is actually a seek beyond 2GB, or is it actually an error by attempting to seek backwards?

Listen, seeking backwards is seeking outside of the file. Seeking beyond 2GB in a file that's smaller than 2GB is seeking outside of the file. So no change in comparison to what we had before. Where is the problem then ?

Originally Posted by Thomas Richter

At this point, my usual canon starts: The problem is that the *interface* of the dos.library is not properly defined to handle such cases, and it is too late to change an interface if we have implementations in the wild that instead depend on the actual implementation details since the documentation of this interface was so lousy (and in case of the dos.library, that was unfortunately really the case).

The interface does not really need to change. Only addition is that what was previously an error now has a meaning.

[Thomas Richter]

Originally Posted by meynaf

Guess what, I already knew all this ! (except that v47 detail which i don't care about anyway)

If you don't care about new releases, then why all the rumble?