RTG API replacement - Lets finally do this

[Heiroglyph]

Given all the recent drama regarding the current situation in Amiga RTG land, I propose that we finally sit down and create our own open standard and have community control over our own future.

At this point, we can't count on the potentially opened CGX sources. I'm afraid that may be tied up in license hell for some time to come. If we do get that, great, but we can't count on it.

I suggest that we start pulling from and feeding into the Aros CGX-like implementation since nobody seems to have issues with the legal status and it is pretty far along already.

I do have issues with the Aros codebase that are directly related to their development process.

They jumped to an new architecture and completely foreign compiler without making 68k stable first.

That in turn made testing any one component a nightmare and introduced code that has seemingly more macros than actual code. It's like reading a foreign language in places.

I strongly believe that if they had replaced just one library at a time on a 68k system using all other stock libraries, we'd have a stable, fast 68k Aros by now.

This also allowed them to weave in Aros specifics throughout all components to the point that it's almost all or nothing. For example, graphics devices currently need their HID, etc.

So IMHO, that makes Aros more like a reference than usable code, but it's still very valuable.

My suggestion is to do what they could have done to start with.

1. Use a real Amiga compiler such as vbcc since it is still developed and the developer is quick to fix bugs or add features if the Amiga community needs them.

2. Using Aros code as a reference, write a new, APL licensed Graphics.library compatible replacement that doesn't use any Aros specific code. It should drop into any Amiga system and just work. At this point, even Picasso96 and CGX should still work with this new library because it's a OS3.x work-alike.

3. Do this for each library that requires heavy patching for RTG.

4. Port the Aros RTG code to our new libraries. This may or may not break the ability to run Picasso96 or CGX on our libraries.


This approach also gives us a start on being in control of the OS itself.

The existing version of OpenPCI gives us a way to support Mediator PCI backplanes without signing an NDA.

I have the authority to relicense the OpenPCI source as I see fit as long as it's open, so this could go APL in order to support future PCI backplanes.


What are your thoughts?

Also posted at http://www.amiga.org/forums/showthread.php?t=70973

[Locutus]

1. VBCC definitely sounds like the best idea.

2/3/4; So, actually what i'm wondering here. do you plan a RTG system Ala CGX/P96 that patches the hell out of preexisting Graphics/Layers/Intuition. Or a AROS-derived-where-possible clean replacement with 'RTG functionality' integrated?

The second option sounds like the sanest choice as that also opens up the possibilities of further development of those components.

In any case, if you have the time to do this (alas i don't) you are a hero :-)

[wawa]

i think there has been some work towards vasm being an option to gcc here. two interesting threads that come to my mind:
http://eab.abime.net/showthread.php?...ight=aros+vasm
http://eab.abime.net/showthread.php?...ight=aros+vasm
sorry for my own unedicated input therein.

[Toni Wilen]

I strongly believe that if they had replaced just one library at a time on a 68k system using all other stock libraries, we'd have a stable, fast 68k Aros by now.

Not really.. It is impossible to replace "hardware/low level level" libraries (like exec, expansion, graphics, intuition, layer etc..) just one by one because AmigaOS does not really have any kind of real private data, many libraries access other libraries undocumented/private data directly or do other undocumented things.

Result would have been AmigaOS clone that has exact same limitations and restrictions (see below).

2.

Impossible. AROS is totally different internally and has built-in low level graphics driver support. AROS can have different low level device driver for different graphics mode IDs automatically, without any kind of stupid patching. This is not possible with AmigaOS. (and which makes AmigaOS RTG support total mess!)

[nogginthenog]

Toni is correct. The cybergraphics API is not hard. The difficult bit is the patches CGFX/P96 apply to AmigaOS.

[Heiroglyph]

Originally Posted by Toni Wilen

Not really.. It is impossible to replace "hardware/low level level" libraries (like exec, expansion, graphics, intuition, layer etc..) just one by one because AmigaOS does not really have any kind of real private data, many libraries access other libraries undocumented/private data directly or do other undocumented things.

Result would have been AmigaOS clone that has exact same limitations and restrictions (see below).

That's good to know.

I'm OK with having the same limitations at first when replacing libraries so that it can be validated as a direct replacement.

It's possible that to get an RTG system working in a reasonable timeframe, it might have to patch the existing libraries just like P96 and CGX did.

It's not pretty for long-term use, but if it works it works and I can move on to other projects.

Impossible. AROS is totally different internally and has built-in low level graphics driver support. AROS can have different low level device driver for different graphics mode IDs automatically, without any kind of stupid patching. This is not possible with AmigaOS. (and which makes AmigaOS RTG support total mess!)

I'm not talking about a straight port of AROS components, but rather using it as a reference for "this is how they made it work" and "Oh, I can reuse this part verbatim".

[matthey]

Originally Posted by Heiroglyph

That's good to know.

I'm OK with having the same limitations at first when replacing libraries so that it can be validated as a direct replacement.

It's possible that to get an RTG system working in a reasonable timeframe, it might have to patch the existing libraries just like P96 and CGX did.

CGFX/P96 patched everything at least because of the following problems.

1) graphics.library is in ROM
2) they didn't want to recreate the low level Amiga chipset support

The result is a patching nightmare but it does work better than I ever thought it would have (both CGFX and P96 work and are stable on a wide range of Amigas). If wanting to create a better replacement, then it would be good to create a whole new graphics.library IMO. Leaving out the custom chipset hardware banging gets you what AROS has now. Including it has two big issues.

1) C is probably not going to give the best code quality on the 68k
2) changing the hardware banging chipset code could cause compatibility problems and not changing them could violate copyrights

I support most of your conclusions about how to go about improving/replacing the AmigaOS but it is a difficult task with all the road blocks. There isn't a modern compiler which generates good quality code for the 68k, including vbcc. Yes, it has potential and I have worked on it myself (bug fixes, ease of use, inlines, vclib code, better C99 support, improved FPU support, better GCC compatibility with builtin.lib, etc). I had a conversation with Frank Wille recently about improving the code generation. I told him that old CPU targets like the 68k are unlikely to get improved support without new hardware and standards (Volker may still make some improvements). Then you have the AmigaOS 68k road blocked by the A-EON/PPC fans who demote 68k users to 2nd class citizens and censor them. They have blocked more 68k Amiga development than their broken promises and fake support ever suggested. They ignore reasonable plans, old AmigaOS developers and the bigger Amiga 68k market including FPGA technology. They are still trying to get us to upgrade to overpriced, outdated and incompatible hardware designs. If A-EON doesn't have majority control of Hyperion after bailing them out of bankruptcy then they don't have any business sense either. Maybe it would be better to wait until the assets are for sale after they fail and have the community buy them. Of course, you may still have a problem with the crooks at Amiga Inc and their claims.

[idrougge]

</end matthey off-topic rant>

[Heiroglyph]

Originally Posted by matthey

CGFX/P96 patched everything at least because of the following problems.

1) graphics.library is in ROM
2) they didn't want to recreate the low level Amiga chipset support

The result is a patching nightmare but it does work better than I ever thought it would have (both CGFX and P96 work and are stable on a wide range of Amigas). If wanting to create a better replacement, then it would be good to create a whole new graphics.library IMO. Leaving out the custom chipset hardware banging gets you what AROS has now. Including it has two big issues.

1) C is probably not going to give the best code quality on the 68k
2) changing the hardware banging chipset code could cause compatibility problems and not changing them could violate copyrights

I support most of your conclusions about how to go about improving/replacing the AmigaOS but it is a difficult task with all the road blocks. There isn't a modern compiler which generates good quality code for the 68k, including vbcc. Yes, it has potential and I have worked on it myself (bug fixes, ease of use, inlines, vclib code, better C99 support, improved FPU support, better GCC compatibility with builtin.lib, etc). I had a conversation with Frank Wille recently about improving the code generation. I told him that old CPU targets like the 68k are unlikely to get improved support without new hardware and standards (Volker may still make some improvements).

My plan is to start by patching in anyway.

I assumed I'd start one entry point at a time.
Write a work-alike (as much as is feasible)
Run a test app that makes some test calls, patches in my replacement, then makes the same test calls again and compare the result.

That will pretty quickly tell me where there is internal data and I can potentially put off the chipset functions until later.

The chipset parts are mostly available in Aros from what I understand though, they're just slow. They might want some of this code if we can improve 68k performance. It is the only platform I'm targeting with this, so it will have to be reasonably fast or else be rewritten.

I want to stick with C because ASM isn't clear enough for most people to contribute or even just read.

I'm confident that it will be fast enough for the most part with only certain sections needing optimizations in assembly.

Is there a compiler that you feel produces better code and is still well supported?

If not, I like the idea of vbcc because it's better integrated with the platform, still developed, open source and the developers are a known quantity.

[matthey]

Originally Posted by idrougge

</end matthey off-topic rant>

It is good to know who the bad players are (not always clear) who will try to sabotage or road block your efforts before you begin. Yes, I would likely get censored on certain other web sites for saying something similar. Hmm, actually I already did.

Originally Posted by Heiroglyph

My plan is to start by patching in anyway.

I assumed I'd start one entry point at a time.
Write a work-alike (as much as is feasible)
Run a test app that makes some test calls, patches in my replacement, then makes the same test calls again and compare the result.

That will pretty quickly tell me where there is internal data and I can potentially put off the chipset functions until later.

Sounds reasonable. Some of the internal data accesses are not so hidden. Most undocumented LVO functions are known and setpatches can be watched even if a pain. I don't think the newer AmigaOS libraries will be too much of a problem.

Originally Posted by Heiroglyph

The chipset parts are mostly available in Aros from what I understand though, they're just slow. They might want some of this code if we can improve 68k performance. It is the only platform I'm targeting with this, so it will have to be reasonably fast or else be rewritten.

It remains to be seen how well you could do without assembler. Optimizing with 68k assembler probably could do better than the AmigaOS graphics.library which is probably some C and 68000 optimized code. Compatibility is the hard part. I haven't done much Amiga custom chip programming so I can't help you much there.

Originally Posted by Heiroglyph

I want to stick with C because ASM isn't clear enough for most people to contribute or even just read.

68k assembler is easy to read and compact but it is not as structured or forgiving as C. It took me longer to learn C than 68k assembler but then there is more to know with C and some of it is hidden.

Originally Posted by Heiroglyph

I'm confident that it will be fast enough for the most part with only certain sections needing optimizations in assembly.

Is there a compiler that you feel produces better code and is still well supported?

If not, I like the idea of vbcc because it's better integrated with the platform, still developed, open source and the developers are a known quantity.

GCC 2.95.3 gives better integer code quality on the 68k usually. It is not compatible with modern GCC, doesn't fully support C99 and the FPU support is basic. I would use vbcc also. If you stick with C99, then it should be compatible with GCC and other compilers. There are likely to be some improvements in code quality even if the 68k backend doesn't get much attention (what I expect but I hope I'm wrong). The active support is great and more important than code quality at this point. Vbcc could give worse code quality. It is right there with SAS/C and GCC 3.x depending on the code. It does best with 32 bit integer datatypes while doing a poor job of trying to promote most smaller datatypes to 32 bits. Promoting integers is good for most modern processors, including the 68060 sometimes, but is sometimes slower and generates larger code on the 68k (the 68020/68030/68060 are slowed by fetching larger instructions). The later ColdFire received the instructions needed to efficiently promote everything but even then dropping .w and .b sizes reduced the performance compared to the 68060. The ColdFire v5 is only faster than the 68060 because it is clocked higher.

[JimDrew]

If you need assembly optimizations, count me in.

[Heiroglyph]

Originally Posted by matthey

It is good to know who the bad players are (not always clear) who will try to sabotage or road block your efforts before you begin. Yes, I would likely get censored on certain other web sites for saying something similar. Hmm, actually I already did.



Sounds reasonable. Some of the internal data accesses are not so hidden. Most undocumented LVO functions are known and setpatches can be watched even if a pain. I don't think the newer AmigaOS libraries will be too much of a problem.



It remains to be seen how well you could do without assembler. Optimizing with 68k assembler probably could do better than the AmigaOS graphics.library which is probably some C and 68000 optimized code. Compatibility is the hard part. I haven't done much Amiga custom chip programming so I can't help you much there.



68k assembler is easy to read and compact but it is not as structured or forgiving as C. It took me longer to learn C than 68k assembler but then there is more to know with C and some of it is hidden.



GCC 2.95.3 gives better integer code quality on the 68k usually. It is not compatible with modern GCC, doesn't fully support C99 and the FPU support is basic. I would use vbcc also. If you stick with C99, then it should be compatible with GCC and other compilers. There are likely to be some improvements in code quality even if the 68k backend doesn't get much attention (what I expect but I hope I'm wrong). The active support is great and more important than code quality at this point. Vbcc could give worse code quality. It is right there with SAS/C and GCC 3.x depending on the code. It does best with 32 bit integer datatypes while doing a poor job of trying to promote most smaller datatypes to 32 bits. Promoting integers is good for most modern processors, including the 68060 sometimes, but is sometimes slower and generates larger code on the 68k (the 68020/68030/68060 are slowed by fetching larger instructions). The later ColdFire received the instructions needed to efficiently promote everything but even then dropping .w and .b sizes reduced the performance compared to the 68060. The ColdFire v5 is only faster than the 68060 because it is clocked higher.

The most important thing is stability and functionality, so I think making it work at any speed using C is the right way to go.

Any of those compilers will produce acceptable code for the 99% cases, but none of them will fix that 1% that needs to be as fast as possible.

Thanks for the input.

[Heiroglyph]

Originally Posted by JimDrew

If you need assembly optimizations, count me in.

Consider yourself counted!

[wawa]

ok, but asm should be used for few critical parts i assume. afair compiling storm mesa anew we even have removed the genuine asm inlays and it didnt have much impact on speed.. strange that..

now to compiler choice, i have my doubts if clang/llvm wiil have noticeable impact on aros instead of plain gcc. noticeable i mean beyond 20-50%. for 68k 2.9.x was the best, then >3.4.x was evil according to bernd and 4.5.x 5.x started to be looking good again, if not ideal. aros is using 4.6.4 now by default. i have noticed that binaries compiled with 6.x are a bit smaller if they are bigger to start with. optimization for size has improved. but the difference is minimal. what about vbcc, do you really think a work of two people can be compard to a whole team involved with gcc. do we have a proof for this? or do we simply count on a pure possibility, that it might be improved, because we know the guys and instead have so little influence on gcc team?

[Locutus]

Originally Posted by Heiroglyph

The most important thing is stability and functionality, so I think making it work at any speed using C is the right way to go.

Any of those compilers will produce acceptable code for the 99% cases, but none of them will fix that 1% that needs to be as fast as possible.

Thanks for the input.

I definitely think that language/compiler choice should take maintainability and the prospect of bitrot into consideration.

Being stuck with a codebase in assembly for someone else to pickup in 5 years time is essentially lost code in practice.

The same with using compilers that are not supported anymore.

Staying alive is a more vital prospect then a bit of speed :-)

[grond]

Regarding private data of libraries, I can imagine that graphics and intuition access each other's data quite a lot. I'm not sure the separation into graphics and intuition makes much sense at all. All programs that need intuition stuff will also at least indirectly open graphics. The few cases where exec, dos, graphics and intuition aren't all open at the same time made a difference when 256kb of RAM were a lot. This being said I wonder whether a more monolithic view of the two libraries could be adopted for this project.

[Heiroglyph]

Originally Posted by Locutus

I definitely think that language/compiler choice should take maintainability and the prospect of bitrot into consideration.

Being stuck with a codebase in assembly for someone else to pickup in 5 years time is essentially lost code in practice.

The same with using compilers that are not supported anymore.

Staying alive is a more vital prospect then a bit of speed :-)

I mentioned in another thread that any assembly needs to have a functionally identical C implementation with a #define so that we can verify both.

[Heiroglyph]

Originally Posted by grond

Regarding private data of libraries, I can imagine that graphics and intuition access each other's data quite a lot. I'm not sure the separation into graphics and intuition makes much sense at all. All programs that need intuition stuff will also at least indirectly open graphics. The few cases where exec, dos, graphics and intuition aren't all open at the same time made a difference when 256kb of RAM were a lot. This being said I wonder whether a more monolithic view of the two libraries could be adopted for this project.

The way existing software is written, it would be a hack to try to combine them.

Assuming they are that tightly coupled, they might have to be a matched pair. It's too early to say.

P96 and Cgx managed without it, so we know it's possible to at least patch in.

[Devlin]

I wish I had hardware/talent to help with this, because I think it's a great idea to make all the copyright bullshit going on with P96/CGX obsolete with all new drivers.

If there's an active bounty for this, I will try and contribute to it when I next have coin.

[Heiroglyph]

My contracts prevent me from profiting from it, but maybe someone else who is already familiar with the Aros codebase might.

I don't care to do it, I just think we need it and nobody seemed to have tried yet.