Challenge: AmigaOS command to start executables.

[Photon]

This is the second step in my vision, if you don't see it please bare with me, if you do, try to support as free time allows.

Please see this thread, which was met with a mix of responses.

Also please ignore that thread and all of AmigaOS history. Please accept for the moment that it's necessary now to create a very decent such command.

It's not necessary for it to replace commands like run or execute.

Spec: it should read a file with the e flag set, parse relevant hunks, use ROM routines to alloc and reloc them, and execute the code hunk, supporting Amiga hardware with ROM 1.3-3.1 as if the user typed the name of the executable file in CLI from boot with no S-S (not Shell or replacements at this time).

Ideas are appreciated in any language, and the resulting binary will be a command (executable file) written in Assembly, and an open source, complying with CC0 1.0 or other suitable license fit for free distribution.

Whatever you think of this vision, I personally offer it as a challenge and ask that if you don't see the reason, don't question it or discourage contributions with 'why' questions. Everyone already knows Amigas can execute files, you have missed the point.

[meynaf]

Let's say i have a program that just calls SystemTagList from dos.library. It will really do as if the user typed the name of the executable in cli (provided you tell me where to fetch the file name).
If we put aside the fact it's v36+, would it do what you want or not ? If not, what would be wrong then ?

[Thomas Richter]

Even under 1.3, the "Execute()" call of the dos.library exists and can do that. It should just have a ZERO input file handle, and the command name as first argument. The only drawback is that it needs the "Run" command to be present (in 1.2) or at least resident (with the 1.3 SetPatch active).

Side remark: It is quite funny that Execute() depends on Run in 1.3, but Run depends on System() (which is a cousin of Execute()) from 2.0 onwards.

[Bruce Abbott]

Originally Posted by Photon

It's not necessary for it to replace commands like run or execute.

Spec: it should read a file with the e flag set, parse relevant hunks, use ROM routines to alloc and reloc them, and execute the code hunk, supporting Amiga hardware with ROM 1.3-3.1 as if the user typed the name of the executable file in CLI from boot with no S-S (not Shell or replacements at this time).

You say AmigaOS command, then talk about executing code 'as if' the user had typed its name from the CLI. What environment will this 'command' be running in?

[jotd]

can you use LoadSeg?
What about overlayed files?

[Olaf Barthel]

Originally Posted by Photon

This is the second step in my vision, if you don't see it please bare with me, if you do, try to support as free time allows.

Please see this thread, which was met with a mix of responses.

Also please ignore that thread and all of AmigaOS history. Please accept for the moment that it's necessary now to create a very decent such command.

It's not necessary for it to replace commands like run or execute.

Spec: it should read a file with the e flag set, parse relevant hunks, use ROM routines to alloc and reloc them, and execute the code hunk, supporting Amiga hardware with ROM 1.3-3.1 as if the user typed the name of the executable file in CLI from boot with no S-S (not Shell or replacements at this time).

Ideas are appreciated in any language, and the resulting binary will be a command (executable file) written in Assembly, and an open source, complying with CC0 1.0 or other suitable license fit for free distribution.

Whatever you think of this vision, I personally offer it as a challenge and ask that if you don't see the reason, don't question it or discourage contributions with 'why' questions. Everyone already knows Amigas can execute files, you have missed the point.

Challenge accepted

According to your requirements, every AmigaDOS binary file should be loadable and executable, regardless of which operating system it was originally intended to be loaded and run on.

This requirement includes the need to provide for a runtime environment, because just loading and relocating the code, then jumping to the first executable location in the file is not sufficient. The CLI commands of Kickstart 1.x require a specific, at the time (1985-1989) undocumented set of data structures and register configurations which, so it happens, also involved undocumented parts of the dos.library. Never mind them being undocumented at the time, those part of dos.library were deprecated and largely removed in Kickstart 2.x, thus rendering these old CLI commands prone to crash instead of exiting safely.

So, in order to fulfill your requirements, there would have to be not just a reimplementation of the load file loader code (that's the easy part, but it's a bit tedious, to be honest) but also of the underpinnings of the environment in which the program would have to run according to the rules of dos.library in Kickstart 1.x. This is doable, too, but quite tedious, to be honest.

The good news is that the Kickstart 1.x dos.library is, according to what I could find out, only about 17 KBytes in size and already contains a complete, working binary file loader (that supports overlays) with a documented API which could be tweaked to offer the functionality present in Kickstart 2.x but absent in Kickstart 1.x (you don't need all of what's in those 17 KBytes). Making this work according to your goals and requirements is doable and would not include any extra work to implement everything from scratch.

[Thomas Richter]

Originally Posted by Olaf Barthel

This requirement includes the need to provide for a runtime environment, because just loading and relocating the code, then jumping to the first executable location in the file is not sufficient. The CLI commands of Kickstart 1.x require a specific, at the time (1985-1989) undocumented set of data structures and register configurations which, so it happens, also involved undocumented parts of the dos.library. Never mind them being undocumented at the time, those part of dos.library were deprecated and largely removed in Kickstart 2.x, thus rendering these old CLI commands prone to crash instead of exiting safely.

Well, I'm not sure whether making 1.x CLI commands working under 2.x was really part of the challenge because that's harder, at least for *some* of them. The 1.x "Ed" for example will not work as it overloads some of the entries of its GlobVec making the global one from the fake blib unusable.


The other way around is course even harder: Making 2.x commands workable under 1.x as this requires more or less the entire system.

[Dunny]

I'm not sure preventing them from crashing due to incompatible OS calls etc is part of the challenge? I'm pretty sure the spec was "just start them running, what happens after that doesn't matter"

Edit:

Originally Posted by Photon

This code must load the exe into memory correctly under OS1.3-3.1 and then call the first instruction.

Whatever happens after that is of no concern whatever.

To be specific.

[Photon]

An executable file on AmigaOS is one that has its executable flag set and consists of hunks like other well-formed Amiga files.

The goal is to not require files from any WB disk or any existing WB installation, to run such a program that is well-formed.

The challenge is met if an executable file formed by non-special hunks like code, data, and BSS, the code snippet allocates, relocates, and executes it as AmigaOS 1.3 and possibly up, would. If the machine running it has Execute in ROM, the goal is achieved. For the others we would have to do hunk parsing and LoadSeg, using what's left in ROM.

I welcome your input let's quote everyone and clarify.

Originally Posted by meynaf

Let's say i have a program that just calls SystemTagList from dos.library. It will really do as if the user typed the name of the executable in cli (provided you tell me where to fetch the file name).
If we put aside the fact it's v36+, would it do what you want or not ? If not, what would be wrong then ?

A pointer to an ASCIIZ command string as executed from a line in startup-sequence on all Kickstarts would be the ideal, if it were possible on OS versions without Execute in ROM.

Originally Posted by Thomas Richter

Even under 1.3, the "Execute()" call of the dos.library exists and can do that. It should just have a ZERO input file handle, and the command name as first argument. The only drawback is that it needs the "Run" command to be present (in 1.2) or at least resident (with the 1.3 SetPatch active).

Side remark: It is quite funny that Execute() depends on Run in 1.3, but Run depends on System() (which is a cousin of Execute()) from 2.0 onwards.

1.3 is the minimum requirement, so the 1.2 drawback is not a drawback for this challenge. Maybe a combination of Execute in 1.3 and SystemTagList (whatever that is) in 2.0+ is the simplest solution?

Originally Posted by Bruce Abbott

You say AmigaOS command, then talk about executing code 'as if' the user had typed its name from the CLI. What environment will this 'command' be running in?

CLI, under Kickstart 1.3-3.1. You may substitute "as if typed" with "as if a line in startup-sequence". Same thing, as long as I can put it in the code snippet and point to it and go.

Originally Posted by jotd

can you use LoadSeg?
What about overlayed files?

See above. Executable files can have different hunk types to these 3, but the 3 are enough to satisfy the challenge. Quirks such as BSS being automatically cleared or not have no bearing.

Originally Posted by Olaf Barthel

Challenge accepted

According to your requirements, every AmigaDOS binary file should be loadable and executable, regardless of which operating system it was originally intended to be loaded and run on.

Awesome, looking forward to code, gists, ideas and I wish this to be anchored in the community, with full credit if desired.

This project is important and also as far from "me" as you could possibly get.

But as answer: no, this is not the goal. The solution should parse the hunks and start execution, if the file is well-formed, no questions asked. But the actual code executed would itself have to be compatible with whatever versions it wants to support.

I.e. if the command allocates, relocates, and executes the file correctly, it is the same as for every such executable file for AmigaOS. The hunk contents (code or data structures) may have their own minimum spec and dependencies that are not at all present, and fail spectacularly or gracefully, and the goal is still met.

I.e. just as a program made for AGA and 3.1 could be run by this command on OCS and 3.1, so it could with the Execute command, and neither command should be blamed for being run on a non-AGA machine.

Originally Posted by Olaf Barthel

The good news is that the Kickstart 1.x dos.library is, according to what I could find out, only about 17 KBytes in size and already contains a complete, working binary file loader (that supports overlays) with a documented API which could be tweaked to offer the functionality present in Kickstart 2.x but absent in Kickstart 1.x (you don't need all of what's in those 17 KBytes). Making this work according to your goals and requirements is doable and would not include any extra work to implement everything from scratch.

The size of the resulting command is not a dealbreaker, but the goal is not met if large portions of code is copied outright from a Kickstart ROM or a WB disk.

If the code consists of 68k logic and usage of libraries and structures present in Kickstart 1.3-3.1, it's normal, system-friendly code that guidelines desire from any Amiga application.

[copse]

I'm reminded of amitools here. Link to related code on Github, to be clear not saying it fits the requirements, just technically similar although it has a restrictive license.

[meynaf]

Originally Posted by Photon

A pointer to an ASCIIZ command string as executed from a line in startup-sequence on all Kickstarts would be the ideal, if it were possible on OS versions without Execute in ROM.

Yes but the question was : where does the file name come from ?
I mean, it could be specified as command line :

runit someprogram

or put inside another file :

echo "someprogram" >xx
runit

or even a constant in the code :

 dc.b "someprogram",0

Furthermore, things will get crazy if you want to do it exactly like in the CLI. As this implies running programs that are potentially resident, and use the system's path to find commands, in addition to the ability to give command-line parameters to the program we run (which simple loadseg + jsr won't do).

Originally Posted by Photon

The size of the resulting command is not a dealbreaker, but the goal is not met if large portions of code is copied outright from a Kickstart ROM or a WB disk.

That may be problematic, as disassembling OS routines to back-port missing calls to previous OS version seems to be the easiest way.

Alternatively, you could just use the system() call from some C library.

[Olaf Barthel]

Originally Posted by Photon

But as answer: no, this is not the goal. The solution should parse the hunks and start execution, if the file is well-formed, no questions asked. But the actual code executed would itself have to be compatible with whatever versions it wants to support.

Hey, you're breaking my heart. I was ready to spring (or maybe stumble) into action and to deliver the "Manhattan Project" of obscure AmigaDOS intrinsics, including support for overlays and the arcane, nay, forbidden depths of the Kickstart 1.x dos.library.

The job you are asking for has been basically done and dusted in 1988 already, at least the really hard part.

It can be found on the Amiga Developer CD 2.1 under "Reference/DevCon/Washington_1988/DevCon88.3/AutoBoot", the contents of which I am attaching below in a zip archive file.

The relevant portion of this is the "loadseg.asm" file, which is designed to go into a mass storage disk controller (MFM/RLL/SCSI/IDE) boot ROM for the purpose of loading a file system from the Rigid Disk Block used by the autoboot feature introduced in Kickstart 1.3. If memory serves, the "loadseg.asm" code is what went into the A590 boot ROM and subsequently became the dos.library/LoadSeg implementation of dos.library V36-V40. This is the good stuff (but not necessarily the best stuff).

If I remember correctly, "loadseg.asm" is preset to read the executable from a file on disk (it makes use of dos.library/Read), but this read function can be changed to retrieve this data from anywhere, e.g. from the file system block chain of the Rigid Disk Block.

This code was released by Commodore for the purpose of developers to adopt and make use of it, because that whole loading, processing and relocating business is not for the faint of heart. The only thing it's missing is a cache flush operation on the 68020 and beyond, but I reckon that could be easily added with a minimum of fuss. Speaking of fuss, the support for the modern Kickstart 2.x hunk types would have to be added as well. But now I'm so demotivated I would gladly leave the challenge of making all of this work to much keener minds than mine.

[Olaf Barthel]

Originally Posted by meynaf

That may be problematic, as disassembling OS routines to back-port missing calls to previous OS version seems to be the easiest way.

Alternatively, you could just use the system() call from some C library.

That 'C' library likely calls either dos.library/Execute or dos.library/SystemTagList anyway. Not much to be gained here

[Olaf Barthel]

Originally Posted by copse

I'm reminded of amitools here. Link to related code on Github, to be clear not saying it fits the requirements, just technically similar although it has a restrictive license.

And it's written in Python, which would add the additional burden of having to run it on an Amiga Python interpreter, which then in turn would emulate the 68k CPU code, too. Would have to be a Python 3 interpreter, too (not sure if this has been ported to the Amiga yet).

[Thomas Richter]

Originally Posted by Photon

An executable file on AmigaOS is one that has its executable flag set and consists of hunks like other well-formed Amiga files.

And,silly me, what is it that keeps you from using LoadSeg() and/or Execute()? The latter does all you need: It is a ROM-based function (1.2 excluded, 1.3 only wth SetPatch running), it loads a binary from disk, and executes it.


So please tell me - what is it that this is not the right soluion.

Originally Posted by Photon

The goal is to not require files from any WB disk or any existing WB installation, to run such a program that is well-formed.

Nothing. You need the dos.library.

Originally Posted by Photon

The challenge is met if an executable file formed by non-special hunks like code, data, and BSS, the code snippet allocates, relocates, and executes it as AmigaOS 1.3 and possibly up, would.

LoadSeg()? Execute()? I mean, which kind of a challenge is this?

Originally Posted by Photon

If the machine running it has Execute in ROM, the goal is achieved.

The dos.library is part of the ROM - so we are done? Note that the Execute() call of the dos.library has *absolutely nothing* to do with the Execute command in C: The execute command does not actually anything, it is only a shell script preprocessor.

Originally Posted by Photon

1.3 is the minimum requirement, so the 1.2 drawback is not a drawback for this challenge. Maybe a combination of Execute in 1.3 and SystemTagList (whatever that is) in 2.0+ is the simplest solution?

SystemTagList() does not exist in 1.3 and below, but it is just an extended form of Execute(). If all you need is loading and running an executable, then Execute() does this as well.

Originally Posted by Photon

See above. Executable files can have different hunk types to these 3, but the 3 are enough to satisfy the challenge. Quirks such as BSS being automatically cleared or not have no bearing.

Yes, and Execute() calls LoadSeg(), and LoadSeg() knows them. Which type of a challenge is this?

Originally Posted by Photon

Awesome, looking forward to code, gists, ideas and I wish this to be anchored in the community, with full credit if desired.

Code? Really?

Execute("C:List",0,Output())

is that good enough?

Originally Posted by Photon

This project is important and also as far from "me" as you could possibly get.

It seems utterly trivial as the Os supports all this "out of the box".

Originally Posted by Photon

I.e. just as a program made for AGA and 3.1 could be run by this command on OCS and 3.1, so it could with the Execute command, and neither command should be blamed for being run on a non-AGA machine.

You don't need the Execute command to execute anything. Forget the command, it does not do what you think it does.

Originally Posted by Photon

If the code consists of 68k logic and usage of libraries and structures present in Kickstart 1.3-3.1, it's normal, system-friendly code that guidelines desire from any Amiga application.

Your desired code is a one-liner (plus opening the dos.library), and I provided the solution above, so can we close this topic?

[a/b]

Does that solution fully clear all BSS hunks, so that the exe behaves the same on all KS versions?

[Thomas Richter]

About which compatibility problem do we talk here? Is there a 1.3 program that has such a hunk where this causes trouble? Do you plan to run >2.x programs on 1.3? This won't work anyhow. Do you plan a bugfix for 1.3? That was done with 2.04 already.

IOWs, what kind of problem shall be solved here? Bugfixing 1.3? Done, it's an existing product. Loading 2.x programs? Won't work for reasons completely outside the loader.

[AestheticDebris]

Originally Posted by a/b

Does that solution fully clear all BSS hunks, so that the exe behaves the same on all KS versions?

Behaving the same is explicitly not a requirement, but the problem is too vague to really say whether that is an issue or not.

[alkis]

I am not 100% sure that I understand what you need, but here is a "system" command that I used to avoid having anything like c:run available (as stated in http://amigadev.elowar.com/read/ADCD.../node015E.html for 1.3)

It's a wrapper for "SYSTEM0 execute command from disk. From Sozobon C distribution. Copyright © 1988 by Ralph Babel."

[Thomas Richter]

This is again something different. This assembler function overloads the currently running process with another segment and executes a loaded segment within the context of the current process. This is what the dos.library function RunCommand() does from 2.04 onwards.

Execute() does all that as well, though also creates a new shell, and performs the loading and execution within that new shell, i.e. a new process. Execute() thus has the full syntax of the shell available, including input/output redirection or alias substitution.