Exec style Devices

[bloodline]

Hi all,

Believe it or not, but in all my years of Amiga coding, I’ve managed to side step using devices to any great degree. I’m ashamed to admit, I tended to use OpenDevice() to just “lock” a piece of hardware and then hit the chipset regs directly.

I hope to be able to resume work on my much neglected Amiga projects in the not too distant future, and one thing that I will probably need to do is write a Exec style device.

Now I need to make sure I’ve got a few things clear in my head (and all documentation covering Devices is rubbish), I’ll write here my understanding and some of the AmigaOS experts can correct me where I’m wrong!

From a user point of view the only interaction one has with a device is via IORequestStructures and, IORequest structures are just extended Exec style messages.

So as a theoretical example, to read some data from a device one would create an iorequest structure then OpenDevice() the device you are interested in, specifying the unit number you want.

The iorequest is now a shared block of memory between your program and the device.

Now we call the BeginIO(iorequest) Exec function and wait for a reply to whatever message port you set in your iorequest.

As I understand it devices have their own iorequest structures which include the standard fields (command/length/error/etc), so when the reply is received I can expect it to contain the requested data or whatever.

As long as I have possession of the iorequest (i.e. it has been replied to), I may alter the fields of the iorequest, so as to issue new commands. Then call BeginIO().

When I’m finished with the device, I just call CloseDevice(iorequest).

Points to note, there is no way to find out what devices are available other than to probe execbase’s deviceList (with appropriate locking), or to read the system disk/Devs: dir. This is no standard way to establish how many units a device may have without trying to open a particular unit and check for success/failure?



From a device developer point of view, a device is essentially just a library with two extra “standard functions” to initiate and abort an i/o operation. A device may spawn a task to handle the request, if it doesn’t spawn its own task, I assume this is only vaild for simple devices and it does its processing in the context of the exec (the calling task?).

When a task calls exec’s BeginIO(iorequest), exec will then call the device’s BeginIO() which then Handles the command.

From the documentation it seems devices may have other functions, but it isn’t clear how the calling task is supped to obtain the devicebase... unless it is permitted to use the address placed into the iorequest structure during the OpenDevice() call. Like most structures in AmigaOS it isn’t clear just how opaque it’s supposed to be.

Ok, I think that all makes sense!

It seems to me that given how robust it is, the intention was that the device interface was meant to be the normal “high level” AmigaOS interface, with libraries, signals, and messages all supposed to be the hidden lower levels. I guess it just didn’t work out that way

[Thomas Richter]

Originally Posted by bloodline

The iorequest is now a shared block of memory between your program and the device.

In AmigaOs, everything you have is shared memory. From a conceptual point of view, it is not only "struct IORequest" (or typically struct IOStdReq). What is also shared is the memory buffer you pass over to the device to read into, or to write out, i.e. IOStdReq->io_Data.

Originally Posted by bloodline

Now we call the BeginIO(iorequest) Exec function and wait for a reply to whatever message port you set in your iorequest.

Actually, in almost all cases, it is not BeginIO(), but it is DoIO() for synchronous calls (i.e. the call returns once the IO is done) or SendIO() for asynchronous calls (i.e. your code continues to run, and you can call CheckIO() to test whether the IO is done, and WaitIO() to wait for its completion).

Originally Posted by bloodline

As I understand it devices have their own iorequest structures which include the standard fields (command/length/error/etc), so when the reply is received I can expect it to contain the requested data or whatever.

The structure is always an extension of struct IORequest (i.e. some additional fields at the end), and how it is extended is not only device dependent, but also command extended, i.e. io->io_Command. In many cases, it is a struct IOStdReq which contains a buffer pointer, a buffer size and an offset.

Originally Posted by bloodline

As long as I have possession of the iorequest (i.e. it has been replied to), I may alter the fields of the iorequest, so as to issue new commands. Then call BeginIO().

Then call SendIO() or DoIO(). DoIO() is simpler, SendIO() is more flexible. I believe the only device that requires BeginIO() is the audio.device.

Originally Posted by bloodline

When I’m finished with the device, I just call CloseDevice(iorequest).

Yes, though make sure that all IORequests you send out have been returned, and wait for any non-completed requests with WaitIO().

Originally Posted by bloodline

Points to note, there is no way to find out what devices are available other than to probe execbase’s deviceList (with appropriate locking), or to read the system disk/Devs: dir. This is no standard way to establish how many units a device may have without trying to open a particular unit and check for success/failure?

No, there isn't. The exec device list only contains devices that are currently loaded in memory; those that are not loaded, but waiting in DEVS: do not show up there. The only way how to find out whether a device can be opened is to open it. That is a shortcoming of exec, missing some sort of Gestalt manager. There was NSD which was a misguided attempt to partially solve the problem - but it made the problem worse. You still need to open a device first, and you do not know which side effects this has.


In short: Allow the user to pick a device by name. Do not rely on NSD, it doesn't solve that problem.

Originally Posted by bloodline

From a device developer point of view, a device is essentially just a library with two extra “standard functions” to initiate and abort an i/o operation. A device may spawn a task to handle the request, if it doesn’t spawn its own task, I assume this is only vaild for simple devices and it does its processing in the context of the exec (the calling task?).

If a device does not spawn its own task, it clearly cannot perform an IORequest asynchronously, i.e. any SendIO() will perform the IORequest immediately and reply the request directly. Yes, this happens in the context of the caller.


The ramdrive device is such a device, to give an example.

Originally Posted by bloodline

When a task calls exec’s BeginIO(iorequest), exec will then call the device’s BeginIO() which then Handles the command.

No. BeginIO() is the name of the entry vector of the device jump table, and not a function of exec. The corresponding exec function is SendIO(), which clears the flags, and calls BeginIO(). Or DoIO(), which sets flags, calls BeginIO() and then WaitIO() to await the completion of the request.

Originally Posted by bloodline

From the documentation it seems devices may have other functions, but it isn’t clear how the calling task is supped to obtain the devicebase...

OpenDevice(), of course. You still need an IoRequest. You find the device base address in the IORequest structure then.

Originally Posted by bloodline

It seems to me that given how robust it is, the intention was that the device interface was meant to be the normal “high level” AmigaOS interface, with libraries, signals, and messages all supposed to be the hidden lower levels. I guess it just didn’t work out that way

The Amiga device interface is supposed to provide asynchronous command interfaces, typical for IO operations. Libraries provide synchronous helper functions.

[bloodline]

Many thanks for you reply here, I do really appreciate you taking the time to clean up my knowledge! It does seem that, broadly, I understand how this works.

Originally Posted by Thomas Richter

In AmigaOs, everything you have is shared memory. From a conceptual point of view, it is not only "struct IORequest" (or typically struct IOStdReq). What is also shared is the memory buffer you pass over to the device to read into, or to write out, i.e. IOStdReq->io_Data.

Indeed! But only my task and the device should know about it

Actually, in almost all cases, it is not BeginIO(), but it is DoIO() for synchronous calls (i.e. the call returns once the IO is done) or SendIO() for asynchronous calls (i.e. your code continues to run, and you can call CheckIO() to test whether the IO is done, and WaitIO() to wait for its completion).

Ok, the RKM I have (1.3) is confusing as it implies all IORequest is dispatched via BeginIO(), which will then decide to Do or Send. I'm getting confused as BeginIO() is in amiga.lib not Exec!

The structure is always an extension of struct IORequest (i.e. some additional fields at the end), and how it is extended is not only device dependent, but also command extended, i.e. io->io_Command. In many cases, it is a struct IOStdReq which contains a buffer pointer, a buffer size and an offset.

Ok that's good, I like the way AmigaOS structures "inherit" from base structures

No, there isn't. The exec device list only contains devices that are currently loaded in memory; those that are not loaded, but waiting in DEVS: do not show up there. The only way how to find out whether a device can be opened is to open it. That is a shortcoming of exec, missing some sort of Gestalt manager. There was NSD which was a misguided attempt to partially solve the problem - but it made the problem worse. You still need to open a device first, and you do not know which side effects this has.

In short: Allow the user to pick a device by name. Do not rely on NSD, it doesn't solve that problem.

I thought I had missed something. It seems odd that such a thing wasn't thought of at the beginning.

If a device does not spawn its own task, it clearly cannot perform an IORequest asynchronously, i.e. any SendIO() will perform the IORequest immediately and reply the request directly. Yes, this happens in the context of the caller.


The ramdrive device is such a device, to give an example.

Are there any tips about the nature of a device’s attendant task? I would be tempted to give it a relatively high priority, but I have no idea if that’s a good idea.

No. BeginIO() is the name of the entry vector of the device jump table, and not a function of exec. The corresponding exec function is SendIO(), which clears the flags, and calls BeginIO(). Or DoIO(), which sets flags, calls BeginIO() and then WaitIO() to await the completion of the request.

Yes, I was confused by the amiga.lib function.

OpenDevice(), of course. You still need an IoRequest. You find the device base address in the IORequest structure then.

I’ve found the page in the RKM which explains this now, I’m so used to operating system structures being opaque now I forget how AmigaOS lets you access almost anything

I'm reading Wendal's guide right now... not sure it really helps me : http://wandel.ca/homepage/execdis/devices_doc.txt

[Wepl]

Maybe it should be noted that BeginIO/DoIO will clear all io_Flags and so make it impossible to deliver any flags to the device. If this is needed (e.g. IOTDF_INDEXSYNC/WORDSYNC for trackdisk.device) you need to call DEV_BEGINIO manually.

[Thomas Richter]

Originally Posted by Wepl

Maybe it should be noted that BeginIO/DoIO will clear all io_Flags and so make it impossible to deliver any flags to the device. If this is needed (e.g. IOTDF_INDEXSYNC/WORDSYNC for trackdisk.device) you need to call DEV_BEGINIO manually.

BeginIO() does not clear any flags. DoIO() does, and SendIO() does.

[bloodline]

Ok, so I've been using Scout to examine real devices, I notice that the input.device attendant task has the top two signals set (30 and 31), does each of these signals relate to keyboard and the mouse interrupt handlers? I've always been reluctant to set a signal during an interrupt... as IIRC signal causes an immediate reschedule.

[a/b]

Yup, what Thomas said. I've been doing some experiments with trackdisk.device and raw MFM data recently and ran into that particular "problem". DoIO() sets them to IOF_QUICK, SendIO() clears them. If you are linking with amiga.lib you can simply call BeginIO()/WaitIO() and be done with it, otherwise you have to go down to device's BeginIO() and handle eventual specifics.

[Thomas Richter]

Originally Posted by bloodline

Ok, so I've been using Scout to examine real devices, I notice that the input.device attendant task has the top two signals set (30 and 31), does each of these signals relate to keyboard and the mouse interrupt handlers?

It does not matter what these relate to. Signal allocation is up to the input.device.

Originally Posted by bloodline

I've always been reluctant to set a signal during an interrupt... as IIRC signal causes an immediate reschedule.

Signal() can be called within an interrupt without any problem. Just to give one example: The graphics.library vertical blank interrupt calls Signal() to wake up tasks waiting in WaitTOF(). PutMsg() is also interrupt-callable. It would be very bad for any IO device implementation if it wouldn't.

[bloodline]

Originally Posted by Thomas Richter

It does not matter what these relate to. Signal allocation is up to the input.device.

This is perhaps more idle curiosity on my part, if anyone would know the internals of the input device it would be you! But I digress, if I were to write my own device it would need to be able to wake a task from an interrupt, Signal() would be a great way to do that.

Signal() can be called within an interrupt without any problem. Just to give one example: The graphics.library vertical blank interrupt calls Signal() to wake up tasks waiting in WaitTOF(). PutMsg() is also interrupt-callable. It would be very bad for any IO device implementation if it wouldn't.

This is good! My only worry was regarding the priority of interrupts, I have always assumed (perhaps wrongly) that exec’s scheduler ran in a VBL interrupt (with a reschedule occurring every 4 interrupts), which is relatively low priority, if a reschedule were in progress and it was interrupted by a higher priority interrupt, which then rescheduled, wouldn’t the task lists become corrupted? Or are all reschedules handled in the VBL interrupt, with the finest granularity of rescheduling being one VBL?

I hope that makes sense?

[Thomas Richter]

Originally Posted by bloodline

This is perhaps more idle curiosity on my part, if anyone would know the internals of the input device it would be you!

I would, and my answer is that it does not matter what these signals are used for. The input.device task just uses AllocSignal() to get them, and uses them for whatever it seems fit.

Originally Posted by bloodline

This is good! My only worry was regarding the priority of interrupts, I have always assumed (perhaps wrongly) that exec’s scheduler ran in a VBL interrupt (with a reschedule occurring every 4 interrupts), which is relatively low priority, if a reschedule were in progress and it was interrupted by a higher priority interrupt, which then rescheduled, wouldn’t the task lists become corrupted?

No. Exec is smart enough to distinguish between calls from interrupts and calls from user space. No worries. Actually, along with Signal(), GetMsg(), PutMsg() and ReplyMsg() are also fine. As said, it would be very bad if exec couldn't handle this case. If you want to off-load heavier work from the interrupt code, another strategy would be to cause a soft-interrupt with Cause() and defer activity to the soft interrupt. Cause() is of course interrupt-callable.

[bloodline]

Originally Posted by Thomas Richter

I would, and my answer is that it does not matter what these signals are used for. The input.device task just uses AllocSignal() to get them, and uses them for whatever it seems fit.

Again, I was just curious how the system devices approach the problem, not that I have any interest specifically in the input.device task.

Though I've always been curious about the "signals reserved for system usage"...

No. Exec is smart enough to distinguish between calls from interrupts and calls from user space. No worries. Actually, along with Signal(), GetMsg(), PutMsg() and ReplyMsg() are also fine. As said, it would be very bad if exec couldn't handle this case. If you want to off-load heavier work from the interrupt code, another strategy would be to cause a soft-interrupt with Cause() and defer activity to the soft interrupt. Cause() is of course interrupt-callable.

Yes, I'm over thinking this with too little knowledge. Really, I just need to RTFM.

I've never really understood about software interrupts, as I've never had cause (excuse the pun) to use them. I'll devote a bit of time tonight to read about them.

[Thomas Richter]

The signals 0-15 are reserved for system usage. Signals 16-31 are just what AllocSignal() returns, and this is what the input device uses for its ports.

[bloodline]

I accidentally stumbled across DOS's concept of devices. I wondered if I could write a Device which would expose an interface that could be accessed via a command line, like SER: for example...

I'm totally lost, DOS is another world...

[hooverphonique]

Originally Posted by bloodline

I accidentally stumbled across DOS's concept of devices. I wondered if I could write a Device which would expose an interface that could be accessed via a command line, like SER: for example...

Yes, conceptually you can interface almost anything using amiga OS devices and handlers, but figuring out how to write them is not an easy task, unfortunately.

[Thomas Richter]

Originally Posted by bloodline

I accidentally stumbled across DOS's concept of devices. I wondered if I could write a Device which would expose an interface that could be accessed via a command line, like SER: for example...

I'm totally lost, DOS is another world...

That is the tripos version of a "device", which is a horse of a different color. Sorry, "colour", as Tripos comes from GB, not US.



So, such a "device" is essentially a HUNK executable that is launched as a standard dos process, and finds in proc->pr_MsgPort "DosPackets" as command packages. These DosPackets encode what to do and on which data to operate.


So the concepts are related, but evolved in parallel and independent of each other. What is an "IORequest" in the exec world is a "DosPacket" in the Tripos world. What "DoIO()" is in the exec world is "DoPkt()" in the Tripos world.


Unfortunately, documentation about Tripos "Devices" aka "Handlers" is weak. The "best" first hand source is the AmigaDos Manual, 3rd Edition, by Bantam Books, which is more or less an evolved version of the Tripos manual.


The best, and most recommended source for this business is Ralph Babel's "Guru Book", and you would typically want to look into this source, and not into the Bantam book.

[bloodline]

Originally Posted by Thomas Richter

That is the tripos version of a "device", which is a horse of a different color. Sorry, "colour", as Tripos comes from GB, not US.

So do I! Though right now I’m embarrassed to admit to being British

So, such a "device" is essentially a HUNK executable that is launched as a standard dos process, and finds in proc->pr_MsgPort "DosPackets" as command packages. These DosPackets encode what to do and on which data to operate.


So the concepts are related, but evolved in parallel and independent of each other. What is an "IORequest" in the exec world is a "DosPacket" in the Tripos world. What "DoIO()" is in the exec world is "DoPkt()" in the Tripos world.


Unfortunately, documentation about Tripos "Devices" aka "Handlers" is weak. The "best" first hand source is the AmigaDos Manual, 3rd Edition, by Bantam Books, which is more or less an evolved version of the Tripos manual.


The best, and most recommended source for this business is Ralph Babel's "Guru Book", and you would typically want to look into this source, and not into the Bantam book.

I found some online resources and did a bit of reading at lunch, DOS actually doesn’t look that bad (at least the BCPL free 2.0 version), I’m taken by what appears to be duplication of effort between Exec and DOS... packets for example seem rather hacked on, rather than integrated with Exec.

[bloodline]

Originally Posted by hooverphonique

Yes, conceptually you can interface almost anything using amiga OS devices and handlers, but figuring out how to write them is not an easy task, unfortunately.

Ok, so I’ve got a much clearer view of DOS now, it seems the key to making things work are indeed handlers!

Say I created a simple piece of hardware (imagine a simple 5v tolerant uC which exposes an SD card and a serial port, connected via the A1200 clock port), I could easily write an exec Device to interface with it using IORequests. The Exec interface is really clean I like it.

Now if I want to expose the two hardware features (the serial port and the SD card interface) via DOS, I would need to write a pair of handlers. One for the serial port would seem to be quite simple as it is a simple in/out device... write a stream of bytes to it, read a stream of bytes from it.

The one for the SD card would be more complex, firstly because the SDCard exposes a filesystem, and secondary the SDCard spec requires that cards use the FAT file system (I’m aware that AOS3.0 supports FAT12 at least), but would expect to write ExFAT support into my handler. Am I correct that the responsibility of dealing with the filesystem is with the handler (that’s the impression I get looking at the packet types), this way the exec device can just deal with the data stream and DOS can just deal with human readable paths.

I’m having difficulty conceptualising is the chain of responsibility here.

If I’m picturing this correctly, DOS is only aware of the handlers, of which it maintains a list, called the DOSList. At some point handlers are added to this list via the DOS function AddEntry (a mount list at boot time? The mount command at run time?). Upon opening (mounting), the handler opens the exec style device which it needs to interface with the hardware, and exposes a DOSpacket interface to DOS. So a handler is essentially a virtual device which acts as a go between DOS and the exec style device?

Now I see that handlers are to be found in the Sys:L dir (I guess a TripOS legacy)and their configuration files in DevsOSDrivers.

It also seems that I could write a single handler for my device, and have two “DOSDriver” configuration files for each of the features provided by my device.

Again sorry if this is time wastingly simple/naïve... I’m just trying to get a handle (excuse the pun) on all of this.

-edit- also it seems some exec devices can accept DOSPackets directly!? i.e. the trackdisk.device doesn’t seem to have a handler between dos and the device.

[phx]

Originally Posted by bloodline

-edit- also it seems some exec devices can accept DOSPackets directly!? i.e. the trackdisk.device doesn’t seem to have a handler between dos and the device.

It does. This would usually be the FastFileSystem handler from the ROM.

[Thomas Richter]

Originally Posted by bloodline

Say I created a simple piece of hardware (imagine a simple 5v tolerant uC which exposes an SD card and a serial port, connected via the A1200 clock port), I could easily write an exec Device to interface with it using IORequests. The Exec interface is really clean I like it.

If this is a "block-access" device, i.e. a device which is comprized of multiple storage units that are individually addressable, then this is the way to go. On top of this exec device, you could mount a dos "device" aka "handler" aka "file system" to organize data on it. This is a two-level design:


exec devices provide block I/O, on top of them sit dos type handlers which provide file access by organizing the blocks.

Originally Posted by bloodline

Now if I want to expose the two hardware features (the serial port and the SD card interface) via DOS, I would need to write a pair of handlers. One for the serial port would seem to be quite simple as it is a simple in/out device... write a stream of bytes to it, read a stream of bytes from it.

Yes, that would be a lookalike of the serial.device. On top of that could sit the "port-handler", the dos.device which handles access to such type of devices. From 3.1.4 on, it is also able to handle devices other than the serial.device.

Originally Posted by bloodline

The one for the SD card would be more complex, firstly because the SDCard exposes a filesystem, and secondary the SDCard spec requires that cards use the FAT file system (I’m aware that AOS3.0 supports FAT12 at least), but would expect to write ExFAT support into my handler. Am I correct that the responsibility of dealing with the filesystem is with the handler (that’s the impression I get looking at the packet types), this way the exec device can just deal with the data stream and DOS can just deal with human readable paths.

You only need an exec type device to address the blocks. The rest is up to the user, namely to handle the correct file system on top of it. CrossDos exists, and can be mounted on such a device, and then would provide a FAT filing system on it.

Originally Posted by bloodline

If I’m picturing this correctly, DOS is only aware of the handlers, of which it maintains a list, called the DOSList.

Yes. That is a BCPL-chained (of course) list of DosNodes, each of which represents a handler, an assign, a file system or a volume within a file system.

Originally Posted by bloodline

At some point handlers are added to this list via the DOS function AddEntry (a mount list at boot time? The mount command at run time?).

That is either user-triggered by the mount command to add handlers, or user-triggered by the assign command to add directories, or file-system triggered to add volumes (this is what a file system does if you insert a volume, and the file system can validate the volume), or system triggered through expansion/AddDosNode at boot time once a bootable extension detects partitions on a harddisk it is responsible for.

Originally Posted by bloodline

Upon opening (mounting), the handler opens the exec style device which it needs to interface with the hardware, and exposes a DOSpacket interface to DOS.

If the dos handler sits on top of an exec device, of course. There are also dos handlers that do not. Such as the queue-handler, or the ram-handler. Both are stand-alone.

Originally Posted by bloodline

So a handler is essentially a virtual device which acts as a go between DOS and the exec style device?

Correct. Tripos is essentially a construction that allows "virtual file systems". Which is quite contrary to the unix way of doing things where the file system is just a subroutine call from the user, and hence executed in the context of the caller. It is one of the few points were Tripos is a bit ahead of Unix.

Originally Posted by bloodline

Now I see that handlers are to be found in the Sys:L dir (I guess a TripOS legacy)and their configuration files in DevsOSDrivers.

L: is the tripos legacy mount-volume for "L"ibraries, which is what the tripos handlers are, in a sense. BCPL code-snippets that get (as of tripos legacy) their own GLobVec, and execute file system functions for the user. In AmigaDos, the interface became a message ping-pong through the process message port.

Originally Posted by bloodline

It also seems that I could write a single handler for my device, and have two “DOSDriver” configuration files for each of the features provided by my device.

Yes, in principle.

Originally Posted by bloodline

-edit- also it seems some exec devices can accept DOSPackets directly!?

No. One could in principle design such devices, though it does not make too much sense as their responsbilities are quite different. Exec devices provide low-level abstractions of hardware devices, and provide functions such as byte-access. Dos Handlers provide file-system abstractions on top of them. An exec device has no idea about files, and has no idea how to share its resources or organize its device underneath. It is rather stupid byte I/O (serialized) or block I/O (random access). The dos handler on top provides abstractions in forms of files on the device.

Originally Posted by bloodline

i.e. the trackdisk.device doesn’t seem to have a handler between dos and the device.

Yes, of course. What sits between them is the FastFileSystem. So DF0: has a port that communicates to the fast file system, and the fast file system communicates to the trackdisk.device.

[bloodline]

Thomas & phx, Many thanks guys I finally feel like I’m understanding this now.

At first it felt like a really messy design, but the more I think about it, the more elegant it seems, at least in concept if not implementation