Some questions about devices and compilers

[Mathesar]

I have finally started work in this idea I had for a super simple Amiga 500 SPI controller. The hardware for this is now done so I am turning my attention to the software. The SPI controller has two (shared) ports that I want to use for an SD card and an ENC28J60 ethernet controller.

I was looking at Mike Stirling's excellent K1208 drivers https://github.com/mikestir/k1208-driversand I plan to adapt those for my hardware. It will involve some extra work as I will need some mechanism to arbitrate access to the shared SPI bus and I need to make the ethernet controller a polling one as I have no interrupt but those are things I can handle.

What I cannot handle though is the whole GCC/Linux/Cygwin/make-file/help!!! business though. I found Niklas's SPI parallel port projecthttps://github.com/niklasekstrom/ami...to-spi-adapter and it appears he has the same problem that I have and has thus diverted to VBCC which seems much more straightforward. However, he has also converted the main driver into a threaded one by diverting all IO requests to a helper task.

And here is my first question about that:
Why does Mike's original driver work without a helper task? My understanding is that BeginIO() could be called at any time by any task so in theory BeginIO could be interrupted while handling an SD-card operation? Is it maybe that the OS handles subsequent calls to BeginIO so that it waits until the previous call returns before doing another call?

Question 2:
Mike's driver uses some macro's to generate the romtag structure and jump tables it seems. He also uses the -lamiga-dev GCC option that I assume handles this table generation. However, I cannot find any description of this option anywhere on the net. Jorgen Bilander has published a skeleton driver that generates the table using some inline assembly https://github.com/jbilander/SimpleDevice. That seems more obvious to me. As i can't get GCC to run on my PC and as ADE from aminet (which is GCC 2.95 based!) does not recognize the -lamiga-dev option, can't I use that method instead? But what is the linker option then to disable standard linking? Is it -nostartfiles? And will it work on the Amiga hosted ADE as well?


I know these are many questions but I hope someone can help me here. I was hoping to just change some low-level code and hit "build" but as this is not a standard piece of software and because of the whole compiler trickery this is more complex than I hoped.....

[Thomas Richter]

Originally Posted by Mathesar

Why does Mike's original driver work without a helper task? My understanding is that BeginIO() could be called at any time by any task so in theory BeginIO could be interrupted while handling an SD-card operation? Is it maybe that the OS handles subsequent calls to BeginIO so that it waits until the previous call returns before doing another call?

No, the Os does nothing like this. BeginIO() is called for every SendIO() and DoIO(), both of which are only thin wrappers around BeginIO().



Note that BeginIO() is supposed to be asynchroniously, unless the IOF_QUICK flag is set in which direct execution in the caller task is permitted. There are multiple strategies how it could be done without a helper task. The worst is by simply ignoring IOF_QUICK and always performing the operation synchronously, but this makes the whole point of SendIO() absurd and limits performance. Another one would be to queue up the IORequests in a list and work them off in some other kind of asynchronous operation such as an interrupt.


One way or another, the device task approach is considered the gold standard, so look somewhere else for a suitable example.


What BeginIO() typically does is that it checks the operation requested. If it is something very simple and the IOF_QUICK flag is set, then handle the operation immediately and return. Otherwise, clear the flag, and queue the IORequest up at the message port of the device task, and let it perform the operation. Such "quick" operations should only be those that require little to no time, and do not have to wait for hardware to complete an operation - the whole point of BeginIO() is to allow asynchronous I/O such that the caller *does not* have to wait for the operation to complete.



Concerning using gcc, I afraid I'm not of big help. For such very Amiga specific stuff, I would always recommend SAS/C or assembler stubs.

[Mathesar]

Originally Posted by Thomas Richter

The worst is by simply ignoring IOF_QUICK and always performing the operation synchronously, but this makes the whole point of SendIO() absurd and limits performance.

Thanks for your reply.
I think this is exactly what the Mike's driver does. As does the A500IDE driver from Mika. But then there should be some mechanism in the driver to prevent recurring calls from another task right? I mean, is it possible that BeginIO() gets called again from another process while it is still processing another request? For example, reading some blocks of data from the SD card takes quite some time. Another, higher priority, process could theoretically pre-empt that and also call BeginIO(). However, i cannot see any mechanism inside the driver to handle that case.

[Mathesar]

Now that I think of it, the SD card driver would be called by DOS right? Maybe that is the reason it works as maybe DOS will make sure BeginIO is not called from multiple processes?

[Thomas Richter]

BeginIO() can surely be called by multiple processes, but it is then up to the processes to ensure that the data is consistent. So for example, if you have two partitions on an SD card, there will be two processes accessing the SD card simultaneously, one for each partition. They won't interfere because they operate on different parts of the SD card, i.e. address different sectors, but it is up to the device to "sequence" operations, that is, perform one after another.

One way of doing this is to ensure (e.g. by a semaphore) that there can be only one calling task within the critical part of the device driver, but this is very sloppy as it prevents the other task calling from progressing. Instead, you queue up the operations at a work queue (a message port) and then the device task works on them one by one.

This is not a "hypothetical" setting, but happens every time there are more partitions on a device, and any well written file system such as the ones delivered with the Os perform IO asynchroniously, that is, they initiate an operation by SendIO(), and are then free to do something else and react on the user while the device is busy with other queued operations.

Only lazy file systems such as FAT95 "hang" as long as a device is busy performing the IO, but the Amiga is a multitasking operating system, and the device and the file system should take advantage of it and free the task or caller to do something else while the IO is operating.

BTW, there is no "DOS" process in AmigaOs that performs this operation. The "dos" is just a thin "virtual file system" which dispatches commands to file systems, i.e. other processes, to perform the actual operation. This is very close to the way how exec interacts with devices, just that the call is not "DoIO()" or "SendIO()" but "DoPkt()" and "PutMsg". So a file system works very much like a device driver, by very similar principles, but due to some historical accident, it uses a different set of functions for it.

[phx]

Originally Posted by Mathesar

Mike's driver uses some macro's to generate the romtag structure and jump tables it seems.

On first sight I didn't find them in the linked github repository, but you are probably talking about the macros from

exec/initializers.h

? Or did he make his own?

He also uses the -lamiga-dev GCC option that I assume handles this table generation.

I doubt that a linker library handles table generation. The -l option specifies a library to link with and I guess that libamiga-dev is a modified version of libamiga?

Jorgen Bilander has published a skeleton driver that generates the table using some inline assembly https://github.com/jbilander/SimpleDevice. That seems more obvious to me.

A simple method is certainly to do the library/device init and Romtag stuff in a separate assembler source. But you should be able to do it in C as well. But note that the

OFFSET

macro in

exec/initializers.h

is not really compatible with vbcc and should rather be implemented with

offsetof()

from

<stddef.h>

.

[Mathesar]

On first sight I didn't find them in the linked github repository, but you are probably talking about the macros from

exec/initializers.h

? Or did he make his own?

The line is here:
https://github.com/mikestir/k1208-dr.../device.c#L205
To me it suggest this Macro is used to generate the romtag table, but I can't find where the macro comes from. It's not from initializers.h. But yeah, I think I am gonna try my luck with VBCC anyway which can be hosted on Amiga. The seperate assembler file is way more comprehensible to me.

I doubt that a linker library handles table generation. The -l option specifies a library to link with and I guess that libamiga-dev is a modified version of libamiga?

Ah yes, that is probably it.

[Mathesar]

BeginIO() can surely be called by multiple processes, but it is then up to the processes to ensure that the data is consistent. So for example, if you have two partitions on an SD card, there will be two processes accessing the SD card simultaneously, one for each partition. They won't interfere because they operate on different parts of the SD card, i.e. address different sectors, but it is up to the device to "sequence" operations, that is, perform one after another.

One way of doing this is to ensure (e.g. by a semaphore) that there can be only one calling task within the critical part of the device driver, but this is very sloppy as it prevents the other task calling from progressing. Instead, you queue up the operations at a work queue (a message port) and then the device task works on them one by one.

This is not a "hypothetical" setting, but happens every time there are more partitions on a device, and any well written file system such as the ones delivered with the Os perform IO asynchroniously, that is, they initiate an operation by SendIO(), and are then free to do something else and react on the user while the device is busy with other queued operations.

I am pretty sure that Mike's driver and all it's derivatives like the SDbox driver do not use any semaphores (or any other mechanism) to prevent multiple IO requests from corrupting the data transfer. So, if one would create multiple partitions on the SD using that driver it could go wrong. Niklas's version uses a helper task and properly queues requests to that task so that one would work correctly. I guess most people just create a single FAT partition. However, my controller will be an internal controller to act as an Amiga HDD (and a network interface ) so I will use Niklas's version as a base. Ultimately I would like to make it autobooting as well....
But first things first!


Another quick question: The SPI bus will be shared between the network and SD device. I could make the SPI part a device on it's own but that would make things very complicated as SPI accesses are often very complex. So, the plan is to include the SPI code in each driver but make sure that once a driver asserts the chip select line for it's device (SD for example), the other device (network) will wait untill the chip select lines are all de-asserted (and thus "free") again. I was thinking of using a public semaphore for that. The first driver that comes up will check if the semaphore exists (during device init) and if not create the semaphore and make it public. The second driver will then find that semaphore and not create one of it's own. I think that will work right?

BTW, there is no "DOS" process in AmigaOs that performs this operation. The "dos" is just a thin "virtual file system" which dispatches commands to file systems, i.e. other processes, to perform the actual operation. This is very close to the way how exec interacts with devices, just that the call is not "DoIO()" or "SendIO()" but "DoPkt()" and "PutMsg". So a file system works very much like a device driver, by very similar principles, but due to some historical accident, it uses a different set of functions for it.

Thanks for the clarification! Now that you mention it, it probably has to do with the fact that AmigaDOS is actually Tripos right? It has been too long since I have done some Amiga coding...

[alkis]

Originally Posted by Mathesar

The line is here:
https://github.com/mikestir/k1208-dr.../device.c#L205
To me it suggest this Macro is used to generate the romtag table, but I can't find where the macro comes from. It's not from initializers.h. But yeah, I think I am gonna try my luck with VBCC anyway which can be hosted on Amiga. The seperate assembler file is way more comprehensible to me.
Ah yes, that is probably it.

The macro comes from stabs.h
In modern m68k-amigaos-gcc it's at m68k-amigaos/sys-include/stabs.h

[Thomas Richter]

Originally Posted by Mathesar

Another quick question: The SPI bus will be shared between the network and SD device. I could make the SPI part a device on it's own but that would make things very complicated as SPI accesses are often very complex. So, the plan is to include the SPI code in each driver but make sure that once a driver asserts the chip select line for it's device (SD for example), the other device (network) will wait untill the chip select lines are all de-asserted (and thus "free") again. I was thinking of using a public semaphore for that. The first driver that comes up will check if the semaphore exists (during device init) and if not create the semaphore and make it public. The second driver will then find that semaphore and not create one of it's own. I think that will work right?

I would recommend another approach, and use the trackdisk.device as blue-print for a design. The trackdisk device uses the disk.resource to allocate access to the actual hardware, and thus, you could create an SPI.resource for that, which potentially, could also handle the lower-level hardware abstraction and leave the media-specific handling to the device.
At bare minimum, the resource would just provide the semaphore, but from a software design perspective, it's probably good to have an abstraction around SPI available for multiple devices.

Originally Posted by Mathesar

Thanks for the clarification! Now that you mention it, it probably has to do with the fact that AmigaDOS is actually Tripos right? It has been too long since I have done some Amiga coding...

Well, that's certainly part of the Tripos design, but it is also part of the exec-Design and its devices. There is some parallelism between exec devices and tripos handlers. If you look into the tripos specs, its low-level device drivers work pretty much like exec devices. Potentially Carl took the exec design also from this document, or added this later to ease porting of Tripos to AmigaDos.

[Olaf Barthel]

Originally Posted by Mathesar

IJorgen Bilander has published a skeleton driver that generates the table using some inline assembly https://github.com/jbilander/SimpleDevice. That seems more obvious to me.

You do not need to resort to inline assembly. The same data structures can be built using 'C' only. However, there are no public 'C' data structures defined in the Amiga operating system header files which would allow you to replace the auto_init_tables data.

As for the simple device code, it indeed is simple to the point of leaving the entire device BeginIO function implementation as an exercise to the reader. This is hard, but very easy to get subtly and fatally wrong

The device initialization function init_device ought to verify that it is running on the operating system version it expects. As written, it will accept anything, including Kickstart 1.0. While this may sound picky, you can save yourself hours of debugging by testing for operating system versions which your driver does not support. Even if it does not crash on your system, somebody else might see her or his machine freeze for no apparent reason.

The device function open would benefit from checking if the I/O request size matches what the device expects. If it happens to be shorter than expected (check the IORequest->io_Message.mn_Length field; it ought to be at least as large as sizeof(struct IORequest)) all kinds of random side-effects may occur, making it much harder to diagnose the problem.

[Mathesar]

Thanks all for the helpful comments so far. A small update:
I have managed to install vbcc on my A3000. It compiles Niklas's version of the driver just fine after adding a missing header define. I now have a development flow going where I edit my code using stormC and then compile with vbcc using a script that automatically copies the compiled binaries to a floppy with which I can boot the A500 for testing.
This way I was able to verify that the hardware indeed works as I hoped it would. For me, that means I have at least 3 milestones done (hardware designed, compiler setup, hardware verified).
The next step is to replace Niklas's SPI code by my own.
I'll keep you posted with my progress.

[Mathesar]

Small update again. Although this is actually a hardware project, most work is in the software. So I guess this is the right sub forum.
I implemented my bit-banging SPI routines and the driver actually works. I can boot from a floppy now that mounts the SD card and then hands over to the workbench partition.
It is not perfect though as sometimes, on some sectors, the SD card responds with an "illegal command" error. Only an SD card reset helps then. So I have hacked the SD code a bit so that the card is automatically reset and the command re-issued whenever that happens. I suspect the original SD code because it does some things I haven't seen before. Since it also handles the CS signal in a weird way which makes my life harder when I will add the ethernet controller I will replace this part with the SD driver from the MiniMig ARM controller. That code is mature and quite easily ported.

So, it works but it is not fast, sysinfo reports just under 37Kbyte/s.... However, when navigating through ClassicWB, starting some programs, etc, it is actually not too bad It certainly beats floppy.

Some pictures of SysInfo and how the hardware looks with the SD card connected.

[Mathesar]

Originally Posted by Mathesar

I suspect the original SD code because it does some things I haven't seen before.

I eat my words. It was my own fault and the original code is (99%) fine. The problem was I had some code commented out while fixing an issue with the driver not initializing some cards correctly. I had to make some minimal changes so that the driver is less fussy. Once I fixed that though I forgot to un-comment it again.... Anyway, the driver is now working 100% .

Question (@Thomas, Olaf): Is there any information about coding a resource? I agree that it is the nicest way of sharing the SPI bus between drivers but I can't find any information on how to code such a resource. After that the next step is to port the Ethernet driver!

[Thomas Richter]

There is not much documented about resources because there is really not much to say. A resource is only a "struct Node" of type NT_RESOURCE, a valid name, and you can add one to exec with AddResource(). That's really all.

Resources cannot be loaded from disk, they cannot be flushed, they don't have open and close vectors, actually they don't have to have vectors at all.

It is probably advisable to give the resource a "struct Library" and provide function vectors, that is, make it a library look-a-like such that you can provide custom functions through the resource. But that's not mandatory.

[Mathesar]

Another update, progress was slow but steady. The last update was that the issue with the SD card driver was on my side That has been fixed now and the SD card works 100%. I also created a resource as suggested that so far holds a semaphore to arbitrate access to the shared SPI bus. I then started work on the network driver and that took more time than expected. I needed to change the driver into a polling one as I have no interrupt capability on my simple bit-banging SPI hardware. I also had to port the driver over to VBCC and modify the code to use the shared SPI bus. The network adapter is now polled on the VBlank interval. And I am very happy to report that this evening I got AmiTCP running succesfully!

Pictures below... Only pinging my favorite website so far but it means the driver and TCP/IP stack are working. I am using a cheap ENC28J60 module and a cheap SD card module both attached via du-pont cables to the SPI controller. The SPI controller itself is just 5 standard logic chips so this is a very cheap build. The Amiga 500 further has PeteAU's trapdoor ram with Gary adapter installed for a total of 1MB chip and 1.5MB fast ram. That's all. Now I need to install some software that actually uses the network. I think I'll try an FTP client first

[nogginthenog]

That is very impressive. Well done!
You should start a new thread.

[Mathesar]

Yes, I will do that. Everything will be released as open-source when it's reasonably done.
It's fun to see a more or less standard A500 connected to the internet. It's slow as hell though. When pinging the machine with 1Kb packets from my laptop. The mouse even stutters... But still, I am going to try and run FTP or maybe even FTPmount. The big usecase for this is to have an easy way for moving files from/to a PC or to download stuff from Aminet.

My idea is that anyone can build this in his spare time and have a HDD and Ethernet on the cheap.

[malko]

Originally Posted by nogginthenog

That is very impressive. Well done!
[...]

You take the words out of my mouth


Well done Mathesar

[Mathesar]

Most of the hard work was done by Mike and Niklas who's code I'm using.
My drivers are basically their drivers adapted to work on a shared SPI bus.
Yesterday I installed AmiTCP and at first got just 1,4 kB/s download speeds from ftp.aminet.net. Then I realized the driver was not setting the SPI bus to high speed (for SD cards the SPI interface needs to start in low speed).
When I fixed that I got about 8..9 kB/s download speeds. Quite respectable for a 7Mhz 68000 without fast RAM