Writing a dynamic library

[Sim085]

For the next phase of my project I want to write dynamic libraries. I have found an example dynamic library (screennotify) on this forum, here;
http://eab.abime.net/showthread.php?t=77885

After going through the code I re-wrote the library with mainly two source files, one is mylib.c (based on screennotify.c) which contains the LibraryInit, LibraryOpen, LibraryClose and LibraryExpunge functions, and the other one is the functions.c which contains just one function that returns int value when called named giveMeInt()

The library compiles successfully and I use the following code to call it.

struct Library *lib = OpenLibrary("System:MyLib/mylib.library",0);
if(lib)
{
   printf("Loaded.\n");
}

When I run the above code I get "Loaded" printed in the console. So I am assuming the Library is getting loaded. At this point however I am a little stuck as I cannot understand how to call the library function after I load this.

What I did try is to include functions.h from the library (which has the prototype for giveMeInt() function) in the application, but when I try to compile the application I get an error "Reference to undefined symbol _giveMeInt".

Is this because I do not have the proto, pragma and clib header files? The proto and clib header files look straight forward to write, but the pragma header file looks like it is generated.

This is from screennotify;

#pragma libcall ScreenNotifyBase AddCloseScreenClient 1e 09803
#pragma libcall ScreenNotifyBase RemCloseScreenClient 24 801

...

I do not know what the numbers at the end come from. Also cannot understand why the function names do not start with '_' like they do in functions.c.

I use vbcc as a compiler which is why I tried to base my solution around screennotify.

[Thomas Richter]

The numbers in the pragma files identify the registers through which the functions receive parameters. This requires some compiler magic to retrieve the parameters from there, and this magic is of course compiler specific. I'm not sure how or if vbcc can do this for you.

To call library functions from your code, you need to include a pragma file as above, though this file is also compiler specific. There is a FD2Pragma program out in Aminet that can create these pragma files automatically for you, from a library ".fd" file.

Thus, the following steps need to be done:

1) Prepare your library implementation such that it retrieves parameters from registers, not from the stack.
2) document these registers in a ".fd" file, listing the parameters.
3) Create a library "prototypes" file that contains all the function prototypes
4) Create from the ".fd" file a "pragma" file for your target compiler.
5) Include in your project using the library both the prototype file and the pragma file.

[phx]

Usually there are two ways to call shared library functions from Amiga compilers:

1. Write stub routines which receive the arguments on the stack (following the standard m68k C-ABI). These routines copy the arguments into the appropriate registers and call the library function via its LVO. These stubs are usually implemented in assembler and part of amiga.lib for the standard libraries. You would have to make your own linker library here.
2. Use a compiler-specific feature to call the function directly. For vbcc, this would be by calling an assembler inline function through a macro, to pass the library base.

In any case your library should come with a clib/libname_protos.h, which declares all prototypes for your library functions, and a proto/libname.h, which includes the aforementioned prototypes and optionally a compiler-specific call method. For vbcc, it would include the assembler inlines and macros from inline/libname_protos.h.

But fear not, you don't have to create all these headers manually. As Thomas already mentioned, there are tools like FD2Pragma, which generate them automatically for all common compilers. All you need is an up to date FD or SFD file for your library.

Looking at the ScreenNotify source, an FD file might look like:

##base _ScreenNotifyBase
##bias 30
##public
AddCloseScreenClient(scr,port,pri)(a0/a1/d0)
RemCloseScreenClient(node)(a0)
etc..
##end

Then use fd2pragma to generate all the headers you need. For example for vbcc:
Proto header:

fd2pragma special 38 my_lib.fd

Inlines:

fd2pragma special 70 voidbase my_lib.fd clib path_to/clib/mylib_protos.h

Link library with stubs:

fd2pragma special 12 hunkname CODE onlycnames my_lib.fd

Make sure you have a recent fd2pragma version. The one on Aminet is quite old. Here is the last release:
http://phoenix.owl.de/fd2pragma.tar.gz

Looking at the ScreenNotify code I noticed that it uses a static library base. This should be avoided. Use the library base passed as an argument via register A6 instead. You can also call the vbcc-specific assembler inlines directly to pass the library base as an argument.

[thomas]

fd2pragma is included with vbcc.

And... vbcc does not use pragmas!
You should rather create vbcc inline files with fd2pragma.

[Sim085]

Using fd2pragma that comes with vbcc I generated the required files and managed to get the application using the library to compile. However it crashes the moment I call the library function. I am trying to see what I did wrong. However in the meantime wanted to ask about the below. What does "Use the library base passed as an argument via register A6 instead" mean?

Originally Posted by phx

Looking at the ScreenNotify code I noticed that it uses a static library base. This should be avoided. Use the library base passed as an argument via register A6 instead. You can also call the vbcc-specific assembler inlines directly to pass the library base as an argument.

[phx]

Originally Posted by Sim085

Using fd2pragma that comes with vbcc I generated the required files and managed to get the application using the library to compile. However it crashes the moment I call the library function.

Should be easy to find the reason with a debugger. Library base not initialized?
Did you call your library function via the stub routine or the assembler inline? When including <proto/my..._protos.h> it should be the latter.

What does "Use the library base passed as an argument via register A6 instead" mean?

AmigaOS shared libraries always receive the library base in A6 when doing a function call. You should use that and avoid your static library base. So instead of...

SAVEDS BOOL _RemCloseScreenClient(REG(a0) struct Node *n)

...use this:

SAVEDS BOOL _RemCloseScreenClient(REG(a6) struct ScreenNotifyBase *ScreenNotifyBase, REG(a0) struct Node *n)

Avoid all static variables, unless constant, and store them in your ScreenNotifyBase structure. That's where they belong. A good library will only write to the base structure and never to its own code or data. For example:

ScreenNotifyBase->SysBase

. Then you can call OS functions directly via their inline function, by passing the base as the first argument (this is vbcc-specific). For example:

  __Remove(ScreenNotifyBase->SysBase, n);

[Sim085]

So if I am understanding correctly, when opening a library the LibraryInit function is called. This calls the MakeLibrary function which returns a Library (+ extra space as per the dSize parameter for further initialization) and then through AddLibrary function this is sort of registered so that it can be accessed from other applications.

The registered library is then passed to any of the library function calls automatically (this parameter does not need to be passed by the function caller as is done by the OS).

Originally Posted by phx

AmigaOS shared libraries always receive the library base in A6 when doing a function call. You should use that and avoid your static library base.

To be honest still a little out of my depth with these Libraries

Originally Posted by phx

Should be easy to find the reason with a debugger. Library base not initialized?

For example I cannot understand why the ScreenNotify library (and therefore the copy I made from this) stops compiling the moment I put '#include <stdio.h>' in the source file where the MakeLibrary function is called (given error: invalid types for assignment). I do not understand why including the '#include <stdio.h>' triggers such an error.

Something else which is confusing me is if the Library is really getting initialized or not. Reason I say this is that I am trying to write into a file from the LibraryInit function (just for testing), however, while when calling OpenLibrary I do get a Library back, I cannot see the file being created . I also try using kprintf but likewise, nothing gets printed on sashimi.

[Samurai_Crow]

Use only DOS.library functions to access files. The static libs invoked by fopen and fprintf are not reentrant code. Never use stdio.h from a shared library. It might crash the system.

Also: be sure to open DOS.library explicitly in the library startup code and close it in the shutdown code. Auto opening invokes another static library so it also can crash the system.

[Thomas Richter]

Originally Posted by Sim085

So if I am understanding correctly, when opening a library the LibraryInit function is called. This calls the MakeLibrary function which returns a Library (+ extra space as per the dSize parameter for further initialization) and then through AddLibrary function this is sort of registered so that it can be accessed from other applications.

It depends on the AUTOINIT flag in the resident structure that identifies your library. If not set, you have to create the library yourself. I would not do that but rather depend on RTF_AUTOINIT in which case exec calls MakeLibrary for you.


Whatever it is, I would probably suggest a compiler that has somewhat better support to create libaries. SAS/C with its LIBCODE flag can take over a lot of the lower level magic for you.

Originally Posted by Sim085

The registered library is then passed to any of the library function calls automatically (this parameter does not need to be passed by the function caller as is done by the OS).

No, there is really nothing done automatically. The registered libary is just a "struct Library *", and it is the job of the user of a library to load register a6 with a pointer to your library before passing. The stub functions or pragmas created by FD2Pragma do that, i.e. they instruct a user of a library to load a6 with the library base pointer so you can it there from your code.


A library is just a big table of function pointers at negative offset from a struct Library, plus a calling convention to put the pointer to this function table in register a6.

Originally Posted by Sim085

For example I cannot understand why the ScreenNotify library (and therefore the copy I made from this) stops compiling the moment I put '#include <stdio.h>' in the source file where the MakeLibrary function is called (given error: invalid types for assignment). I do not understand why including the '#include <stdio.h>' triggers such an error.

Whatever the reason, library code is not the right place for C style I/O. A library is supposed to be used by multiple tasks simultaneously at the same time, i.e. it can happen that two programs run your same library function at the same point in time. C stdio is not well equipped to manage this situation.


For the very same reason "global" and static objects should not be used in the library unless they are "const". Again, such objecs exist only once per library intance, and as the library is just a thin layer around your function, they are shared by every caller of your library. Thus, if you modify an object with global linkage in a function,that change becomes visible immediately to every other caller of the same function (or every other function that uses that object). Since you do have no control on which task calls which library function at which point in time, you run into the risk of "data races", of objects that are modified (by a second task) under the feet of the calling task.


Actually, C stdio depends on a couple of such global objects (such as the file table) and these objects are not even initialized properly as the C startup code (which usually does that) is not run. Thus - don't. stdio and library code do not mix.

Originally Posted by Sim085

Something else which is confusing me is if the Library is really getting initialized or not. Reason I say this is that I am trying to write into a file from the LibraryInit function (just for testing), however, while when calling OpenLibrary I do get a Library back, I cannot see the file being created . I also try using kprintf but likewise, nothing gets printed on sashimi.

LibInit should probably not do much of file writing as it is called from within the ramlib process, and thus if much activity is run there, you are putting any other request to open a library back.


Actually, traditional debuggers are not well prepared to debug library code. However, if your library opens, libinit has been called - at some point in time. Probably from of an earlier version of your same library if you run through mulitple debug cycles. Your library does not go away just because the using program dies. Instead, the Os removes libraries whenever it "feels like it", potentially because memory runs low. In such a case, loaded but unused libraries are flushed from memory. Actually, your library has to do that itself through the LibExpunge() vector which you have to provide, and which has to do the cleanup job, all provided there are no users of your library left. So again, it is all up to you again to remove your library from the system if the Os asks for it.


To trigger a library expunge, type "avail flush" from a shell, or use the "flush libraries" from the workbench debug menu if that is enabled.

[phx]

Originally Posted by Sim085

The registered library is then passed to any of the library function calls automatically (this parameter does not need to be passed by the function caller as is done by the OS).

At least the C compiler hides that from you. I don't know if you ever did 68k assembly programming on the Amiga? Then you would know that you always have to load A6 with the library base pointer. There are only very few functions in AmigaOS which can do without, and this is documented.

When calling such a function from C then either the amiga.lib stub routine will load the library base for you (from a static variable with a fixed name, like SysBase), or a macro/pragma will do it, before calling compiler-specific inline code.

In any case, you can always rely on A6 having your library base.

For example I cannot understand why the ScreenNotify library (and therefore the copy I made from this) stops compiling the moment I put '#include <stdio.h>' in the source file where the MakeLibrary function is called

As others already said: never use stdio in a shared library! Stdio routines are provided by the compiler's clib (vc.lib in this case) and require a proper startup code for initialization. A shared library must not have any C startup code! For vbcc this means you have to link your library with the -nostdlib option (very important!), so neither the startup code nor vc.lib is linked with it.
There are only few clib functions which are safe to use, like those from <string.h>.

(given error: invalid types for assignment). I do not understand why including the '#include <stdio.h>' triggers such an error.

Showing the code for this error message would help analyzing it, but I would guess that you are using NULL as interger-zero, although it is meant for pointers only. By including vbcc's stdio.h you will get NULL defined as

#define NULL ((void *)0)

, which helps you to detect these bugs (unlike the AmigaOS headers, which define NULL as 0L).

Something else which is confusing me is if the Library is really getting initialized or not.

Adding to the hints already given I can only say that a good way to debug a library would be by using the UAE debugger, because it is not part of AmigaOS. Set a rare no-op breakpoint in your LibInit function, like:

__asm("\tdc.w\t$cf4f");

. Then type the command

fi cf4f

in the UAE-debugger to break on it. Then trace through the following code.

[Sim085]

Thank you all for all the help. I took some time to try and absorb everything and then I started again.

First I wanted to ensure my library was initializing and opening correctly, so I simply created the library, found how to enable debugging and using kprintf confirmed library is loading correctly from the application. While I still have MyLibBase declared as global, I do not make use of this reference directly from any function (apart when initializing this).

The problem starts when I try to add a new function to call from the application.

Given I do not want to use MyLibBase global reference I have declared my function as follows;

SAVEDS int mylib_myfunc(REG(a6) struct Library *base)
{
      DEBUGLOG(kprintf("mylib_myfunc() called.\n");) 
      return 1;
}

Included the function name in the LibraryVectors and compiled successfully. So far so good.

Next I created /clib/mylib_protos.h as follows;

...
int mylib_myfunc(struct Library *);
...

Then I created /fd/mylib_lib.fd as follows;

...
mylib_myfunc(base)(a6)
...

Using the fd file I created the Proto, Inlines headers and Link Library. Only deviation from the given commands is I removed voidbase when generating the Inlines as this is no longer the case, i.e. -

fd2pragma special 70 mylib_lib.fd clib /include/clib/mylib_protos.h

So my /inlines/mylib_protos.h has the following;

...
int __mylib_myfunc(__reg("a6") struct Library *)="\tjsr\t-30(a6)"; 
#define mylib_myfunc() __mylib_myfunc(MyLibBase) 
...

So...

I moved /clib/mylib_protos.h, /proto/mylib.h, /inline/mylib_protos.h and mylib.lib to my application source directory.

Only think I changes is the includes in /proto/mylib.h so instead of including <clib/mylib_protos.h> and <inline/mylib_protos.h> I include them as "/clib/mylib_protos.h" and "/inline/mylib_protos.h" (as these are in my source directory).

I updated my makefile to include the link library (... -lamiga mylib.lib)

I compile...

But I get the following error...

Reference to undefined symbol _MyLibBase
vlink failed return code 20

I know mylib.lib is found because if I delete it the compiler will complain saying "No match for mylib.lib".

From what I can see the "undefined symbol _MyLibBase" is in relation to the following entry in the /inlines/mylib_protos.h file.

#define mylib_myfunc() __mylib_myfunc(MyLibBase)

Not sure what step I did wrong or what am I missing. The /proto/mylib.h header file does have an entry as follows;

#ifndef __NOLIBBASE__ 
extern struct Library *MyLibBase; 
#endif

And the inclusion of "/inlines/mylib_protos.h" only comes after this.

So I am little bit stuck

[bebbo]

Originally Posted by Sim085

Thank you all for all the help. I took some time to try and absorb everything and then I started again.



...



I compile...

But I get the following error...

Reference to undefined symbol _MyLibBase
vlink failed return code 20

...

So I am little bit stuck

you need


struct Library *MyLibBase;

in your code.

[alkis]

Also, take a look here
https://github.com/alexalkis/library

It might or might not help.

[phx]

Right. Your test application does not link, because you didn't define a Library Base. The

extern

keyword means that the symbol is externally defined. No memory will be allocated by this statement. It is just a type declaration. Somebody has to define it (typically your application)!

But there are more serious problems in the library itself.

Originally Posted by Sim085

While I still have MyLibBase declared as global, I do not make use of this reference directly from any function

Then you can also delete it (in the library code, not in the test app!).

Given I do not want to use MyLibBase global reference I have declared my function as follows;

  SAVEDS int mylib_myfunc(REG(a6) struct Library *base)

Correct so far, but you should make sure that

mylib_myfunc

is not your library's real function name, to avoid name conflicts. This name is only important for entering it into your

LibraryVectors

table. Nothing else. It may even be static (local), when the function table is part of the same file.

On a sidenote, SAVEDS doesn't hurt, but is not required at all, as long as you don't use the small data model inside your library (which is quite uncommon).

Next I created /clib/mylib_protos.h as follows;

int mylib_myfunc(struct Library *);

This would be bad. Don't use the same name as in your actual library code! Also don't specify your llibrary base as a real C argument. Your real library function is probably:

int myfunc(void);

 mylib_myfunc(base)(a6)

See above. This would be:

myfunc()()

So my /inlines/mylib_protos.h has the following;

...
int __mylib_myfunc(__reg("a6") struct Library *)="\tjsr\t-30(a6)"; 
#define mylib_myfunc() __mylib_myfunc(MyLibBase) 
...

I wonder that fd2pragma was smart enough not to write the Library base twice here...

I updated my makefile to include the link library (... -lamiga mylib.lib)

That doesn't break anything, but specifying

mylib.lib

as an object file would effectively add the whole link library to your executable, instead of just the objects which have symbol references. Tell the linker that it is a library:

... -lamiga -lmylib

. Also add another search path with

-L

, when needed.


To give another example, including the AddThese() function from Alkis' example library would look like this:

int mylib_AddThese(REG(a6) struct Library *MyLibBase, REG(d0) int a, REG(d1) int b)
{
    return a + b;
}

Clib:

int AddThese(int, int);

FD-File:

AddThese(a,b)(d0,d1)

And if you have to use functions from other shared libraries, open them in your LibraryInit function and store their base in your library base. Example for a function using DOS Delay():

#include <proto/dos.h>
...
void mylib_WaitSeconds(REG(a6) struct MyLibBase *base, REG(d0) int seconds)
{
    __Delay(base->DOSBase, seconds*50);
}

Clib:

void WaitSeconds(int);

FD:

WaitSeconds(seconds)(d0)

Originally Posted by alkis

Also, take a look here
https://github.com/alexalkis/library

Good example library, but as I understand the task here is to write it in pure C.

[alkis]

Originally Posted by phx

#include <proto/dos.h>
...
void mylib_WaitSeconds(REG(a6) struct MyLibBase *base, REG(d0) int seconds)
{
    __Delay(base->DOSBase, seconds*50);
}

Clib:

void WaitSeconds(int);

FD:

WaitSeconds(seconds)(d0)

A usual trick, *if your compiler has macro-inline calls*, is to:

#define DOSBase base->DOSBase

that lets you call Delay normally Delay(ticks);

(example here http://franke.ms/cex/z/sfGhsj )

[Sim085]

I have applied all the fixes (renamed function in library, modified fd and clib header file, used -L in make file for the link library, etc.) and the good news is that it works .

Thank you so much for all the help with this.

Originally Posted by phx

Right. Your test application does not link, because you didn't define a Library Base. The

extern

keyword means that the symbol is externally defined. No memory will be allocated by this statement. It is just a type declaration. Somebody has to define it (typically your application)!

Originally Posted by bebbo

you need
struct Library *MyLibBase;

in your code.

With regards to the library variable name; so in summary it means that when opening the library I always need to define the variable name to be exactly the same as in the inline/mylib_protos.h header file (in my case MyLibBase);

struct Library *MyLibBase = OpenLibrary(".../.../MyLib.library");
if(MyLibBase)
{
   mylib_myfunc();
}

[phx]

Originally Posted by Sim085

and the good news is that it works .

With regards to the library variable name; so in summary it means that when opening the library I always need to define the variable name to be exactly the same as in the inline/mylib_protos.h header file (in my case MyLibBase);

Correct!

[Sim085]

I just realized my test application is compiling and running successfully regardless if I include the link library or not (-LMyLib.lib).

When is the link library required (or not required)?

[Samurai_Crow]

The link library is not required if you use the inlines/pragmas and required if you don't.

[Sim085]

Thank you.

Originally Posted by Samurai_Crow

The link library is not required if you use the inlines/pragmas and required if you don't.