WinUAE tools that use x86 dll

[thellier]

I have tried to make a list for those tools that use an x86 .dll:

Alain thellier

======================================================
WinUAE tools that use x86 dll
======================================================
QuarkTex: Use native 3D hardware
http://aminet.net/package/driver/video/QuarkTex
agl.library --> winuae/alib/QuarkTex.alib
Warp3D.library --> winuae/alib/QuarkTex.alib
======================================================
Wazp3D: Use native 3D hardware
http://aminet.net/package/driver/video/Wazp3D
Warp3D.library --> soft3d.library -> winuae/sof3d.dll
======================================================
WinLaunch: Launch native Windows apps
http://aminet.net/package/util/misc/winlaunch
winlaunch --> winuae/winuae_dll/winlaunch.alib
======================================================
Anaiis: Use native USB stack
http://aminet.net/package/driver/other/anaiis
http://aminet.net/package/driver/other/anaiis_hostusb
Anaiis --> winuae/hostusb.dll -->libusb0.dll
======================================================
winuaevidrec: TV-Watching and Videorecording Tool
https://www.google.fr/#q=winuae_vidrec.lha
winuavidrec --> winuae/winuae_dll/amigavideo.alib
======================================================
MacroWave-native: software synthesizer (project)
http://amigaworld.net/modules/newbb/...orum=15#716260
Moog Cascade/BiQuad (Butterworth)/Simple Differential Filter --> ???.dll
======================================================

[James]

http://www.david-mcminn.co.uk/bernd/...ue_lame1.0.lha

[thellier]

Ok thanks . Added to list...

======================================================
WinUAE tools that use x86 dll
======================================================
QuarkTex: Use native 3D hardware
http://aminet.net/package/driver/video/QuarkTex
agl.library --> winuae/alib/QuarkTex.alib
Warp3D.library --> winuae/alib/QuarkTex.alib
======================================================
Wazp3D: Use native 3D hardware
http://aminet.net/package/driver/video/Wazp3D
Warp3D.library --> soft3d.library -> winuae/sof3d.dll
======================================================
WinLaunch: Launch native Windows apps
http://aminet.net/package/util/misc/winlaunch
winlaunch --> winuae/winuae_dll/winlaunch.alib
======================================================
Anaiis: Use native USB stack
http://aminet.net/package/driver/other/anaiis
http://aminet.net/package/driver/other/anaiis_hostusb
Anaiis --> winuae/hostusb.dll -->libusb0.dll
======================================================
winuaevidrec: TV-Watching and Videorecording Tool
https://www.google.fr/#q=winuae_vidrec.lha
winuavidrec --> winuae/winuae_dll/amigavideo.alib
======================================================
MacroWave-native: software synthesizer (project) with native x86 filters
http://amigaworld.net/modules/newbb/...orum=15#716260
Moog Cascade/BiQuad (Butterworth)/Simple Differential Filters --> ???.dll
======================================================
winuae_lame: use Windows lame as a seperate windows thread for encoding MP3
http://www.david-mcminn.co.uk/bernd/...ue_lame1.0.lha
winuae_lame --> winuae/alib/lame.alib --> lame.exe
======================================================

[Leandro Jardim]

Thanks! I was looking for that.

[Gilloo]

Please take care of vectors...
http://eab.abime.net/support-winuae/...-settings.html
check uae.resource before using $ffc0 to access Windows(TM)(c) dlls

[Der Wanderer]

I am the author of MacroWave (not released yet).

I just want to confirm that I see some nice speedups with the native code, depending on the algorithm it is 1x up to 20x, I guess depends how much complex instructions like vectorisation can be used on the x86 side. This also indicates there is room for improvement for the JIT.

One thing I would like to see though is some kind of standard across emulations to embed x86 code (or say generally native code) into 68k.
E.g. a special 68K instruction like JNSR (jump native sub routine). Then we could make a shared library that contains native code. Right now it is a bit cumbersome to write a library that opens a DLL that must be installed separately on the Windows side/filesystem. It would be much better to be able to just replace a library with a mixed binary that contains the 68K stubs and the native code.

[Gilloo]

@ Der Wanderer,
Yes it is a heavy work to make a dll only for WinUAE, but it rocks!

I hope that one day a kind of WinUAE will run on a Raspberry Pi, so specific x86 code will be useless.

[thellier]

One thing I would like to see too : A simple way to convert a pointer between Amiga and PC side

I explain : if the adress is in Z3 memory there is an z3-offset but if the adress is in chip there is an other offset to use for converting
But often you dont know what reference a pointer in an OS structure (chip ? z3 ? other ?)
as you didnt allocated it yourself

Should be better to have builtin functions PointerToAmiga() PointerToPC()

Alain Thellier - Wazp3D

[FrodeSolheim]

@thellier It would probably be useful to be able to give the native libraries a list of function pointers it can use to call back to UAE. For example by adding function pointers to the struct passed as one parameter, or by giving a struct with pointers to a library initialization call. One such function could be void *get_real_address(uint32_t amiga_address)...

Regarding native code interface, embedded native code, and cross platform concerns:

There is a couple of quite different approaches for native code interfaces:

1. Allow Amiga-code to load/reference a dynamic library installed on the host, look up functions in the library and call them. The advantage here is that you're using the normal dynamic library support in the OS, so the library can further link to other system libraries and do useful stuff (such as calling Win32 functions on Windows, or interfacing with OpenGL on all platforms). But this also requires that a library is provided for all host architectures you want it to work with (i.e. separate libraries for windows-x86, windows-x86-64, linux-x86, linux-x86-64, macosx-x86, macosx-x86-64, etc). This is the native code interface approach currently implemented by WinUAE.

2. Another approach is to bypass the dynamic library support and just load native binary code into executable memory and allow the Amiga to execute it. This could allow a generic "x86 binary" to be loaded, and the same x86 code could work on Windows, Linux, and OS X (and other x86 platforms). It could for example be loaded from a .so file in ELF format. But note that for example x86-64 version of UAE would need separate x86-64 code. As this binary does not use the dynamic linker of the host OS, it will not be able to "easily" use other APIs on the host system. But it could be useful to for example use the host CPU as a kind of CPU accelerator, do heavy calculations, etc.

2a. The x86 (or another CPU architecture) binary blob could then in theory be loaded from Amiga memory. It could also in theory execute directly from emulated Amiga memory, but since we generally want to leave the execute bit off for these memory areas, it would be better to load (copy) the binary blob from Amiga memory to executable host memory before it runs. Of course, this has security implications, and "new Amiga viruses" could suddenly be a lot more scary... (and such a feature could never be enabled by default). It could of course be combined with some simple code signature verification so only code from user-white-listed authors would be loaded through such an interface.

2b. As a variant of 2., one could still allow somewhat easy access to other native libraries and APIs by providing the binary blob with a way to access some functions exported by UAE, such as "open_library", etc. (basically reinventing dynamic library loading, but possibly in a cross-platform way - as far as that is possible).

Approach (1) is by far the simplest and flexible in terms of what it can do But code delivery requires local libraries for all architectures. The other approaches are not especially difficult either, but definitively more work. There are many possibilities (and other approaches not mentioned here), depending on what one wants to achieve

[Toni Wilen]

Most importantly it MUST NOT freeze the emulation while native code is running.

UAE "boot rom" native wrapper code that m68k code calls would first pass parameter structure (not registers) to separate thread (or thread pool in future) which executes native code, then calling Amiga task is put to sleep by calling exec/Wait(some standard reserved signal bit). When native code returns, m68k-side parameter structure is updated and task is woken up by exec/Signal().

[FrodeSolheim]

Being able to run native code asynchronously must surely be useful. Perhaps there are some use cases for running a native function synchronously too? A data register could carry a flags bit field which could let the user choose between synchronous and asynchronous call.

Regarding "parameter structure", is that necessary? It seems that saving the registers (similar to the current behavior) and associating those with the native function would work well enough. For return value(s), the Amiga client can pass a pointer in one or more of the registers, and the native function can just write the result(s) directly to Amiga memory? Perhaps I am missing something...

[Toni Wilen]

Originally Posted by FrodeSolheim

Regarding "parameter structure", is that necessary? It seems that saving the registers (similar to the current behavior) and associating those with the native function would work well enough. For return value(s), the Amiga client can pass a pointer in one or more of the registers, and the native function can just write the result(s) directly to Amiga memory? Perhaps I am missing something...

Because I want some kind of spec, not just random parameters, "do what you want". Data structure can also contain pointer to table(s) that contain useful parameters that native code can use directly, including all kinds of pointers to helper functions (in native C code).

It really needs to have official way to support new features and extensions.

[FrodeSolheim]

Well, I completely agree that the native function itself will receive a data structure, which can contain the UAE native interface version number, arguments from m68k and helper function pointers. Extensibility would be ensured by only adding new stuff to the end of the struct of course, and version information in the struct can allow the native function to know what struct members are valid. Another good idea is to let the native library specify a minimum supported "UAE Native Interface" version, so the native functions are not used at all if the library requires too new features.

But you also said "m68k-side parameter structure" and that's where I don't immediately see an advantage. Since there isn't a compiler involved to match the function calls, there isn't any way to enforce a standard anyway (I mean check that it is used correctly).

Sure we could mandate that the m68k sets up a struct, but that would either have to be something like this predefined struct (ignoring data types):

struct {
    result;
    param1;
    param2;
    param3;
    param4;
}

or something like this:

struct {
    result;
    ....
}

(and the native function must know what fields following result are valid).

I don't think either of these cases are significantly better than just passing values in the registers. In all cases, there is an implicit contract between the Amiga-side caller and the native callee - knowing what data is passed where - and breaking that contract will have bad consequences. I guess one use for a standard argument passing structure is if the native interface layer would always like to be able to pass some kind of exception/error information back to the caller, but I'm not sure how useful that would be.

As for the native functions themselves, they could look something like this:

void (*native_function)(struct uni_data* uni);

Where the uni_data struct could look like something like this:

struct uni_data {
    uae_u32 version;  // native interface version
    uae_u32 a1, a2, a3, a4, a5, a7;
    uae_u32 d1, d2, d3, d4, d5, d6, d7;
    void * (*get_real_address)(uae_u32 ptr);
    /* more helper functions and extensibility for future fields */
};

Library should define a function like this (just reusing the previous struct here in this example though the register fields are obviously not useful here):

uae_u32 uni_init(struct uni_data* uni) {
    if (uni->version < UNI_MIN_VERSION) {
        return 0;
    }
    return 1;
}

Of course, versioning the native libraries themselves is also useful, but that could be left to the library authors (call a native function from the library to check library version and/or capabilities).

[Der Wanderer]

The emulation freeze is indeed a problem, because the native function is not subject to the AmigaOS task scheduling and would freeze the entire emulation until the function call returns. This is ok for short function calls, but of course a disaster for lengthy calls.

I had the idea that lengthy calls could be run in a separate host-thread and the calling AmigaOS task is put to sleep, until the host thread returns. This way it behaves like a normal blocking call but does not stop emulation and other 68K tasks. AmigaOS would even get some kind of limited multi core support.

However, whatever you guys do, keep it simple. I managed to write a native library (a pair of Amiga Shared Library and DLL) for WinUAE, but it was not that straight forward and is a little ugly/unsafe here and there. A little more complicated would have scared me away.
What I don't like is that the installation of this library must modify the host system file system. It is not possible to install the library on Amiga side only, e.g. it cannot be installed by an installer.
You can install the Amiga Shared Library, but still must manually copy the .dll to WinUAE directory or Win32 folder.

You could push the idea further and patch OpenLibrary in a way that it can open directly DLL or SO. This way all the glue code would become obsolete and creating a native library becomes really easy, plus you can install the 68K equivalent in parallel, if the AmigaOS install will run in a different environment that doesnt support native libs.

[Der Wanderer]

> Well, I completely agree that the native function itself will receive a data structure, which can contain interface version number,
Just design it is a way it works for once and for all. If the version number doesn't match, there is anyway not much you can do.

[FrodeSolheim]

I wrote an example of how an library using the UAE Native Interface could look like:

#include <string.h>
#include <uae/uni.h>

UNI_DEFINE_LIBRARY(NULL);

UNIAPI void UNICALL hello_world(struct uni *uni) {
    char* data = UNI_POINTER(a1);
    strcpy(data, "Hello, World!");
}

UNIAPI/UNICALL are there to specify C calling convention and dll export, for compilers which need it (=msvc ) UNI_DEFINE_LIBRARY defines a default initialization/version check function with an optional pointer to a user-supplied library initialization routine.

Example C code, Amiga side:

char data[64];
int library = uni_load_library("hello_world_library");
int function = uni_get_function(library, "hello_world");
uni_call_1(function, UNI_SYNCHRONOUS, &data);
/* data now contains the string "Hello, World!" */

uni_call_1 being a possible convenience function which takes a function, a set of flags, and a single integer parameter which is written to (for example) register a1 - which is referenced by the native code - before invoking the native interface. The library and function are referenced by integers/handles instead of actual addresses. The native interface layer will contain the required mapping between library/function handles and real addresses. This makes it impossible to call native functions which does not exist, and also makes support for 64-bit libraries transparent to the Amiga.

Originally Posted by Der Wanderer

I had the idea that lengthy calls could be run in a separate host-thread and the calling AmigaOS task is put to sleep, until the host thread returns.

Yes... that's that Toni said...

Originally Posted by Der Wanderer

However, whatever you guys do, keep it simple.

No matter how you look at it, stuff like this is always going to be a bit difficult. You have to be familiar with compilers and build environments on both Amiga and native side (and be able to build dynamic libraries for one or more platforms). And you need to know about byte order, and struct packing/padding if you pass those. But convenient header files and good examples helps a lot... -And I don't see any reason why it would become more difficult than it already is.

Originally Posted by Der Wanderer

You could push the idea further and patch OpenLibrary in a way that it can open directly DLL or SO.

Most (all?) operating systems use memory mapped files for dynamically loaded libraries. So the library must exist on the host file system. There is of course nothing which prevents a theoretical native interface from allowing the Amiga to temporarily or permanently copy a dynamic library to the host file system (with security implications of course). There are also other possible approaches for code which does not need to use the host dynamic loader to access other native libraries (did you read my previous post?).

Originally Posted by Der Wanderer

Just design it is a way it works for once and for all. If the version number doesn't match, there is anyway not much you can do.

Designing it once and for all "never" works in real life. It is important to be able to add new features without breaking older libraries (but I also think you misunderstood what I meant with the version number in that context. It could be used by a newer library to gracefully support older *UAE versions for example if some feature or functionality is optional).

[Toni Wilen]

Originally Posted by FrodeSolheim

Example C code, Amiga side:

char data[64];
int library = uni_load_library("hello_world_library");
int function = uni_get_function(library, "hello_world");
uni_call_1(function, UNI_SYNCHRONOUS, &data);
/* data now contains the string "Hello, World!" */

This is exactly what I meant and wanted! Real and transparent API, with possibility to validate at least part of parameters, not some "lets pass random registers that goes to specifically prepared x86 code, if even one thing is wrong, everything blows up"

Hmmm...

Patching loadlibrary and friends is ugly (trying to merge too different worlds) but there could be virtual standard amiga library (added by UAE boot rom if virtual calls are allowed) that has above functions, all other "nice to have" helper functions, and helper data in library data area and so on. Also no need for Amiga side assembly code, all Amiga compilers support library calls directly if required header files are included.

Version check would be also become automatic, just increase library version number when needed.

[FrodeSolheim]

@Toni great that we're on the same page, more or less

The param(s) are still sent via registers - up to 6 params in d1..d6, and flags in d7, but the helper functions do not have to use all those. I was thinking to perhaps have a few helper functions, such as uae_call_p (one pointer argument), uae_call_ll (two long arguments) and uae_call_full (can pass all arguments).

I have coded on the feature today, and it is mostly implemented (except that asynchronous calls are not actually implemented yet, and nothing on the "Amiga side"). I have also implemented a compatibility mode - support for the existing native interface in WinUAE, so in theory - if WinUAE merges uni.cpp later, all native interface stuff can be found in one place (and 32-bit FS-UAE for Windows should be able to use existing native dlls unless they depend on WinUAE-specific stuff such as the WinUAE HWND window handle).

What are your thoughts about the bridge between Amiga and UAE? Right now, the existing functionality is hosted in ahi_winuae (which seem to rather be a swiss army knife which quite a bit of responsibility beyond AHI). There is also a commented-out native code execution function in uaelib.cpp.

Should the handlers ideally be put in uaelib ("uaelib_demux") or keep extending "ahi_winuae" (or a new one?) Right now I'm just adding new switch cases to ahi_winuae together with the old functions.

[Der Wanderer]

It would be a good idea if the .dll or .so is NOT aware of being used by UAE.
Means, there should be no glue code on the x86 side. Why? Simply because you may want to use existing libraries that you don't want to contaminate with Amiga specific stuff or you don't even have the source code.

E.g. I wrote a synth.dll that is used by Windows and WinUAE software. On the 68K side, I have a synth.library that simply makes use of the very same .dll that is used for Windows software. Of course there is a pure 68K version of the synth.library too, but it is slower. Unfortunately, you cannot have them installed in parallel. So all environments the AmigaOS install runs on must be able to run this .dll. Alternatively, you could write a mixed-binary 68k library that runs 68K if the .dll is not found or if not running on a .dll capable system. But this is only possible if you own the library.

[Toni Wilen]

I can do the Amiga side library, including some skeleton code that does the important Amiga-side bits.

It is better to be in separate file and it does not need to use demux or similar stuff, it should be built it as a normal Amiga library (or any other resident object), just like uaescsi.device, uae.resource etc.., comes with automatic calltraps, no need for any ugly absolute addresses and non-Amiga like calling methods.

Compatibility code (at least calls to new code) unfortunately have to be in ahi_demux.