Organizing .asm projects

[juice]

Hiya,

Trying to get into Amiga asm on a more serious level, and trying to figure out the best way to structure my projects. I just moved from the "big mess of included files" pattern to using a linker.

I'm using vasm / vlink and the amiga assembly extension for VSCode.

What would be the best way to use a piece of external asm code in my project?
For example a mod replay routine.

The amiga assebly extension just assembles / links every .s file in the project folder to build the .exe ... this would mean I'd have to copy the replayer source into my project... and update the file everytime there's a change / new version.

I'd like to keep the replay routine (and other similar reusable code) in its own folder and just kind of "include" it in my project/workspace as a dependency without actually using an inline include directive.

Is this possible, and what would be the best way to do it?

[juice]

Nevermind, just figured it out myself

You can add multiple folders to the VSCode workspace, and the Amiga Assembly extension will pick up source files from all the folders to asseble...

[JoeJoe]

In my solutions i use includes. My routines i have stored in a file. With defines i decide which functions i want to use from the collection.

Here a small example from a project of mine

main.s

; set include folder as first
INCDIR "../Share/Source"

; Define needed Functions
FS_AAC: SET $1		; AllocAndCopy
...

; Alloc memory: Sin data
move.l    #SinDataQuadrantSize,d1                        ; Size
sub.l     a0,a0                                          ; Source a0=0 > do not copy
lea       (SinDataAdr,PC),a1                             ; (new alloc) Target Address
bsr       AllocAndCopy
tst       d0
beq       ExitProgramm

...

; include functions at last
INCLUDE includes/functions.i

functions.i

; function is not needed as default
IFND FS_AAC
  FS_AAC: SET $0
ENDC
...

; - - - - - - - - - - - - - - - - - - - - - - -
; AllocAndCopy:
;  - allocate memory by given size on D1 and store the pointer to address of A1
;  - if source data A0 set, then copy the content to the allocate memory
;  D1 <  Size of needed memory in bytes
;  A0 <  Source Data
;  A1 <  Target Address to store new alloc memory adr
;  D0 >  $0 =Error
; - - - - - - - - - - - - - - - - - - - - - - -
  IF FS_AAC=1
AllocAndCopy:
    movem.l    d1/a0-a1,-(SP)
    move.l     d1,d0             ; Set size
    move.l     ExecBase,a6
    move.l     #Clear,d1         ;Chip-RAM + clear up
    jsr        AllocMem(a6)      ;memory please
    movem.l    (SP)+,d1/a0-a1
    move.l     d0,(a1)           ; store addess into a1
    beq        .aacend
    cmp.l      #$00,a0           ; Allocate  ONLY!
    beq.s      .aacend
    ; Copy source data to new allocated memory
    move.l     (a1),a1           ; get address behind a1
    subq       #$01,d1
  .aaccopy:
    move.b	(a0)+,(a1)+
    dbra	d1,.aaccopy
  .aacend:

    rts
  ENDC

This has the advantage of adding only the source code that is needed. For example, I don't need the music playback function in all projects.
On the old platforms space and computing time is precious.
You can also use this method to assign a function name multiple times. The desired function is then determined via defines. So i have stored several music playback functions (a fast one only 8 grid lines long, one for NoiseTracker, one for FastTracker and a big one that beats all of them) and the call in the main program remains the same.

Man könnte natürlich auch für jede Funktion eine Includefile erstellen, wenn man will.

[meynaf]

Originally Posted by JoeJoe

In my solutions i use includes. My routines i have stored in a file. With defines i decide which functions i want to use from the collection.

You're not alone doing something like that.

Originally Posted by JoeJoe

This has the advantage of adding only the source code that is needed. For example, I don't need the music playback function in all projects.
On the old platforms space and computing time is precious.

You could perhaps go one step further. Instead of defining by hand what you intend to use, you could include another file that will define a macro. Then you use that macro instead of jsr to call the routine : the macro calls the routine of course but it also sets the define and now you can call everything without any special care. If it's used, it's there.
After that it's easy in the include to generate a call to some init function that will open all the necessary libraries, devices, etc (i also perform automatic cleanup but it's a little more tricky).

Originally Posted by JoeJoe

You can also use this method to assign a function name multiple times. The desired function is then determined via defines. So i have stored several music playback functions (a fast one only 8 grid lines long, one for NoiseTracker, one for FastTracker and a big one that beats all of them) and the call in the main program remains the same.

Sometimes features available in a reusable routine are not all needed and this fact can be used to make it simpler. I use a macro for this too.

[phx]

Originally Posted by JoeJoe

In my solutions i use includes. My routines i have stored in a file. With defines i decide which functions i want to use from the collection.

The only advantage in such an approach is that the assembler sees all code at once and might do some optimizations which the linker cannot do. But it's also a bit complicated and messy.

When you save your routines as single object files and put those into a linker library, the linker would automatically pull only those objects into the final executable where you referenced a symbol from. No defines or includes needed. Just "xref externalfunction".

[meynaf]

Originally Posted by phx

The only advantage in such an approach is that the assembler sees all code at once and might do some optimizations which the linker cannot do. But it's also a bit complicated and messy.

Why would this be complicated and messy ?
The one and only argument against this, is that the assembler will always have to reassemble everything, using unnecessary time. But considering (in my case) the performance of Phxass under Winuae Jit, this argument is moot.

Originally Posted by phx

When you save your routines as single object files and put those into a linker library, the linker would automatically pull only those objects into the final executable where you referenced a symbol from. No defines or includes needed. Just "xref externalfunction".

If a function uses another, the linker will probably not be able to see it and will keep unneeded code. Or you have to alter your makefile for every function you use and wasn't there before (eventually leading to a depency hell).

In any case, i would go from 2 files to 200+ (100+ functions, with sources and objects). THAT would be complicated and messy (and, in fact, simply impossible because my startup/cleanup code wouldn't know what to do).

And as said, included code can be adaptative. My screen opener for example needs to know what is used, otherwise it will support ST screens, C2P and 24-bit (HAM8) even where it's of no use.

[phx]

Originally Posted by meynaf

Why would this be complicated and messy ?

Because you have to construct something with includes, conditional assembly and macros in your source texts which the linker can do automatically. It would also improve readability.

The one and only argument against this, is that the assembler will always have to reassemble everything, using unnecessary time.

True, but I didn't mention that, as you said yourself, only few people still assemble on a real Amiga.

If a function uses another, the linker will probably not be able to see it and will keep unneeded code. Or you have to alter your makefile for every function you use and wasn't there before (eventually leading to a depency hell).

Sorry, but I don't understand what you mean. The linker sees all symbol references. Care to elaborate and give an example?

[meynaf]

Originally Posted by phx

Because you have to construct something with includes, conditional assembly and macros in your source texts which the linker can do automatically. It would also improve readability.

Every function has ifd/endc to add it or not. An include is similar to adding a file in a project, it's just in the source rather than in some makefile.
There is nothing complicated in here.

Of course one might want to do something more complex.
My startup/cleanup code for example is automatic, depending on what functions are used (yes i have full resource tracking ). This makes the code not very readable indeed, but guess what : the linker just can't do this.
I can also optimize some parts if a feature isn't used, f.e. my copymem is a true memmove, but it can be configured as mere memcpy to make code shorter.

Remember that the linker can only remove whole functions (actually whole sources !), it can not alter code in a function to remove or rewrite a small part of it.

Furthermore, an include can provide lots of useful macros to circumvent missing features of 68k. I can't live without mine.

Originally Posted by phx

Sorry, but I don't understand what you mean. The linker sees all symbol references. Care to elaborate and give an example?

If function A calls function B, will it be able to remove them both or will it keep function B because it has been referenced ?

Let's admit it can remove everything that's not really used, the project still has to include all these object files. And if some function is called but the coder forgot to add it to his project, the linker won't fetch it for him. Of course he could just add all functions in the project but adding a hundred files to all projects does not look like a good idea.

[phx]

Originally Posted by meynaf

Every function has ifd/endc to add it or not. An include is similar to adding a file in a project, it's just in the source

Ok. So the include directive would be comparable to an xref.

My startup/cleanup code for example is automatic, depending on what functions are used (yes i have full resource tracking ). This makes the code not very readable indeed, but guess what : the linker just can't do this.

Don't be so sure about it.
Linkers support automatic constructor/destructor handling. For example vlink recognises vbcc-style constructors starting with __INIT.. and destructors starting with __EXIT... (SAS/C-style and gcc-style are also supported). It even builds a list of constructor and destructor pointers for you, to call all these function on program startup and exit. These lists can be referenced by __CTOR_LIST__ and __DTOR_LIST__.

So when you have an object file with a function which needs some initialisation on startup you would just add an __INIT_myfunc function to this same object.

I can also optimize some parts if a feature isn't used, f.e. my copymem is a true memmove, but it can be configured as mere memcpy to make code shorter.

This is usually done by putting different versions of a function into your linker-library. It doesn't matter how big your linker-library is, as only those objects you really need (reference) will be in the output.

Remember that the linker can only remove whole functions (actually whole sources !),

Usually linkers operate on the object file level and only load an object from a linker library when one of its symbols was referenced. Object files which are not part of a library are always in the ouput.

But some linkers (including vlink) can also do section garbage collection. Which works by tracing all symbol references from the starting point to find out which sections are referenced by the program flow. All other sections will be removed. FreePascal uses this vlink-feature.

it can not alter code in a function to remove or rewrite a small part of it.

Some linkers do that. They can even optimise and rearrange code like an assembler when a resolved reference makes a smaller instruction possible (no, vlink doesn't, yet).

Furthermore, an include can provide lots of useful macros to circumvent missing features of 68k. I can't live without mine.

Absolutely. Includes are still useful for many other things.

If function A calls function B, will it be able to remove them both or will it keep function B because it has been referenced ?

When the object wth function A is removed from the linking process then there is no longer a reference to function B, so there is also no need to load function B's object file.

Let's admit it can remove everything that's not really used, the project still has to include all these object files.

No.
I think we reached the core of your problem here. You think you have to specify all single object files manually on the command line and also remove those manually which you don't need?

No, you won't do that. You will put all these object files in a linker library(!). For Amiga hunk-format a linker library is as simple as

join #?.o as my.lib

Then link your current project with "-lmy" and the linker will automatically pick only those objects you need.

And if some function is called but the coder forgot to add it to his project, the linker won't fetch it for him. Of course he could just add all functions in the project but adding a hundred files to all projects does not look like a good idea.

Nonono. See above.

[meynaf]

Originally Posted by phx

Ok. So the include directive would be comparable to an xref.

Not really ! Single include, many xref.
I don't want to have bunches of xref in source code. It has to be automatic, so the xref would have to be... in an include.

Originally Posted by phx

Don't be so sure about it.
Linkers support automatic constructor/destructor handling. For example vlink recognises vbcc-style constructors starting with __INIT.. and destructors starting with __EXIT... (SAS/C-style and gcc-style are also supported). It even builds a list of constructor and destructor pointers for you, to call all these function on program startup and exit. These lists can be referenced by __CTOR_LIST__ and __DTOR_LIST__.

So when you have an object file with a function which needs some initialisation on startup you would just add an __INIT_myfunc function to this same object.

It's not that simple
This is for individual functions. Usually some library will be used by several of them, but needs to be opened only once.

Are these CTORs allowed to FAIL btw ? I mean, can they prevent other CTORs to be called and trigger a program exit (that will call all relevant DTORs but not the ones related to unitialized things) ?
With single init routine full of ifd/endc it's very easy. How do you do that with local-to-function init code ?

Anyway, having many small routines instead of a single big one isn't very optimal - and not so easy to call in the right order.

Originally Posted by phx

This is usually done by putting different versions of a function into your linker-library. It doesn't matter how big your linker-library is, as only those objects you really need (reference) will be in the output.

Can the linker always choose the right one ? Sometimes it's just 68000 vs 020+ difference and this is easily made automatic with if/endc (with nothing at all to be defined in the program itself 'xcept choosing cpu type once and for all).

Anyway, that means there will be different versions to maintain. There can be a feature disabled, some 68k vs 020 small change, etc. If several unrelated differences occcur in same function there can be many versions of it.
Several versions mean either copy-paste of same function in different places (yuck !) or includes with macros and if/endc (i.e. the exact thing you wanted to avoid).

Originally Posted by phx

Usually linkers operate on the object file level and only load an object from a linker library when one of its symbols was referenced. Object files which are not part of a library are always in the ouput.

But some linkers (including vlink) can also do section garbage collection. Which works by tracing all symbol references from the starting point to find out which sections are referenced by the program flow. All other sections will be removed. FreePascal uses this vlink-feature.

So we need a pretty good linker.
It also needs to be able to tell the coder the source line at which an unresolved reference occurs (in case a nonexisting function is called). Last time i used a linker (Phxlnk) it didn't, but things might have changed.

Originally Posted by phx

Some linkers do that. They can even optimise and rearrange code like an assembler when a resolved reference makes a smaller instruction possible (no, vlink doesn't, yet).

Seems we're very dependent on the quality of the linker

In addition, the main problem here is that it forces the coder to use a linker, leading to several commands to type instead of one (or use scripts, etc). That for zero benefit from the library user's point of view.
This is missing the point of easy-to-use library code, which is to make coder's life easier.

Originally Posted by phx

Absolutely. Includes are still useful for many other things.

Yes but - now that an include is there, why not just use it at its fullest ?

Originally Posted by phx

No.
I think we reached the core of your problem here. You think you have to specify all single object files manually on the command line and also remove those manually which you don't need?

No, you won't do that. You will put all these object files in a linker library(!). For Amiga hunk-format a linker library is as simple as

join #?.o as my.lib

Then link your current project with "-lmy" and the linker will automatically pick only those objects you need.

My problem ? I don't have any problem. My system works fine

But a change to a function is for me altering a single source. With your method i would in addition have to reassemble the function (multiple times if there are different versions of it like you suggested above), then rebuild the linker library. Not very convenient for debugging.

Originally Posted by phx

Nonono. See above.

Ok, ok
But we still end up with a gazillion files on the HD.

[Hedeon]

Originally Posted by phx

True, but I didn't mention that, as you said yourself, only few people still assemble on a real Amiga.

Why? Assembly is quite fast on Amiga :-)

[JoeJoe]

So the method with the defines works very well on real Amiga

[phx]

Originally Posted by meynaf

Are these CTORs allowed to FAIL btw ? I mean, can they prevent other CTORs to be called and trigger a program exit (that will call all relevant DTORs but not the ones related to unitialized things) ?

Sure. It depends on your startup-code to call the constructors and handle their return-code. The linker only provides you with the symbol __CTOR_LIST__ (and __DTOR_LIST__) to reference the function-pointer lists.

How do you do that with local-to-function init code ?

I don't understand what a "local-to-function init code" is.

Anyway, having many small routines instead of a single big one isn't very optimal - and not so easy to call in the right order.

I didn't mention it to keep it simple, but the constructor/destructor function names may also include a priority. For example the pointer to __INIT_1_funcA will be in the list before the pointer to __INIT_9_funcB.

Several versions mean either copy-paste of same function in different places (yuck !) or includes with macros and if/endc (i.e. the exact thing you wanted to avoid).

Why would I want to avoid includes and conditional assembly in situations where it is the best option?

What I am doing is to make a single source with conditional assembly for different CPUs or features and assemble the same source with different defines into different object files using a Makefile. So when I change something, a simple "make" rebuilds all objects and also the linker library with them.

So we need a pretty good linker.
It also needs to be able to tell the coder the source line at which an unresolved reference occurs

You know that it cannot work, because this information is lost. It may be possible when including source-level debugging info.

In addition, the main problem here is that it forces the coder to use a linker,

Strange argument. I could also say that your solution forces you to include all your giant blob every time.

leading to several commands to type instead of one (or use scripts, etc).

Of course you would use a Makefile for bigger projects. But linking your source with "myobj.o -lmylib" is not so much to type IMHO.

That for zero benefit from the library user's point of view.

That's nonsense and everybody can see it. Saying that your approach has zero benefit would be as stupid, so I don't. When there is no benefit then there wouldn't be so many developers using libraries. Maybe the benefit is not important for you? That's ok.

My problem ? I don't have any problem. My system works fine

It is not my intention to convince you to change your development style. You have put a lot of work in your environment and it works well for you. Then all is fine.

I just wanted to mention that there are alternatives to a huge include-block, because the original post asked for it.

[meynaf]

Originally Posted by phx

Sure. It depends on your startup-code to call the constructors and handle their return-code. The linker only provides you with the symbol __CTOR_LIST__ (and __DTOR_LIST__) to reference the function-pointer lists.

So i have to do the work anyway. Better do everything at the same place then.

Originally Posted by phx

I don't understand what a "local-to-function init code" is.

One init routine for one library function, so that the init routine is somehow local to that function.
You didn't say how i can use this method to open dos.library only once, btw.

Originally Posted by phx

I didn't mention it to keep it simple, but the constructor/destructor function names may also include a priority. For example the pointer to __INIT_1_funcA will be in the list before the pointer to __INIT_9_funcB.

In which ways would this have any advantage over a single routine doing everything in an order that's obvious by just reading the code ?

Originally Posted by phx

Why would I want to avoid includes and conditional assembly in situations where it is the best option?

I don't know. There is indeed no reason for that. But it is the solution you are defending here.

Originally Posted by phx

What I am doing is to make a single source with conditional assembly for different CPUs or features and assemble the same source with different defines into different object files using a Makefile. So when I change something, a simple "make" rebuilds all objects and also the linker library with them.

So we are doing the same thing. Only difference is that you have unneeded extra steps and many more files.
Be honest : is a pair of ifd/endc added to every routine that much to handle ?

Originally Posted by phx

You know that it cannot work, because this information is lost. It may be possible when including source-level debugging info.

Right, it may be possible. Guess what, with the asm doing all the work this information is presented right away.

Originally Posted by phx

Strange argument. I could also say that your solution forces you to include all your giant blob every time.

You could, but it's not the same thing. My way only needs an assembler to be used, something the user would have anyway.
Yours on the other hand doesn't work without a linker.

Btw. My "giant blob" is probably smaller than what the equivalent linker library would be and is just 170kb, wow so big.
Oh, and of course it can perfectly use sub-includes not there unless used, so it's not fully included every time.

Originally Posted by phx

Of course you would use a Makefile for bigger projects. But linking your source with "myobj.o -lmylib" is not so much to type IMHO.

Not that much to type, except when you discover it's not found because not in the source directory so a long path has to be added...
You also forget the command to assemble the source. All that together is single, small command for me.

Originally Posted by phx

That's nonsense and everybody can see it.

Am I starting to pick on your nerves ?

Originally Posted by phx

Saying that your approach has zero benefit would be as stupid, so I don't.

The benefits of my approach are listed throughout this thread. You can get a short list below, shortcomings of one are advantages of the other. Where's your list already ?

Originally Posted by phx

When there is no benefit then there wouldn't be so many developers using libraries. Maybe the benefit is not important for you? That's ok.

The fact many people do something doesn't make it smarter, but anyway, these "so many developers" using libraries are mostly not doing 100% asm projects so the situation is very different.

You're saying there is benefit ? But where is it ? I only see shortcomings :
- more steps to build the project
- lots and lots of files
- requires a linker (and a good one)
- imprecise error messages (you know, that missing line number)
- init/exit code not very manageable (because order isn't obvious)
- suboptimal code (in comparison)
Do you have such a list ? If so I want to see it.

Originally Posted by phx

It is not my intention to convince you to change your development style. You have put a lot of work in your environment and it works well for you. Then all is fine.

I just wanted to mention that there are alternatives to a huge include-block, because the original post asked for it.

This isn't what happened. This has nothing to do with just mentioning an alternative : what you wrote is that the include block is complicated and messy, strongly suggesting your solution is best.
It was probably not the intent, but it is equivalent to jumping in here saying : hey your solution is just BS, i've something better. You see ?