How does my static variable survive expunge?

[phx]

I didn't read everything, so just as a side note: I think that Small Data is generally a bad choice for a shared library. You have to save and restore the Small Data base pointer (to the __MERGED section in A4) in every single library function and callback (using

__saveds

). Additionally you steal register A4 from the code generator. This is quite some waste as a good library shouldn't have any static data sections at all. Best practice is to store everything in the library base structure, which also provides you fast base-relative addressing (with A4 still available).
I wouldn't be surprised if this also solves your static variable problems.

Edit: And constant data, like your TagItem structure, should get a

const

attribute, so the compiler can place it into the code section for PC-relative addressing.

[hceline]

Originally Posted by Thomas Richter

This is an indication that the base pointer addressing the __MERGED segment is not setup correctly in the library function that uses the static variable. Is there possibly a __saveds missing?

I tried adding __saveds to new_app_entry(), who the variable is local to, earlier. After finding that was one of few differences I could see between that, and the only other example I could find of a function-local static variable in the library code (See earlier post).

To be on the safe side, I tested again now.
It does not seem to make any difference.

The library was at 084c08a4.
And the variable in question at 0811DA9A again.

I also checked that all library functions that call new_app_entry() has __saveds, and they do.
They are all in the same .c file, and are the previously mentioned L_WB_AddAppMenuItem(), L_WB_AddAppIcon() and L_WB_AddAppWindow().

Attaching disassembly of the file in question with __saveds added to new_app_entry().

Originally Posted by phx

I didn't read everything, so just as a side note: I think that Small Data is generally a bad choice for a shared library. You have to save and restore the Small Data base pointer (to the __MERGED section in A4) in every single library function and callback (using

__saveds

). Additionally you steal register A4 from the code generator. This is quite some waste as a good library shouldn't have any static data sections at all.

You might be right about that.
But as I understand it there is no way to avoid using __saveds when making libraries the "SAS/C way", because it uses the the presence of the combination of __asm and __saveds to identify the functions that should be exported by the library.

Originally Posted by phx

Best practice is to store everything in the library base structure, which also provides you fast base-relative addressing (with A4 still available).
I wouldn't be surprised if this also solves your static variable problems.

Also, since you did not read everything; I did approximately that* to solve it to begin with. This thread is me trying to solve the mystery of why this variable behaved like this in the first place. Mostly to learn from it.

*Edit: For this variable only, not everything. And I must admit that I did not trace back the original allocation of the struct I put it in. It might be a allocated with AllocVec().

Originally Posted by phx

Edit: And constant data, like your TagItem structure, should get a

const

attribute, so the compiler can place it into the code section for PC-relative addressing.

Yea, I do believe my first task after this is to go trough the library code and add const to maybe all the static variables. While I found only on function-local example, there where lots of similar examples outside function scope.

[hceline]

To know for sure the disassembly matched the code I was running I put a MuForce hit after the first KPrinF() in both L_AddScrollBars() and new_app_entry() like this:

    {
        volatile char *a = 0;
        *a = 0;
    }

I found that the code that is run matches the disassembly. And that this is the right diagnostic even with __saveds:

Originally Posted by Thomas Richter

This is an indication that the base pointer addressing the __MERGED segment is not setup correctly in the library function that uses the static variable.

The library was at 08fa568c during this test.
And A4 is 08FA56BC in the hit from L_AddScrollBars(), and 08394EC4 in the hit from new_app_entry().
I attached two files with the first hit from each function.


I am beginning to get an idea of what is happening, even if I do not understand exactly why.
I would have realized sooner if I had just looked at the top of the .c file in question earlier.
This is one of the "different .c files", as I've called them in my head in lacking the means to put a more descriptive term on it.

I encountered them when I was adding library based resource tracking to the code-base. They are usually CreateNewProc() or LoadSeg() but some seem unrelated to these functions (Or maybe I just have not realized the connection yet).
Code in these files would fail to find the library base for my resource tracking library even if it was available everywhere SysBase or ExecBase was.
And I quickly discovered that all these files start with something like:

#define UtilityBase    (wb_data->utility_base)
#define ExecLib        ((struct ExecBase *)*((ULONG *)4))

And I would have to add equivalents for my resource tracking library base, where used.

I have just now come to realize that I can refer to them as the ".c files that are unable to get correct A4 value" or "where __saveds does not really work".

I get a feeling that this issue might be very specifically tied to uncommon things this code-base does.
And I do not expect other people to dig trough the whole library code to find an explanation for me.
So unless someone immediately recognizes this off the top of their head, from my half-explanation, I'll declare the quest for at truth of the immortal variable over.

Yet again, thank you everyone for the help.

[hceline]

A few corrections to my statements in this thread. Replying to myself here, because it feels wrong to edit the original posts after such a long time.

Originally Posted by hceline

I use SAS/C.
You are right, it uses its own _LibInit(), _LibOpen(), _LibClose() and _LibExpunge()

Originally Posted by hceline

In place of lib:libinit.o it uses its own init.o.
But I do not understand why, as I can not see any difference from the original except the removing of code pertaining to Devices and libraries with separate data section for each opener.

After discovering that lib:libinit.o was replaced in my debug build only. I remembered that this change was made by me, to get debug symbols in it.

Originally Posted by hceline

But as I understand it there is no way to avoid using __saveds when making libraries the "SAS/C way", because it uses the the presence of the combination of __asm and __saveds to identify the functions that should be exported by the library.

This is apparently not how it works. As there are several functions in the library in question that are declared with __asm only, but are clearly exported by the library.

Originally Posted by hceline

Not related but since you mention it, I've been wondering. Does KPrintF() from debug.lib break Forbid()?
I've been thinking, it is a IO operation and all IO operations can break Forbid(), so yes?

I seem to have forgot that the statement is usually/always "dos.library I/O functions" not "I/O functions".
Anyway I finally found a statement, in "The Guru Book", to the fact that none of the (d)debug.lib functions break Forbid().

[Thomas Richter]

Originally Posted by hceline

I seem to have forgot that the statement is usually/always "dos.library I/O functions" not "I/O functions".
Anyway I finally found a statement, in "The Guru Book", to the fact that none of the (d)debug.lib functions break Forbid().

They don't, indeed. Actually, they would be not very valuable if they would. Even if they are patched by Sashimi or the syslog module of 3.2.

[hceline]

Originally Posted by Thomas Richter

They don't, indeed. Actually, they would be not very valuable if they would. Even if they are patched by Sashimi or the syslog module of 3.2.

Yea, it sounds logical when you put it that way.
Thank you for confirming this.

In other news, I tried moving on to other things, but my mind would not let this issue go. And I kept coming up with possible explanations as to the "why" of it.
And finally it dawned on me, that this might be due to SetFunction() anyway, albeit indirectly.

So I put back the static variable declaration, the KPrintF() line and the MuForce hit for one final test, to check the contents of A6.
And upon every call to the function i question A6 contained the base address for workbench.library.
Case closed (for real this time).

[aros-sg]

Originally Posted by hceline

The smallest memory loss ...

Some memory loss you see may be fake news. The OS for example may cache cliprects (used to divide layers/windows into visible/hidden parts and for user installed clipping regions). So for example opening and closing a window immediately after may seem like it leaks memory, but it doesn't. It may just have cached cliprects in other windows/layers behind which were created when their visible/hidden areas changed because of your window hiding them (partly). Do check at least twice and compare last 2 results.



avail flush
prog
avail flush
prog
avail flush


Also the console/shell may cache/buffer things for stuff like scrollback if you are using one that features this.

[hceline]

Originally Posted by aros-sg

Some memory loss you see may be fake news. The OS for example may cache cliprects (used to divide layers/windows into visible/hidden parts and for user installed clipping regions). So for example opening and closing a window immediately after may seem like it leaks memory, but it doesn't. It may just have cached cliprects in other windows/layers behind which were created when their visible/hidden areas changed because of your window hiding them (partly). Do check at least twice and compare last 2 results.

Thank you for bringing this to my attention.
Does this also hold true when the windows I open is on a screen that I open and close between the measurements?

Originally Posted by aros-sg

avail flush
prog
avail flush
prog
avail flush

As a matter of fact, if I do what you describe, there is usually* no loss between the state after the second and third FLUSH.
So this might account for some, if not all, of the loss I am seeing.

*Apparently randomly there would still be a loss. And if I continue repeating the steps, it would sometimes lose a little more, sometimes not.
The loss has been varying between around 1k to around at-least 6k of fast ram, and 0 to around 2k of chip.

But then again this was while the library was writhing to memory it did not own.

Originally Posted by aros-sg

Also the console/shell may cache/buffer things for stuff like scrollback if you are using one that features this.

This I've been bitten by before. That is how I learned to use MemLeakZ in place of Avail FLUSH.

[Thomas Richter]

I suggest:

http://aminet.net/dev/debug/MemLog.lha

which can restrict the analysis to a particular task or process and will also provide you the symbols or functions within which the memory was allocated if SegTracker is also installed (recommended).

[hceline]

Originally Posted by hceline

But then again this was while the library was writhing to memory it did not own.

I really should have ran new tests with the non-memory-trashing version, before answering that last post.

But the result is not very different from before.
After a few tests it seems the fast ram losses are a little smaller.
And It seems the the apparent random losses when continuing to repeat the steps are fewer and and require more repetitions to trigger.

Originally Posted by Thomas Richter

I suggest:

http://aminet.net/dev/debug/MemLog.lha

which can restrict the analysis to a particular task or process and will also provide you the symbols or functions within which the memory was allocated if SegTracker is also installed (recommended).

Tried it, and all the other resource trackers I could find on Aminet.
None of them are really usable on this code-base due to it's multi-library design, and it's heavy reliance on CreateNewProc() and friends.
I've had to roll my own by converting one of the link-library based ones to a real Amiga .library.
It works by redefining the functions to itself like:

#define Open(file,mode)         __rtl_Open(file,mode,__FILE__,__LINE__)

It is limited by my knowledge of which functions I should track, and my inability to redefine functions that take "struct TagItem *, ..." as parameters* without ugly workarounds.
And it clearly can not account for memory the OS uses on my behalf.

*As I understand this would require C99.

[hceline]

Originally Posted by Thomas Richter

I suggest:

http://aminet.net/dev/debug/MemLog.lha

which can restrict the analysis to a particular task or process and will also provide you the symbols or functions within which the memory was allocated if SegTracker is also installed (recommended).

Originally Posted by hceline

Tried it, and all the other resource trackers I could find on Aminet.

Correction, I tested most of them. Some I just read the documentation and concluded that they would not work.
I can not guarantee I tried yours, but I believe I did as I had apparently copied it to C: on my test setup.
Anyway, to make sure I did not misstate the facts again, I tried it without supplying TASKNAME or MODULE on the commandline now.
And with a memory loss of 1744 bytes, the resulting file, where I redirected the output, was 747666 lines and ended with:

0x00983E02 bytes required in total, 0x005F92C5 not yet released.

This tool would be helpful in this situation if it had a commandline switch to to not care what task/process did the allocation/de-allocation when called without TASKNAME.
And even more helpful if it was wired up to make use of the nice new ability you added to segtracker of locating functions and line-numbers from debug hunks.
EDIT: I meant to write "function- and source file-names"