01 July 2024, 02:42 | #1 |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
How does my static variable survive expunge?
I got a library that does SetFunction() on AddAppMenuItem(), AddAppIcon(), and AddAppWindow().
They in-turn call another function in the same .c file that adds the Menus/Icons/Windows to a tracking-list. In this process it increments its own lib_OpenCnt with one if it finds the tracking-list empty to begin with. A cleanup function elsewhere removes items from the tracking-list and decrements the lib_OpenCnt when the list gets empty. But sometimes the tracking-list would be emptied without the involvement of said cleanup function, which lead to a runaway lib_OpenCnt. So I had to limit this to only increment lib_OpenCnt in the first call to this function. I made a function-local static BOOL to identify the first call to this function. Today it came back and bit me..... Apparently this variable survives the library being expunged (including the original workbench.library functions being successfully SetFunctioned back) and opened again. I fixed it by placing the variable in a struct that gets allocated in __UserLibInit(), and also set its initial value there. But I am still left with the question, how did the variable survive? I've been reading up on the little I could find on SetFunction(), and searching trough the library code trying to figure out this, but I'm at a loss to what I'm even looking for. Is this all SetFunctions doing? In that case, how does it know of this function at all? If not SetFunction, what can do this, there are no tasks or residents accounting for keeping it alive trough a expunge. The code in question is here: https://github.com/Hagbard-Celin/gal...e/Library/wb.c Since the function in question is riddled with #ifdef/KPrintF() and thus quite hard to read, I'm pasting in a cleaned up version here: Code:
AppEntry *new_app_entry( ULONG type, ULONG id, ULONG userdata, APTR object, char *text, struct MsgPort *port, WB_Data *wb_data) { AppEntry *entry; // This is the immortal variable in question static BOOL dejaVu=FALSE; // Allocate new entry if (!(entry=AllocVec(sizeof(AppEntry),MEMF_CLEAR))) return 0; // Fill out AppEntry entry->type=type; entry->id=id; entry->userdata=userdata; entry->text=text; entry->port=port; NewList((struct List *)&entry->menu); // AppIcon? if (type==APP_ICON) { struct DiskObject *icon_copy; // Copy icon if (icon_copy=L_CopyDiskObject((struct DiskObject *)object,0,wb_data->galileofm_base)) { // Use copy entry->object=icon_copy; entry->flags|=APPENTF_ICON_COPY; } // Couldn't copy else { FreeVec(entry); return 0; } } // Otherwise, save object pointer else entry->object=object; // Lock patch list L_GetSemaphore(&wb_data->patch_lock,SEMF_EXCLUSIVE,0); // Is this the first entry? if ((IsListEmpty((struct List *)&wb_data->app_list))&&(!(dejaVu))) { dejaVu=TRUE; // Bump library open count so we won't get expunged ++wb_data->galileofm_base->ml_Lib.lib_OpenCnt; } // Add to list AddTail((struct List *)&wb_data->app_list,(struct Node *)entry); // Unlock list L_FreeSemaphore(&wb_data->patch_lock); return entry; } |
01 July 2024, 08:57 | #2 |
Registered User
Join Date: Jul 2017
Location: San Jose
Posts: 686
|
I think DOpus is using its own libinit code, which is responsible for initializing static variables. If the library was closed and reopened in the same place and on top the static initializers are not executed, you’ll end up seeing the same values.
If you’re using the GCC/libnix combo, the DOpus libinit code must call the relevant initialization functions in libnix. |
01 July 2024, 10:11 | #3 |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
I use SAS/C.
You are right, it uses its own _LibInit(), _LibOpen(), _LibClose() and _LibExpunge(), and it almost certainly was loaded again at the same location. But, with the caveat that I do not yet fully grasp the library init code, I can not see that those are substantially different form the originals in sc:source/libinit.c. Edit: After doing a diff, I can conclude that he only difference is removing #ifdef/#ifndef for DEVICE and ONE_GLOBAL_SECTION. Diff file attached. Edit 2: Not quite only, it removes calls to __libfpinit()/__libfpterm() too. Edit3: Please ignore Edit2, I lost track of the #ifdef/#ifndef - #endif count, that was also part of a "#ifndef ONE_GLOBAL_SECTION". Last edited by hceline; 03 July 2024 at 02:49. |
01 July 2024, 13:33 | #4 | |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,388
|
Quote:
The original SetFunction only patches a function entry, and that is it. However, depending on what is installed in your system, it may do a bit more. SaferPatches will (in some incarnations) create a trampoline function for the patched function such that patches can be removed in an order that is different from the inverse installation order, and it will also patch the expunge vector to learn when it has to release the trampoline functions. But that works according to the specs, i.e. it only releases trampolines if the library has been really removed. This has some implications on how SetFunction() has to be called, thus for example, to remove a patch, you shall call SetFunction again with the vector returned from the first call (which is a plausible way of assuming how things work), and not with a vector the the patching program obtained by peeking the library base itself before attempting the patch. Also, there are plenty of pitfalls in using semaphores in LibOpen and LibClose. Shortest version of the story: Don't. You'll break things easily, unless you really really know what you are doing. Don't Wait() in LibOpen and LibClose(), don't break the Forbid(). If there is anything asynchronously to be done, do it in LibInit() as part of the ramlib process, there nothing bad can happen. Even more so, don't break Forbid() in Expunge(), avoid, avoid. Again, unless you totally know what you are doing. I'm saying, because the Os 3.9 boopsi classes were full of naive library implementations that had races. |
|
02 July 2024, 10:30 | #5 | |||
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Quote:
I must admit that I never checked the return value of Expunge(). But I did check that the library was no longer on the system library list in ShowConfig/ARTM. That is the same, is it not? Quote:
The only thing I've added is SegTracker, PatchWork, MuForce and MuGuardianAngel that runs in some combination, or not, depending on what mouse-buttons I hold down during boot. Disabling them makes no difference in this case. Anyway, this confirms that the issue is not related to SetFunction() itself. Edit: I also got latest version of P96 in the test setups. I slipped my mind, when typing the above. Probably because it could not possibly be related. Also for completeness I might add that I've added disassembler.library to the standard mmu.library install. (I believe that is not done by the standard installer.) I am beginning to wonder if the SAS/C Global Optimizer might in some way be the culprit. Going off to try compiling the library with NoOptimize. Quote:
Not related but since you mention it, I've been wondering. Does KPrintF() from debug.lib break Forbid()? I've been thinking, it is a IO operation and all IO operations can break Forbid(), so yes? But, I never seen any warning about KPrintF() and Forbid() anywhere. If it did break it, there would have been a warning somewhere, as it is a function one might likely put anywhere in the code. And the autodoc only says for KPutChar() that it blocks until completion, so no? Last edited by hceline; 02 July 2024 at 13:32. Reason: Slipping memory. |
|||
02 July 2024, 14:24 | #6 | |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Quote:
I also checked the free memory before first launch of the main exe after doing a FLUSH with MemLeakZ, against the state after quitting and doing another FLUSH. There is a loss, but it fluctuates, and most of the time it's too small for it to contain the library code. The lowest loss I got was 2264 bytes lost. With a library "Maximum code size" as given by slink at 218036 bytes. There where 26 libraries open before the test according to ARTM, an 26 open after. Still when starting the exe again my immortal variable retained the TRUE value from first run. And was located at exactly the same address as in first run. Last edited by hceline; 02 July 2024 at 14:32. Reason: Pasted the wrong library code size |
|
02 July 2024, 15:07 | #7 |
son of 68k
Join Date: Nov 2007
Location: Lyon / France
Age: 51
Posts: 5,377
|
I suggest you try putting another value than FALSE in your immortal variable's initializer and check what's there the very first time your app entry is called.
|
02 July 2024, 15:41 | #8 | |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Quote:
I changed the "FALSE" to "42" and upon first call to new_app_entry() it is "0". All subsequent calls give "1" as before. I assume that this means pipper was right about static initializers not being executed. If that is the case, how does one fix it? |
|
02 July 2024, 16:23 | #9 |
son of 68k
Join Date: Nov 2007
Location: Lyon / France
Age: 51
Posts: 5,377
|
I would fix that by not using static initializers at all.
As you're in a library, you should have some kind of library base somewhere. In the library struct, the neg area is for function calls, the pos area is for library data : i'd put the BOOL there. |
02 July 2024, 16:54 | #10 | ||
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Quote:
Quote:
In boopsi.c:L_AddScrollBars() starting at line 110. It begins like this: Code:
struct Gadget *__asm __saveds L_AddScrollBars( register __a0 struct Window *window, register __a1 struct List *list, register __a2 struct DrawInfo *draw_info, register __d0 short flags) { struct Image *image[4]={0,0,0,0}; struct Gadget *gadget=0; short a; static struct TagItem map_tags[]={ {PGA_Top,ICSPECIAL_CODE}, {TAG_END}}; Code:
KPrintF("!!!boopsi.c line: %ld BEGIN map_tags[0]: %lx at %lx\n", __LINE__, map_tags[0].ti_Tag, &map_tags[0].ti_Tag); KPrintF("!!!boopsi.c line: %ld BEGIN map_tags[1]: %lx at %lx\n", __LINE__, map_tags[0].ti_Data, &map_tags[0].ti_Data); KPrintF("!!!boopsi.c line: %ld BEGIN map_tags[2]: %lx at %lx\n", __LINE__, map_tags[1].ti_Tag, &map_tags[1].ti_Tag); Code:
!!!boopsi.c line: 124 BEGIN map_tags[0]: 80031009 at 8FB73B0 !!!boopsi.c line: 125 BEGIN map_tags[1]: 80040003 at 8FB73B4 !!!boopsi.c line: 126 BEGIN map_tags[2]: 0 at 8FB73B8 And after quitting, doing FLUSH and restarting: Code:
!!!boopsi.c line: 124 BEGIN map_tags[0]: 80031009 at 8FB6F60 !!!boopsi.c line: 125 BEGIN map_tags[1]: 80040003 at 8FB6F64 !!!boopsi.c line: 126 BEGIN map_tags[2]: 0 at 8FB6F68 The only* difference I could see between these functions being the one that worked having "__asm __saveds", I tried adding __saveds to new_app_entry(). No dice. Testing with "42" still in the initializer, it still starts out at "0", and then "1" on every subsequent call regardless. I'm even more confused now. Why would the static initializers work in L_AddScrollBars(), and not in new_app_entry()? *Except that L_AddScrollBars() does not change map_tags after initialization. Last edited by hceline; 02 July 2024 at 17:04. |
||
02 July 2024, 17:01 | #11 |
Registered User
Join Date: Jul 2017
Location: San Jose
Posts: 686
|
Try "avail flush" in the shell to truly get rid of libraries after they have been closed to 0 users.
I suggest disassembling the code and find out where the variable is stored. The compiler may decide to place it into the code segment and actually have the code segment contain the initial value. But this way the variable could only be restored upon reloading the entire code segment. It would not be enough to close the library, expunge it and reopening it. The compiler may decide to put it into a data segment. But then again, closing the lib and reopening would not reset the value. Zero-initialized global variables can also be stored in a bss segment. Libnix (IDK if that is what SAS/C is using as libc?) is at least attempting to restore the bss segment to all-zero: https://github.com/bebbo/libnix/blob...libinit.c#L175 But you only get this behavior if youre actually using Libnix' startup code... which DOpus does not. I think, static variables have undefined semantics in the context of Amiga libraries and thus should be avoided. Are we expecting one copy of each static variable per library user ( I think Libnix has provisions to do that called "Multibase Library" and the startup code is in https://github.com/bebbo/libnix/blob...tup/libinitr.c )? Or is a static considered global across all processes using the library? |
02 July 2024, 17:23 | #12 |
son of 68k
Join Date: Nov 2007
Location: Lyon / France
Age: 51
Posts: 5,377
|
It is possible that the compiler found out that map_tags is never modified and treated it as constant data - which needs no init code at all.
In that case, it works by accident... Normally, this should be declared as static const. |
02 July 2024, 17:53 | #13 | ||||||
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Quote:
I've verified with ARTM/SysInfo that the library is really gone from the system library-list. And I assume this means all the memory belonging to the library should have been freed. Quote:
1 a clean boot 2 then a FLUSH 3 run the main exe and quit it 4 and then another FLUSH and checking that the system library list after 2 was equal to the library list after 4, .. is 2264 bytes. And given that the "Maximum code size" as given by slink for the library was at the time 218036 bytes, I assume the library code is freed, and must be reloaded on next OpenLibrary(). Quote:
Quote:
Especially if this holds true: Quote:
Quote:
Thank you all so much for bearing with me. |
||||||
03 July 2024, 00:05 | #14 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,388
|
Concerning static initializers: How did you compile your source? Because depending on the options, SAS/C will make all non-automatic objects part of the library base, and such objects whose lifetime starts at the point the library is initialized, and whose life time ends when the library is expunged. Please study the SAS/C manual concerning the LIBCODE command line object.
|
03 July 2024, 00:13 | #15 | ||||
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,388
|
Quote:
Quote:
What exactly is right depends on your application. Quote:
Quote:
Even the way how the Amiga system libraries are compiled is not consistent. Some use LIBCODE, others do not, and some use some really dirty tricks to access data relative to the library base even without using LIBBASE. I would need to look up, but I think there is example library code somehwere in SAS/C which may help. |
||||
03 July 2024, 02:36 | #16 | ||
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Quote:
The relevant section from the smakefile is: Code:
$(LBIN)galileofm.library: $(OBJS) slink with lib:utillib.with with << define __ctype=___ctype libprefix _L_ libfd galileofm_lib.fd $(ADDSYM) from lib:libent.o $(OBJS) to $(LBIN)galileofm.library lib $(RTLIB) lib:sc.lib lib:pools.lib lib:amiga.lib $(DEBUGLIB) lib:asyncio.lib libversion $(LIBVER) librevision $(LIBREV) $(STRIPDEBUG) noicons maxhunk 51200 sc sd < But I do not understand why, as I can not see any difference from the original except the removing of code pertaining to Devices and libraries with separate data section for each opener. Feel free to "check my work" with the diff file attached in my second post in this thread. I should probably just try changing this to lib:libinit.o. Anyway the first line of $(OBJS) is: Code:
OBJS = $(OBJ)init.o $(OBJ)libinit.o $(OBJ)data.o $(OBJ)string_data.o \ Quote:
But I am unable to extract from that an answer to the mystery of the immortal variable. My complete library scoptions: Code:
DEBUG=SYMBOLFLUSH NOSTACKCHECK ERRORREXX OPTIMIZE LISTMACROS NOLISTNARROW NOLISTHEADERS XREFERENCEHEADERS NOVERSION LIBRARYCODE UTILITYLIBRARY NOICONS MEMORYSIZE=HUGE MULTIPLECHARACTERCONSTANTS STRINGSECTION=CODE DEFINE _DEBUG_LISTERMEM DEFINE RESOURCE_TRACKING=1 GLOBALSYMBOLTABLE=include:all.gst |
||
03 July 2024, 11:46 | #17 |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
So, while searching the SAS/C manual for clues last night I was reminded it can disassemble the relevant object files while compiling, with no fuss about it.
So I thought, why not. I am not very fluent in assembler, but here is what I managed to learn: Both new_app_entry() and L_AddScrollBars() puts their respective function-local static variable in __MERGED, which is the "Near data hunk" with my compiler options according to manual. The only difference in the disassembly when changing the initial value in the immortal variable from FALSE to 0x42 is the value that appears in __MERGED, the code is the same either way. Edit2: The "static struct TagItem map_tags[]" in L_AddScrollBars() does not get converted to "static const struct TagItem map_tags[]" by compiler. That would have put it in text segment with my compiler options according to manual. I also put a KPrintF() in _LibExpunge() (still not quite sure that is safe) right above return outputting the return value. It did not return zero. I also just recently notices the nice little save button in ARTM upper right corner. So I ran a new test to document everything in case somebody more fluent in assembler would care to look at it. (Edit: this was done with the wb.c where initial value was changed for FALSE to 0x42) 1 Did a clean boot. 2 Opened a shell. Ran "copy ENVARC: all clone ENV:" and closed the shell. (To avoid memory usage for ram to change during test) 3 Opened and closed SysInfo (If I do not do this I usually get 26 libs at step 6 and 27 at step 10, which is not good for tracking memory usage) 4 Opened ARTM. 5 Opened MemLeakZ and did FLUSH. 6 Saved Library list from ARTM. (See ARTM_1before.output.txt) 7 Started Galileo main exe, and quit it. 8 Saved Library list from ARTM. (See ARTM_2After_exit_before_FLUSH.output.txt) 9 Did a FLUSH 10 Saved Library list from ARTM. (See ARTM_3after.output.txt) (At this point the memory loss compared to step 6 was 1792 bytes) Edit3: Also ARTM_3after.output.txt is identical to ARTM_1before.output.txt from step 6. 11 Started Galileo main exe 12 Saved Library list from ARTM. (See ARTM_4after_2nd_start.output.txt) The complete KPrintF() output of this test sequence is in Serial_KPrintF_Output.txt I had to zip the disassemblies due to .txt file size limits. Last edited by hceline; 03 July 2024 at 13:40. |
04 July 2024, 00:19 | #18 |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
I found another clue.
I remembered something that did not quite register the first time I compared behavior of local static variable in L_AddScrollBars() and new_app_entry(). The addresses they where located at seemed to far from each-other. So I put back the KPrintF() statements in boopsi.c (with correct name this time) and ran another test. As opposed to earlier the variable in L_AddScrollBars() kept it's position in memory after library flush and re-open. But then again i did not really check if the library loaded at exact same address during the previous test. Which it did this time. And what I found that while the "static struct TagItem map_tags[]" from L_AddScrollBars() is in the near data section right above/after libbase, the "static BOOL dejaVu" in new_app_entry() most certainly is not. It is actually at a lower address approx 3mb away. And this with both variables in __MERGED according to the disassembly. These are the addresses I found. Code:
Library base at 0845c944 dejaVu at 0811DA9A map_tags[0].ti_Tag at 0845CC50 |
04 July 2024, 01:08 | #19 |
Registered User
Join Date: Nov 2009
Location: Top of the world
Posts: 181
|
Was trying to figure out if the location of my immortal variable matched up with some of the memory I loose between first start of exe and library flush.
And what I found made me hope that someone will tell me that my method was flawed. I used ARTM to save out a list of free memory chunks. Did the start, exit and flush routine. Lost 1320 bytes in the process. Used ARTM to save out the list of free memory chunks again. This time my variable was at address 0811DA9A. End the closest free chunks was before the test: At 080bc1c8 size of 8, and at 08120760 size of 16. And after the FLUSH: At 080bc1c8 size of 8, and at 0812e2f0 size of 40. If I understand this correctly, my variable claims to be in memory that could never have belonged to my library. Because the location was not on the free memory list when the library first loaded. Attaching ARTM and KPrintF() outputs. No recompile since last test. Last edited by hceline; 04 July 2024 at 01:32. Reason: Spelling |
04 July 2024, 10:11 | #20 |
Registered User
Join Date: Jan 2019
Location: Germany
Posts: 3,388
|
This is an indication that the base pointer addressing the __MERGED segment is not setup correctly in the library function that uses the static variable. Is there possibly a __saveds missing?
|
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Did LEFT-Amiga M survive? | nolunchman | Amiga scene | 6 | 12 November 2018 05:25 |
Static adres | jarre | Coders. General | 5 | 24 September 2018 11:59 |
Helping Amiga love survive | elronnightshade | support.Hardware | 19 | 08 June 2010 15:21 |
Help me survive in DM - Chaos Strikes Back | NewDeli | support.Games | 18 | 19 August 2009 16:07 |
|
|