[SOLVED] GCC (g++) Memory allocation issues

[tygre]

Yes, I am running THTTPd with Enforcer in WinUAE. My problem is really that the crash happens kind of randomly, at least to my uneducated eyes

I will try on my Classic Amiga, there, THTTPd seems to crash "faster" and more regularly...

Edit: Actually, I was mistaken (no surprise here! ): in WinUAE too does Enforcer report violations: when running Enforcer FILE=CON: and then thttpd, I get thousands of violations (?), such as:

LONG-WRITE to  00000100 data=00000000 PC: 1025F11E
USP: ...
Data: ...
Stck: ...
Stck: ...
Name: "Background CLI" CLI: "thttpd"

(Copied by hand from the CON:.)

Does this output mean that THTTPd is really busting the memory?

I am going to run SegTracker and save Enforcer's output to a file if one of you wouldn't mind looking at it

[tygre]

Here are parts of Enforcer's output: the first hundred "violations" are like that one:

BYTE-READ from 0E5C10BF                        PC: 102773DC
USP:  10BF26CA SR: 0000 SW: 0521  (U0)(-)(-)  TCB: 10789918
Data: 00000000 0000000A 00000000 10BF7B70 00000012 0402C541 00000007 0E5C10BF
Addr: 0E5C10BF 00000026 10A97CF0 10BF28CC 10E67700 10BF2882 10279DE0 --------
Stck: 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC 10BF28C2 10BF288A
Stck: 10279DE0 0007102B FC8C10A9 7CE910BF 28C210BF 288A0000 00000000 00000000
----> 102773DC - "LIBS:locale.library"  Hunk 0000 Offset 00001AF4
Name: "Background CLI"  CLI: "thttpd"

and then, the next one (only one) is like:

LONG-READ from 5B03971A                        PC: 10A9A68A
USP:  10BF295A SR: 0010 SW: 0501  (U0)(-)(-)  TCB: 10789918
Data: 5B039722 5B0396E8 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B0396E8 0015BDA4 000001D4 10BF398A 10E67700 10BF297E 10000830 --------
Stck: 10CED688 00000138 000009C0 10CED67C 10CED67C 10BF2992 10AC7C4A 10B6DDB0
Stck: 00000002 10BF29A2 10A9AAC4 5B0396E8 10BF8AF0 10BF8B70 00000001 10A99840
----> 10A9A68A - "thttpd"  Hunk 0000 Offset 000212AA
Name: "Background CLI"  CLI: "thttpd"  Hunk 0000 Offset 000212AA

and then, the next dozen or so are like:

BYTE-WRITE to  5B039724        data=00         PC: 1037C0BE
USP:  10BF28E6 SR: 0004 SW: 0421  (U0)(-)(-)  TCB: 10789918
Data: 00000000 00000010 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B039725 5B039724 000001D4 10BF398A 10E67700 10BF28F6 10000830 --------
Stck: 10AC78D0 5B039724 00000000 00000010 10BF290A 10AC7A1C 5B039722 5B039722
Stck: 54412546 10BF2992 10A996BE 0015BDA4 000001D4 4F460000 00E60000 015D0000
----> 1037C0BE - "LIBS:ixemul.library"  Hunk 0000 Offset 0001892E
Name: "Background CLI"  CLI: "thttpd"

finally, the last ones before the Guru are:

BYTE-WRITE to  5B039796        data=00         PC: 10A99870
USP:  10BF297E SR: 0000 SW: 0421  (U0)(-)(-)  TCB: 10789918
Data: 0000003F 00000000 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B039796 5B0397B8 000001D4 10BF398A 10E67700 10BF298E 10000830 --------
Stck: 000001D4 10A9840E 10BF5B03 97962B00 10BF3906 10B2CFB2 0000003F 10BF2A96
Stck: 10BF2AB6 00000000 00000000 10B6DDB0 10BF7B70 00000000 00000000 00000000
----> 10A99870 - "thttpd"  Hunk 0000 Offset 00020490
Name: "Background CLI"  CLI: "thttpd"  Hunk 0000 Offset 00020490

LONG-WRITE to  5B039798        data=00000000   PC: 10A99880
USP:  10BF297E SR: 0004 SW: 0401  (U0)(-)(-)  TCB: 10789918
Data: 10BF2A96 5B039796 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B039798 10BF2A96 000001D4 10BF398A 10E67700 10BF298E 10000830 --------
Stck: 000001D4 10A9840E 10BF5B03 97962B00 10BF3906 10B2CFB2 0000003F 10BF2A96
Stck: 10BF2AB6 00000000 00000000 10B6DDB0 10BF7B70 00000000 00000000 00000000
----> 10A99880 - "thttpd"  Hunk 0000 Offset 000204A0
Name: "Background CLI"  CLI: "thttpd"  Hunk 0000 Offset 000204A0

LONG-WRITE to  5B03979C        data=00000000   PC: 10A99886
USP:  10BF297E SR: 0004 SW: 0401  (U0)(-)(-)  TCB: 10789918
Data: 10BF2A96 5B039796 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B03979C 10BF2A9A 000001D4 10BF398A 10E67700 10BF298E 10000830 --------
Stck: 000001D4 10A9840E 10BF5B03 97962B00 10BF3906 10B2CFB2 0000003F 10BF2A96
Stck: 10BF2AB6 00000000 00000000 10B6DDB0 10BF7B70 00000000 00000000 00000000
----> 10A99886 - "thttpd"  Hunk 0000 Offset 000204A6
Name: "Background CLI"  CLI: "thttpd"  Hunk 0000 Offset 000204A6

LONG-WRITE to  5B0397A0        data=00000000   PC: 10A9988C
USP:  10BF297E SR: 0004 SW: 0401  (U0)(-)(-)  TCB: 10789918
Data: 10BF2A96 5B039796 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B0397A0 10BF2A9E 000001D4 10BF398A 10E67700 10BF298E 10000830 --------
Stck: 000001D4 10A9840E 10BF5B03 97962B00 10BF3906 10B2CFB2 0000003F 10BF2A96
Stck: 10BF2AB6 00000000 00000000 10B6DDB0 10BF7B70 00000000 00000000 00000000
----> 10A9988C - "thttpd"  Hunk 0000 Offset 000204AC
Name: "Background CLI"  CLI: "thttpd"  Hunk 0000 Offset 000204AC

LONG-WRITE to  5B0397A4        data=00000000   PC: 10A99892
USP:  10BF297E SR: 0004 SW: 0401  (U0)(-)(-)  TCB: 10789918
Data: 10BF2A96 5B039796 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B0397A4 10BF2AA2 000001D4 10BF398A 10E67700 10BF298E 10000830 --------
Stck: 000001D4 10A9840E 10BF5B03 97962B00 10BF3906 10B2CFB2 0000003F 10BF2A96
Stck: 10BF2AB6 00000000 00000000 10B6DDB0 10BF7B70 00000000 00000000 00000000
----> 10A99892 - "thttpd"  Hunk 0000 Offset 000204B2
Name: "Background CLI"  CLI: "thttpd"  Hunk 0000 Offset 000204B2

LONG-WRITE to  5B0397A8        data=00000000   PC: 10A998A2
USP:  10BF297E SR: 0000 SW: 0401  (U0)(-)(-)  TCB: 10789918
Data: 10BF2AB6 5B039796 10B6DDB0 10BF7B70 00000012 0402C541 0402BFD7 10A793DC
Addr: 5B0397A8 10BF2AB6 000001D4 10BF398A 10E67700 10BF298E 10000830 --------
Stck: 000001D4 10A9840E 10BF5B03 97962B00 10BF3906 10B2CFB2 0000003F 10BF2A96
Stck: 10BF2AB6 00000000
Alert !! Alert 81000005     TCB: 10789918     USP: 10BF77A0
Data: 25EBB460 00003010 25EB9441 26019B60 00000012 0402C541 0402BFD7 81000005
Addr: 00003000 0015E700 000A30E8 10A793DC 10E67700 00F9F5BE 10000830 --------
Stck: 00F81D38 10A7750C 10B72AD4 00F81D64 10378574 25EB9441 10A7750C 00000012
----> 00F81D38 - "ROM - exec 40.10 (15.7.93)"  Hunk 0000 Offset 00001C7E
----> 00F81D64 - "ROM - exec 40.10 (15.7.93)"  Hunk 0000 Offset 00001CAA
----> 10378574 - "LIBS:ixemul.library"  Hunk 0000 Offset 00014DE4

Here is MCP'S GuruHistory:

Date  : Saturday 07-Jan-12 14:44:23
Task  : 10789918 "thttpd"
Error : 81000005 (DEADEND)
By    : ExecLibrary
Cause : Corrupt memory list detected in FreeMem

Could anyone help me decipher what is happening please?

[Toni Wilen]

Try to find which locale.library call is causing the error by adding some debug prints.

btw, do you have large enough stack? Unix ports may need huge stacks. (100k+)

[tygre]

Before I go on adding debug prints, I just would like to confirm that locale.library is the implementation of the functions provided by locale.h, right?

PS. Tried increasing the stack to 200,000, both in WinUAE and on my Classic Amiga, but no luck: still the same kind of "uninitialised trap #0 vector" gurus...

[NovaCoder]

Originally Posted by tygre

Here are some more details and a code snippet illustrating my problem:

#include <stdio.h>
int main()
{
    printf("Hello World!\n");
    void* ptr = malloc(1000);
    // realloc(ptr, 10000);
    free(ptr);
    printf("Goodbye.\n");
    return(0);
}

Please, let me know what you think!
Tygre

The gcc memory allocation routines seem to be a bit stuffed from what I've seen with calloc. The easiest thing to do is just re-implement relloc if that is what is causing you issues.

With calloc I found that if zero bytes were requested it would return NULL whereas other implementations behave the same way as malloc does when it is asked to allocate zero bytes (eg it doesn't return NULL).

This is the #define I use for the calloc issue:

#if !defined(FORCEINLINE) && (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1))
		#define FORCEINLINE inline __attribute__((__always_inline__))
	#endif

    #ifndef FORCEINLINE
        #define FORCEINLINE inline
    #endif


    // Fix 68k GCC issues with memory allocation/deallocation. 
    static FORCEINLINE void *callocEx(size_t num, size_t size) { 
        if (num == 0) {
            num = 1;
        }
    
        return calloc(num, size);
    }
    #define calloc(num, size) callocEx(num, size)

[tygre]

Thanks Novacoder, I will check the calls to calloc in THTTPd (and use your code to replace them ) and get inspiration from your code to eventually reimplement realloc.

But before doing that, I will follow Toni's advice and try to locate the call(s) causing all these hits in Enforcer related to locale.library, once step at a time .

[Toni Wilen]

Originally Posted by NovaCoder

With calloc I found that if zero bytes were requested it would return NULL whereas other implementations behave the same way as malloc does when it is asked to allocate zero bytes (eg it doesn't return NULL).

This is not a bug. C library memory allocation routines are allowed to return NULL or non-NULL pointer when attempting to allocate zero bytes. Both are valid return values.

[tygre]

I located at least one source of Enforcer's hits related to locale.library. PHP includes this piece of code in Zend/zend_compile.c, which is called when declaring a function in PHP:

static void build_runtime_defined_function_key(zval *result, zval *name, zend_op *opline TSRMLS_DC)
{
    char lineno_buf[32];
    uint lineno_len;
    char *filename;

    lineno_len = zend_sprintf(lineno_buf, "%d", opline->lineno);
    if (CG(active_op_array)->filename) {
        filename = CG(active_op_array)->filename;
    } else {
        filename = "-";
    }

    /* NULL, name length, filename length, line number length */
    result->value.str.len = 1+name->value.str.len+strlen(filename)+lineno_len;
    result->value.str.val = (char *) emalloc(result->value.str.len+1);
    sprintf(result->value.str.val, "%c%s%s%s", '\0', name->value.str.val, filename, lineno_buf);
    result->type = IS_STRING;
    result->refcount = 1;
}

I instrumented this piece of code like that:

static void build_runtime_defined_function_key(zval *result, zval *name, zend_op *opline TSRMLS_DC)
{
    char lineno_buf[32];
    uint lineno_len;
    char *filename;
    char output[32];
    sprintf(output, "\ta.\n");
    write(1, output, strlen(output));

    lineno_len = zend_sprintf(lineno_buf, "%d", opline->lineno);
    sprintf(output, "\tb.\n");
    write(1, output, strlen(output));
    if (CG(active_op_array)->filename) {
        sprintf(output, "\tc.\n");
        write(1, output, strlen(output));
        filename = CG(active_op_array)->filename;
        sprintf(output, "\td.\n");
        write(1, output, strlen(output));
    } else {
        filename = "-";
    }

    /* NULL, name length, filename length, line number length */
    result->value.str.len = 1+name->value.str.len+strlen(filename)+lineno_len;
    sprintf(output, "\te.\n");
    write(1, output, strlen(output));
    result->value.str.val = (char *) emalloc(result->value.str.len+1);
    sprintf(output, "\tf.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%c", '\0');
    sprintf(output, "\tf1.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%s", name->value.str.val);
    sprintf(output, "\tf2.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%c%s", '\0', name->value.str.val);
    sprintf(output, "\tf3.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%c%s", 'a', " strange thing...");
    sprintf(output, "\tf4.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%c%s", '\0', " strange thing...");
    sprintf(output, "\tf5.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%c%s", 'a', name->value.str.val);
    sprintf(output, "\tf6.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%s%c", name->value.str.val, '\0');
    sprintf(output, "\tf7.\n");
    write(1, output, strlen(output));
    sprintf(output, "%c%s", '\0', " strange thing...");
    sprintf(output, "\tf8.\n");
    write(1, output, strlen(output));
    sprintf(result->value.str.val, "%c%s%s%s", '\0', name->value.str.val, filename, lineno_buf);
    sprintf(output, "\tg.\n");
    write(1, output, strlen(output));
    result->type = IS_STRING;
    result->refcount = 1;
}

And here is the output from the console, in which both Enforcer and THTTPd run and output (copied by hand so I skip some details):

a.
b.
c.
d.
e.
f.
f1.
f2.
BYTE-READ from...
...
----> 10274BDC - "LIBS:locale.library"
Name: "Background CLI" CLI: "thttpd"
f3.
f4.
BYTE-READ from...
...
----> 10274BDC - "LIBS:locale.library"
Name: "Background CLI" CLI: "thttpd"
f5.
f6.
f7.
BYTE-READ from...
 ...
 ----> 10274BDC - "LIBS:locale.library"
 Name: "Background CLI" CLI: "thttpd"
  f8.
BYTE-READ from...
 ...
 ----> 10274BDC - "LIBS:locale.library"
 Name: "Background CLI" CLI: "thttpd"
BYTE-READ from...
 ...
 ----> 10274BDC - "LIBS:locale.library"
 Name: "Background CLI" CLI: "thttpd"
BYTE-READ from...
 ...
 ----> 10274BDC - "LIBS:locale.library"
 Name: "Background CLI" CLI: "thttpd"
  g.

which means that anytime a call to sprinf is made with "%c" and '\0' as first format and character, there is a hit in the locale.library. Does it make sense?

Also, does it make sense that the following simple code snippet does not produce a hit in Enforcer?

#include <stdio.h>
int main()
{
    char output[32];

    printf("1.\n");
    sprintf(output, "%c%s", '\0', " strange thing...");
    printf("2.\n");
    sprintf(output, "%s%c", " strange thing...", '\0');
    printf("Done.\n");
    return(0);
}

What do you think?

I found out that I can replace the calls to sprintf with calls to zend_sprintf, which is defined when sprintf is "broken". This replacement yields the expected results and does not produce any hit in Enforcer

#if ZEND_BROKEN_SPRINTF
int zend_sprintf(char *buffer, const char *format, ...)
{
	va_list args;

	va_start(args, format);
	vsprintf(buffer, format, args);
	va_end(args);

	return strlen(buffer);
}
#endif

So, should I guess that sprintf is really broken in IXEmul?

[Toni Wilen]

I think this is GCC variadic function stack parameter size issue which is (was?) problem in m68k AROS too.

GCC always converts all variadic function parameters to LONG. Amiga m68k routines expect WORD unless it has 'l' prefix or parameter is an address.

Quick test: replace %c with %lc. Does it work now?

[tygre]

You are absolutely right, Toni I replaced the original call:

sprintf(result->value.str.val, "%c%s%s%s", '\0', name->value.str.val, filename, lineno_buf);

which yields to hits in Enforcer, with the call:

sprintf(result->value.str.val, "%lc%s%s%s", '\0', name->value.str.val, filename, lineno_buf);

and the code works like a charm without any hits But what a pain to track... and to fix: Toni, does this mismatch between GCC and AmigaOS means that we should systematically add 'l' to any parameter format:

%c -> %lc
%d -> %ld
%e -> %le
...

except if the parameter is a pointer? Actually, what would happen if the parameter is a pointer and we would add 'l'?

Now, its seems that my version of THTTPd works fine, without anymore Enforcer's hit or crash... I will revert the stack to its original size of 40,000 instead of the current 200,000 to see if I can reproduce my original crash, count on me to keep you posted Thanks a lot for your kind help!

[NovaCoder]

Originally Posted by Toni Wilen

This is not a bug. C library memory allocation routines are allowed to return NULL or non-NULL pointer when attempting to allocate zero bytes. Both are valid return values.

That may be true but for the large (multi-plaform) C++ project that I'm building (ScummVM), only the 68k version of gcc seems to behave this way

[Toni Wilen]

Originally Posted by tygre

and the code works like a charm without any hits But what a pain to track... and to fix: Toni, does this mismatch between GCC and AmigaOS means that we should systematically add 'l' to any parameter format:

%c -> %lc
%d -> %ld
%e -> %le
...

Yeah, that works but perhaps there is better solution.

except if the parameter is a pointer? Actually, what would happen if the parameter is a pointer and we would add 'l'?

Nothing happens but it would look even more ugly

Now, its seems that my version of THTTPd works fine, without anymore Enforcer's hit or crash... I will revert the stack to its original size of 40,000 instead of the current 200,000 to see if I can reproduce my original crash, count on me to keep you posted Thanks a lot for your kind help!

Great!

Originally Posted by NovaCoder

That may be true but for the large (multi-plaform) C++ project that I'm building (ScummVM), only the 68k version of gcc seems to behave this way

It still is not a bug, not even if it returns NULL on thursdays and non-NULL on other days

I thought most bigger projects would use some kind of memory allocation wrapper(s) for portability reasons.

[tygre]

Thanks a lot Toni!

A solution could be to wrap any call to the [sn]printf family of functions so that the wrapper could automagically add 'l' in front of each format character... I guess scanf could have the same problem?

When I succeed in reproducing this behaviour, I'll try and keep you posted!

BTW, my apologies for saying that IXEmul was broken, it is not

[tygre]

PS. Dear moderators, would it be possible to add "[Solved]" to the title of this thread? Thanks!

[prowler]

Originally Posted by tygre

PS. Dear moderators, would it be possible to add "[Solved]" to the title of this thread? Thanks!

Done!

[tygre]

PS. Thanks Prowler!

PPS. THTTPd "starts" crashing again. I must have missed some sprintf... I am tracking them down now... It is my mistake, I mean: the solution in this thread works!

[tygre]

Hi all!

Sorry to bump this thread so many years later but I just wanted to confirm that the problem wasn't with the realloc() function, even when I was using wrongly as pointed out but rather due to conflicts between ixemul.library and some patches (like MCP) that I have now sorted out!

Thanks all for your help!