using backtrace(3) in libpthread

[Emmanuel Dreyfus]

Hi

I would like to obtain a backtrace using libexecinfo in
pthread_key_create() and pthread_key_delete(). I installed libexecinfo
from pkgsrc and added the code below at the end of the two functions.

I use snprintf()/syslog() because pthread__errorfunc() has comments
telling this is the way to go to avoid deadlocks.... but I still deadlock.
Any idea how this should be done?

{
void *bt[BT_DEPTH];
char **btstr;
int depth;
char buf[4096];
size_t len = 0;
int j;

len += snprintf(buf + len, sizeof(buf) - len, "*** %s ***\n",
__func__);
depth = backtrace(bt, BT_DEPTH);
btstr = backtrace_symbols(bt, depth);
for (j = 0; j < depth; j++) {
len += snprintf(buf + len,
sizeof(buf) - len, "#%d %s\n",
j, btstr[j]);
}

syslog(LOG_ERR | LOG_USER, "%s", buf);
}

[Christos Zoulas]

Post by Emmanuel Dreyfus
Hi
I would like to obtain a backtrace using libexecinfo in
pthread_key_create() and pthread_key_delete(). I installed libexecinfo
from pkgsrc and added the code below at the end of the two functions.
I use snprintf()/syslog() because pthread__errorfunc() has comments
telling this is the way to go to avoid deadlocks.... but I still deadlock.
Any idea how this should be done?
{
void *bt[BT_DEPTH];
char **btstr;
int depth;
char buf[4096];
size_t len = 0;
int j;
len += snprintf(buf + len, sizeof(buf) - len, "*** %s ***\n",
__func__);
depth = backtrace(bt, BT_DEPTH);
btstr = backtrace_symbols(bt, depth);
for (j = 0; j < depth; j++) {
len += snprintf(buf + len,
sizeof(buf) - len, "#%d %s\n",
j, btstr[j]);
}
syslog(LOG_ERR | LOG_USER, "%s", buf);
}

You want syslog_r

christos

[Emmanuel Dreyfus]

Post by Christos Zoulas
You want syslog_r

What is the proper wy to call openlog_r()? When doing it
in pthread_key_create() I get no better success.

[Christos Zoulas]

On Apr 25, 6:54pm, ***@netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: using backtrace(3) in libpthread

| On Fri, Apr 25, 2014 at 05:17:11PM +0000, Christos Zoulas wrote:
| > You want syslog_r
|
| What is the proper wy to call openlog_r()? When doing it
| in pthread_key_create() I get no better success.

You don't need openlog.

struct syslog_data sd = SYSLOG_DATA_INIT;
syslog_r(..., &sd, ...);

christos

[Emmanuel Dreyfus]

Post by Christos Zoulas
struct syslog_data sd = SYSLOG_DATA_INIT;
syslog_r(..., &sd, ...);

Even this simple case at the end of pthread_key_create() is enough
to deadlock:

static struct syslog_data sd = SYSLOG_DATA_INIT;

syslog_r(LOG_ERR, &sd, "test");

[Christos Zoulas]

On Apr 25, 11:20pm, ***@netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: using backtrace(3) in libpthread

| On Fri, Apr 25, 2014 at 04:09:31PM -0400, Christos Zoulas wrote:
| > struct syslog_data sd = SYSLOG_DATA_INIT;
| > syslog_r(..., &sd, ...);
|
| Even this simple case at the end of pthread_key_create() is enough
| to deadlock:
|
| static struct syslog_data sd = SYSLOG_DATA_INIT;

Where does it deadlock? Can gdb show you?

| syslog_r(LOG_ERR, &sd, "test");

Can you try syslog_ss()?

christos

[Emmanuel Dreyfus]

Post by Christos Zoulas
| syslog_r(LOG_ERR, &sd, "test");
Can you try syslog_ss()?

Backporting it was easy. Now I can log, and that let me trace it to
the place where it deadlocks: backtrace_symbols()

Given that one performs malloc() I guess I should not be surprised.
Is it possible to get the symbols from there?

[Christos Zoulas]

On Apr 26, 3:41am, ***@netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: using backtrace(3) in libpthread

| On Fri, Apr 25, 2014 at 09:33:37PM -0400, Christos Zoulas wrote:
| > | syslog_r(LOG_ERR, &sd, "test");
| > Can you try syslog_ss()?
|
| Backporting it was easy. Now I can log, and that let me trace it to
| the place where it deadlocks: backtrace_symbols()
|
| Given that one performs malloc() I guess I should not be surprised.
| Is it possible to get the symbols from there?

It does one realloc. Why don't you make it use statically allocated space.

christos

[Emmanuel Dreyfus]

Post by Christos Zoulas
It does one realloc. Why don't you make it use statically allocated space.

That helps! I get my first backtrace logged, but the process then
deadlocks somewhere else.

Here is what I got:
*** pthread_key_create ***
#0 0xbb7533e8 <__libc_thr_keycreate+250> at ./libpthread.so.1
#1 0xbb6d1ef1 <_strncasecmp+11709> at /lib/libc.so.12
#2 0xbb6d2ae7 <malloc+116> at /lib/libc.so.12
#3 0xbb6d5a28 <__learn_tree+171> at /lib/libc.so.12
#4 0xbb6d6765 <__learn_tree+3560> at /lib/libc.so.12
#5 0xbb6d6b23 <_sysctlgetmibinfo+80> at /lib/libc.so.12
#6 0xbb6c832d <sysctlbyname+77> at /lib/libc.so.12
#7 0xbb7db129 <nextafterl+553> at /lib/libm.so.0
#8 0xbb7cc898 <_init+2456> at /lib/libm.so.0
#9 0xbb7f2eeb

[Emmanuel Dreyfus]

Post by Emmanuel Dreyfus
That helps! I get my first backtrace logged, but the process then
deadlocks somewhere else.

But this time I can attach with gdb:

0xbb7fa407 in ___lwp_park50 () from /usr/libexec/ld.elf_so
(gdb) bt
#0 0xbb7fa407 in ___lwp_park50 () from /usr/libexec/ld.elf_so
#1 0xbb7f25c5 in _rtld_exclusive_enter () from /usr/libexec/ld.elf_so
#2 0xbb639275 in __register_frame_info () from /lib/libgcc_s.so.1

I see that libexecinfo has a weak reference on __register_frame_info(),
but I do not understand how it relates to the code it contains.

[Emmanuel Dreyfus]

Post by Emmanuel Dreyfus
I see that libexecinfo has a weak reference on __register_frame_info(),
but I do not understand how it relates to the code it contains.

If I skip the two first backtrace (I use a static counter to count
that), it works fine. Since the goal is to track a pthread_key leak,
that will be fine for my usage.

[Christos Zoulas]

On Apr 27, 2:23pm, ***@netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: using backtrace(3) in libpthread

| Emmanuel Dreyfus <***@netbsd.org> wrote:
|
| > I see that libexecinfo has a weak reference on __register_frame_info(),
| > but I do not understand how it relates to the code it contains.
|
| If I skip the two first backtrace (I use a static counter to count
| that), it works fine. Since the goal is to track a pthread_key leak,
| that will be fine for my usage.

It probably can't handle things right during dynamic linking.

christos

[Joerg Sonnenberger]

Post by Emmanuel Dreyfus
I would like to obtain a backtrace using libexecinfo in
pthread_key_create() and pthread_key_delete(). I installed libexecinfo
from pkgsrc and added the code below at the end of the two functions.

Note that backtrace(3) is quite likely to require mutexes and/or malloc.

Joerg

[Emmanuel Dreyfus]

Post by Joerg Sonnenberger

Post by Emmanuel Dreyfus
I would like to obtain a backtrace using libexecinfo in
pthread_key_create() and pthread_key_delete(). I installed libexecinfo
from pkgsrc and added the code below at the end of the two functions.

Note that backtrace(3) is quite likely to require mutexes and/or malloc.

It therefore means it cannot be used there?