|
Hi Stefan,
My previous dump is useless as your elog-debuginfo rpm is stripped.
So i rebuild elog w/ -g , reinstall it and activate core generation. I restst a search of "test" and get this in gdb corefile .
[root@localhost ~]# gdb /var/crash/core.elogd.11613.localhost.localdomain.1580820772
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 11613]
Reading symbols from /usr/local/sbin/elogd...Reading symbols from /usr/lib/debug/usr/local/sbin/elogd.debug...done.
done.
Missing separate debuginfo for
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b5/a5458535a1397fa6baaf5e8c13a6395426a1b2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/elogd -D -c /usr/local/elog/elogd.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f274640aa28 in __GI_abort () at abort.c:90
#2 0x00007f274644be87 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f274655e3b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f2746454679 in malloc_printerr (ar_ptr=0x7f274679a760 <main_arena>, ptr=<optimized out>,
str=0x7f274655e450 "free(): invalid next size (normal)", action=3) at malloc.c:4967
#4 _int_free (av=0x7f274679a760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e456 in display_line (lbs=0x222c968, message_id=426, number=<optimized out>,
mode=0x7fffde328c90 "summary", expand=1, level=0, printable=0, n_line=3, show_attachments=0,
show_att_column=1, date=0x7fffde328b50 "Wed, 04 Dec 2019 07:41:34 +0000",
in_reply_to=0x7fffde328ba0 "", reply_to=0x7fffde329600 "", n_attr_disp=14,
disp_attr=0x7fffde3792a0, disp_attr_link=0x7fffde329460, attrib=0x7fffde32fec0, n_attr=14,
text=0x2378d18 "<div style=\"margin: 0px; padding: 0px; text-indent: 0px;\">\r\n<table border=\"1\" cellpadding=\"1\" cellspacing=\"1\" style=\"height:230px; width:1308px\">\r\n\t<caption>\r\n\t<p style=\"text-align:left\"><strong>MAJ 2"..., show_text=1, attachment=0x7fffde32ccc0,
encoding=0x7fffde328bf0 "HTML", select=0, n_display=0x7fffde328a80, locked_by=0x7fffde328ec0 "",
highlight=0, re_buf=0x7fffde32b380, highlight_mid=0, absolute_link=0, draft=0x7fffde3290c0 "")
at src/elogd.c:18520
#6 0x0000000000482140 in show_elog_list (lbs=lbs@entry=0x222c968, past_n=past_n@entry=0,
last_n=last_n@entry=0, page_n=<optimized out>, page_n@entry=0, default_page=<optimized out>,
default_page@entry=1, info=info@entry=0x0) at src/elogd.c:21859
#7 0x00000000004a74a0 in interprete (lbook=lbook@entry=0x7fffde3c79e0 "MCO", path=<optimized out>,
path@entry=0x7fffde3c7520 "") at src/elogd.c:28454
#8 0x00000000004a7b13 in decode_get (logbook=logbook@entry=0x7fffde3c79e0 "MCO",
string=<optimized out>) at src/elogd.c:28494
#9 0x00000000004aa03a in process_http_request (request=<optimized out>,
request@entry=0x223e288 "GET /MCO/?mode=summary&reverse=0&reverse=1&npp=30&subtext=test",
i_conn=i_conn@entry=0) at src/elogd.c:29272
#10 0x00000000004ad385 in server_loop () at src/elogd.c:30286
#11 0x0000000000402e97 in main (argc=<optimized out>, argv=<optimized out>) at src/elogd.c:31314
Is it better to find problem ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan.
I installed debuginfo and ran elogd with gdb attached.
(gdb) backtrace
#0 0x00007fde38104337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fde38105a28 in __GI_abort () at abort.c:90
#2 0x00007fde38146e87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fde382593b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007fde3814f679 in malloc_printerr (ar_ptr=0x7fde38495760 <main_arena>, ptr=<optimized out>, str=0x7fde38259450 "free(): invalid next size (normal)", action=3)
at malloc.c:4967
#4 _int_free (av=0x7fde38495760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e756 in display_line ()
#6 0x00000000004824b0 in show_elog_list ()
#7 0x00000000004a69b1 in interprete ()
#8 0x00000000004a7023 in decode_get ()
#9 0x00000000004a954a in process_http_request ()
#10 0x00000000004ac9d3 in server_loop ()
#11 0x00000000004035a6 in main ()
Does it help you ?
| Stefan Ritt wrote: |
|
The only way for me is to reproduce it. Could be corrupted data file, or config file. Try with a new logbook with the same configuration, then add logbook files one by one until it happens.
Stefan
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
The problem seems to be specific with our logbooks. Maybe one of the file is corrupted or bad string encoding ?! But how to find the source of the problem ?
Any way to improve debug log ? (using debug level = 9 + logfile)
Also, do you have a procedure to debug ELOG with debuginfo package installed to have more informations in trace ?
.
Laurent
| Stefan Ritt wrote: |
|
Do you see the same problem on this server here?
If not, then it must be related to your specific configuration. You have to teach me how to reporduce the error in order to fix it. Best would be a minimal elogd.cfg which causes the error.
Stefan
| Laurent Jean-Rigaud wrote: |
|
OS: CentOS 7 x86_64 up2date with RPM 3.14 x86_64
- Connexion to ELOG
- open one logbook and click search,
- input "test" and click "Search" : BOOM
- Firefox can not connect anymore on ELOG.
The crash is met with several words now ("test", "post", "4.5", are the one i know). We use a mirror server and the problem appears also on it .
NB: the hangs appears on EL6 x86_64 server with customized RPM built by myself. I retry with 3.14 SRPMS avalaible from ELOG site with same results. Idem with last GIT version. So i tested on EL7 with official RPM x86_64 (not suitable for EL6 as it needs GLIBC_2.14) before to open this ticket.
Terminal traces on server side follows.
[root@localhost /]# /usr/local/sbin/elogd -c /usr/local/elog/elogd.cfg 2>&1
elogd 3.1.4 built Sep 26 2018, 13:14:57 revision 966e3dd
File "/var/run/elogd.pid" exists, using "/var/run/elogd.pid.8080" instead.
Falling back to default group "elog"
Falling back to default user "elog"
CKeditor detected
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
ImageMagick NOT detected. Image scaling will not work.
Indexing logbooks ... done
Server listening on port 8080 ...
*** Error in `/usr/local/sbin/elogd': free(): invalid next size (normal): 0x0000000001c59310 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7f45a898f679]
/usr/local/sbin/elogd[0x45e756]
/usr/local/sbin/elogd[0x4824b0]
/usr/local/sbin/elogd[0x4a69b1]
/usr/local/sbin/elogd[0x4a7023]
/usr/local/sbin/elogd[0x4a954a]
/usr/local/sbin/elogd[0x4ac9d3]
/usr/local/sbin/elogd[0x4035a6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f45a8930505]
/usr/local/sbin/elogd[0x4044d3]
======= Memory map: ========
00400000-004e3000 r-xp 00000000 fd:00 52822202 /usr/local/sbin/elogd
006e2000-006e3000 r--p 000e2000 fd:00 52822202 /usr/local/sbin/elogd
006e3000-007c4000 rw-p 000e3000 fd:00 52822202 /usr/local/sbin/elogd
007c4000-0173a000 rw-p 00000000 00:00 0
01a66000-01c8f000 rw-p 00000000 00:00 0 [heap]
7f459bdea000-7f459bdff000 r-xp 00000000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bdff000-7f459bffe000 ---p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bffe000-7f459bfff000 r--p 00014000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bfff000-7f459c000000 rw-p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459c000000-7f459c021000 rw-p 00000000 00:00 0
7f459c021000-7f45a0000000 ---p 00000000 00:00 0
7f45a010c000-7f45a0114000 r-xp 00000000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0114000-7f45a0313000 ---p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0313000-7f45a0314000 r--p 00007000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0314000-7f45a0315000 rw-p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0315000-7f45a0321000 r-xp 00000000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0321000-7f45a0520000 ---p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0520000-7f45a0521000 r--p 0000b000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0521000-7f45a0522000 rw-p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0522000-7f45a0528000 rw-p 00000000 00:00 0
7f45a0528000-7f45a6a52000 r--p 00000000 fd:00 50336362 /usr/lib/locale/locale-archive
7f45a6a52000-7f45a6ab2000 r-xp 00000000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6ab2000-7f45a6cb2000 ---p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb2000-7f45a6cb3000 r--p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb3000-7f45a6cb4000 rw-p 00061000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb4000-7f45a6cd8000 r-xp 00000000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6cd8000-7f45a6ed7000 ---p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed7000-7f45a6ed8000 r--p 00023000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed8000-7f45a6ed9000 rw-p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed9000-7f45a6edb000 rw-p 00000000 00:00 0
7f45a6edb000-7f45a6ef2000 r-xp 00000000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a6ef2000-7f45a70f1000 ---p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f1000-7f45a70f2000 r--p 00016000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f2000-7f45a70f3000 rw-p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f3000-7f45a70f7000 rw-p 00000000 00:00 0
7f45a70f7000-7f45a710d000 r-xp 00000000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a710d000-7f45a730c000 ---p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730c000-7f45a730d000 r--p 00015000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730d000-7f45a730e000 rw-p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730e000-7f45a7310000 rw-p 00000000 00:00 0
7f45a7310000-7f45a7313000 r-xp 00000000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7313000-7f45a7512000 ---p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7512000-7f45a7513000 r--p 00002000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7513000-7f45a7514000 rw-p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7514000-7f45a7522000 r-xp 00000000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7522000-7f45a7722000 ---p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7722000-7f45a7723000 r--p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7723000-7f45a7724000 rw-p 0000f000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7724000-7f45a7739000 r-xp 00000000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7739000-7f45a7938000 ---p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7938000-7f45a7939000 r--p 00014000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7939000-7f45a793a000 rw-p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a793a000-7f45a793c000 r-xp 00000000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a793c000-7f45a7b3c000 ---p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3c000-7f45a7b3d000 r--p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3d000-7f45a7b3e000 rw-p 00003000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3e000-7f45a7d74000 r-xp 00000000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7d74000-7f45a7f74000 ---p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f74000-7f45a7f90000 r--p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f90000-7f45a7f9d000 rw-p 00252000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f9d000-7f45a7fa1000 rw-p 00000000 00:00 0
7f45a7fa1000-7f45a7fd2000 r-xp 00000000 fd:00 321856 /usr/lib64/libk5crypto.so.3.1
Abandon [sorry, French locale]
Logbook contains 445 entries (shown in ELOG window), 384 files and 4 folders (2017 to 2020).
Thanks for help as i start to be annoying to unuse search function.
|
|
|
|
|
|