ID |
Date |
Icon |
Author |
Author Email |
Category |
OS |
ELOG Version |
Subject |
|
69109
|
Tue Feb 4 13:56:51 2020 |
 | Laurent Jean-Rigaud | lollspam@free.fr | Bug report | Linux | 3.14 RPM | Re: Elogd crashes on search | Hi Stefan,
My previous dump is useless as your elog-debuginfo rpm is stripped.
So i rebuild elog w/ -g , reinstall it and activate core generation. I restst a search of "test" and get this in gdb corefile .
[root@localhost ~]# gdb /var/crash/core.elogd.11613.localhost.localdomain.1580820772
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 11613]
Reading symbols from /usr/local/sbin/elogd...Reading symbols from /usr/lib/debug/usr/local/sbin/elogd.debug...done.
done.
Missing separate debuginfo for
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b5/a5458535a1397fa6baaf5e8c13a6395426a1b2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/elogd -D -c /usr/local/elog/elogd.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f274640aa28 in __GI_abort () at abort.c:90
#2 0x00007f274644be87 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f274655e3b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f2746454679 in malloc_printerr (ar_ptr=0x7f274679a760 <main_arena>, ptr=<optimized out>,
str=0x7f274655e450 "free(): invalid next size (normal)", action=3) at malloc.c:4967
#4 _int_free (av=0x7f274679a760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e456 in display_line (lbs=0x222c968, message_id=426, number=<optimized out>,
mode=0x7fffde328c90 "summary", expand=1, level=0, printable=0, n_line=3, show_attachments=0,
show_att_column=1, date=0x7fffde328b50 "Wed, 04 Dec 2019 07:41:34 +0000",
in_reply_to=0x7fffde328ba0 "", reply_to=0x7fffde329600 "", n_attr_disp=14,
disp_attr=0x7fffde3792a0, disp_attr_link=0x7fffde329460, attrib=0x7fffde32fec0, n_attr=14,
text=0x2378d18 "<div style=\"margin: 0px; padding: 0px; text-indent: 0px;\">\r\n<table border=\"1\" cellpadding=\"1\" cellspacing=\"1\" style=\"height:230px; width:1308px\">\r\n\t<caption>\r\n\t<p style=\"text-align:left\"><strong>MAJ 2"..., show_text=1, attachment=0x7fffde32ccc0,
encoding=0x7fffde328bf0 "HTML", select=0, n_display=0x7fffde328a80, locked_by=0x7fffde328ec0 "",
highlight=0, re_buf=0x7fffde32b380, highlight_mid=0, absolute_link=0, draft=0x7fffde3290c0 "")
at src/elogd.c:18520
#6 0x0000000000482140 in show_elog_list (lbs=lbs@entry=0x222c968, past_n=past_n@entry=0,
last_n=last_n@entry=0, page_n=<optimized out>, page_n@entry=0, default_page=<optimized out>,
default_page@entry=1, info=info@entry=0x0) at src/elogd.c:21859
#7 0x00000000004a74a0 in interprete (lbook=lbook@entry=0x7fffde3c79e0 "MCO", path=<optimized out>,
path@entry=0x7fffde3c7520 "") at src/elogd.c:28454
#8 0x00000000004a7b13 in decode_get (logbook=logbook@entry=0x7fffde3c79e0 "MCO",
string=<optimized out>) at src/elogd.c:28494
#9 0x00000000004aa03a in process_http_request (request=<optimized out>,
request@entry=0x223e288 "GET /MCO/?mode=summary&reverse=0&reverse=1&npp=30&subtext=test",
i_conn=i_conn@entry=0) at src/elogd.c:29272
#10 0x00000000004ad385 in server_loop () at src/elogd.c:30286
#11 0x0000000000402e97 in main (argc=<optimized out>, argv=<optimized out>) at src/elogd.c:31314
Is it better to find problem ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan.
I installed debuginfo and ran elogd with gdb attached.
(gdb) backtrace
#0 0x00007fde38104337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fde38105a28 in __GI_abort () at abort.c:90
#2 0x00007fde38146e87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fde382593b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007fde3814f679 in malloc_printerr (ar_ptr=0x7fde38495760 <main_arena>, ptr=<optimized out>, str=0x7fde38259450 "free(): invalid next size (normal)", action=3)
at malloc.c:4967
#4 _int_free (av=0x7fde38495760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e756 in display_line ()
#6 0x00000000004824b0 in show_elog_list ()
#7 0x00000000004a69b1 in interprete ()
#8 0x00000000004a7023 in decode_get ()
#9 0x00000000004a954a in process_http_request ()
#10 0x00000000004ac9d3 in server_loop ()
#11 0x00000000004035a6 in main ()
Does it help you ?
| Stefan Ritt wrote: |
|
The only way for me is to reproduce it. Could be corrupted data file, or config file. Try with a new logbook with the same configuration, then add logbook files one by one until it happens.
Stefan
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
The problem seems to be specific with our logbooks. Maybe one of the file is corrupted or bad string encoding ?! But how to find the source of the problem ?
Any way to improve debug log ? (using debug level = 9 + logfile)
Also, do you have a procedure to debug ELOG with debuginfo package installed to have more informations in trace ?
.
Laurent
| Stefan Ritt wrote: |
|
Do you see the same problem on this server here?
If not, then it must be related to your specific configuration. You have to teach me how to reporduce the error in order to fix it. Best would be a minimal elogd.cfg which causes the error.
Stefan
| Laurent Jean-Rigaud wrote: |
|
OS: CentOS 7 x86_64 up2date with RPM 3.14 x86_64
- Connexion to ELOG
- open one logbook and click search,
- input "test" and click "Search" : BOOM
- Firefox can not connect anymore on ELOG.
The crash is met with several words now ("test", "post", "4.5", are the one i know). We use a mirror server and the problem appears also on it .
NB: the hangs appears on EL6 x86_64 server with customized RPM built by myself. I retry with 3.14 SRPMS avalaible from ELOG site with same results. Idem with last GIT version. So i tested on EL7 with official RPM x86_64 (not suitable for EL6 as it needs GLIBC_2.14) before to open this ticket.
Terminal traces on server side follows.
[root@localhost /]# /usr/local/sbin/elogd -c /usr/local/elog/elogd.cfg 2>&1
elogd 3.1.4 built Sep 26 2018, 13:14:57 revision 966e3dd
File "/var/run/elogd.pid" exists, using "/var/run/elogd.pid.8080" instead.
Falling back to default group "elog"
Falling back to default user "elog"
CKeditor detected
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
ImageMagick NOT detected. Image scaling will not work.
Indexing logbooks ... done
Server listening on port 8080 ...
*** Error in `/usr/local/sbin/elogd': free(): invalid next size (normal): 0x0000000001c59310 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7f45a898f679]
/usr/local/sbin/elogd[0x45e756]
/usr/local/sbin/elogd[0x4824b0]
/usr/local/sbin/elogd[0x4a69b1]
/usr/local/sbin/elogd[0x4a7023]
/usr/local/sbin/elogd[0x4a954a]
/usr/local/sbin/elogd[0x4ac9d3]
/usr/local/sbin/elogd[0x4035a6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f45a8930505]
/usr/local/sbin/elogd[0x4044d3]
======= Memory map: ========
00400000-004e3000 r-xp 00000000 fd:00 52822202 /usr/local/sbin/elogd
006e2000-006e3000 r--p 000e2000 fd:00 52822202 /usr/local/sbin/elogd
006e3000-007c4000 rw-p 000e3000 fd:00 52822202 /usr/local/sbin/elogd
007c4000-0173a000 rw-p 00000000 00:00 0
01a66000-01c8f000 rw-p 00000000 00:00 0 [heap]
7f459bdea000-7f459bdff000 r-xp 00000000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bdff000-7f459bffe000 ---p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bffe000-7f459bfff000 r--p 00014000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bfff000-7f459c000000 rw-p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459c000000-7f459c021000 rw-p 00000000 00:00 0
7f459c021000-7f45a0000000 ---p 00000000 00:00 0
7f45a010c000-7f45a0114000 r-xp 00000000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0114000-7f45a0313000 ---p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0313000-7f45a0314000 r--p 00007000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0314000-7f45a0315000 rw-p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0315000-7f45a0321000 r-xp 00000000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0321000-7f45a0520000 ---p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0520000-7f45a0521000 r--p 0000b000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0521000-7f45a0522000 rw-p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0522000-7f45a0528000 rw-p 00000000 00:00 0
7f45a0528000-7f45a6a52000 r--p 00000000 fd:00 50336362 /usr/lib/locale/locale-archive
7f45a6a52000-7f45a6ab2000 r-xp 00000000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6ab2000-7f45a6cb2000 ---p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb2000-7f45a6cb3000 r--p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb3000-7f45a6cb4000 rw-p 00061000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb4000-7f45a6cd8000 r-xp 00000000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6cd8000-7f45a6ed7000 ---p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed7000-7f45a6ed8000 r--p 00023000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed8000-7f45a6ed9000 rw-p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed9000-7f45a6edb000 rw-p 00000000 00:00 0
7f45a6edb000-7f45a6ef2000 r-xp 00000000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a6ef2000-7f45a70f1000 ---p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f1000-7f45a70f2000 r--p 00016000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f2000-7f45a70f3000 rw-p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f3000-7f45a70f7000 rw-p 00000000 00:00 0
7f45a70f7000-7f45a710d000 r-xp 00000000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a710d000-7f45a730c000 ---p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730c000-7f45a730d000 r--p 00015000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730d000-7f45a730e000 rw-p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730e000-7f45a7310000 rw-p 00000000 00:00 0
7f45a7310000-7f45a7313000 r-xp 00000000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7313000-7f45a7512000 ---p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7512000-7f45a7513000 r--p 00002000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7513000-7f45a7514000 rw-p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7514000-7f45a7522000 r-xp 00000000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7522000-7f45a7722000 ---p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7722000-7f45a7723000 r--p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7723000-7f45a7724000 rw-p 0000f000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7724000-7f45a7739000 r-xp 00000000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7739000-7f45a7938000 ---p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7938000-7f45a7939000 r--p 00014000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7939000-7f45a793a000 rw-p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a793a000-7f45a793c000 r-xp 00000000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a793c000-7f45a7b3c000 ---p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3c000-7f45a7b3d000 r--p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3d000-7f45a7b3e000 rw-p 00003000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3e000-7f45a7d74000 r-xp 00000000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7d74000-7f45a7f74000 ---p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f74000-7f45a7f90000 r--p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f90000-7f45a7f9d000 rw-p 00252000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f9d000-7f45a7fa1000 rw-p 00000000 00:00 0
7f45a7fa1000-7f45a7fd2000 r-xp 00000000 fd:00 321856 /usr/lib64/libk5crypto.so.3.1
Abandon [sorry, French locale]
Logbook contains 445 entries (shown in ELOG window), 384 files and 4 folders (2017 to 2020).
Thanks for help as i start to be annoying to unuse search function.
|
|
|
|
|
|
|
69110
|
Tue Feb 4 17:07:02 2020 |
| Laurent Jean-Rigaud | lollspam@free.fr | Bug report | Linux | 3.14 RPM | Re: Elogd crashes on search | At #5, there is message_id=426 . You think it's this elog entries # 426 that is the source of crash ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
My previous dump is useless as your elog-debuginfo rpm is stripped.
So i rebuild elog w/ -g , reinstall it and activate core generation. I restst a search of "test" and get this in gdb corefile .
[root@localhost ~]# gdb /var/crash/core.elogd.11613.localhost.localdomain.1580820772
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 11613]
Reading symbols from /usr/local/sbin/elogd...Reading symbols from /usr/lib/debug/usr/local/sbin/elogd.debug...done.
done.
Missing separate debuginfo for
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b5/a5458535a1397fa6baaf5e8c13a6395426a1b2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/elogd -D -c /usr/local/elog/elogd.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f274640aa28 in __GI_abort () at abort.c:90
#2 0x00007f274644be87 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f274655e3b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f2746454679 in malloc_printerr (ar_ptr=0x7f274679a760 <main_arena>, ptr=<optimized out>,
str=0x7f274655e450 "free(): invalid next size (normal)", action=3) at malloc.c:4967
#4 _int_free (av=0x7f274679a760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e456 in display_line (lbs=0x222c968, message_id=426, number=<optimized out>,
mode=0x7fffde328c90 "summary", expand=1, level=0, printable=0, n_line=3, show_attachments=0,
show_att_column=1, date=0x7fffde328b50 "Wed, 04 Dec 2019 07:41:34 +0000",
in_reply_to=0x7fffde328ba0 "", reply_to=0x7fffde329600 "", n_attr_disp=14,
disp_attr=0x7fffde3792a0, disp_attr_link=0x7fffde329460, attrib=0x7fffde32fec0, n_attr=14,
text=0x2378d18 "<div style=\"margin: 0px; padding: 0px; text-indent: 0px;\">\r\n<table border=\"1\" cellpadding=\"1\" cellspacing=\"1\" style=\"height:230px; width:1308px\">\r\n\t<caption>\r\n\t<p style=\"text-align:left\"><strong>MAJ 2"..., show_text=1, attachment=0x7fffde32ccc0,
encoding=0x7fffde328bf0 "HTML", select=0, n_display=0x7fffde328a80, locked_by=0x7fffde328ec0 "",
highlight=0, re_buf=0x7fffde32b380, highlight_mid=0, absolute_link=0, draft=0x7fffde3290c0 "")
at src/elogd.c:18520
#6 0x0000000000482140 in show_elog_list (lbs=lbs@entry=0x222c968, past_n=past_n@entry=0,
last_n=last_n@entry=0, page_n=<optimized out>, page_n@entry=0, default_page=<optimized out>,
default_page@entry=1, info=info@entry=0x0) at src/elogd.c:21859
#7 0x00000000004a74a0 in interprete (lbook=lbook@entry=0x7fffde3c79e0 "MCO", path=<optimized out>,
path@entry=0x7fffde3c7520 "") at src/elogd.c:28454
#8 0x00000000004a7b13 in decode_get (logbook=logbook@entry=0x7fffde3c79e0 "MCO",
string=<optimized out>) at src/elogd.c:28494
#9 0x00000000004aa03a in process_http_request (request=<optimized out>,
request@entry=0x223e288 "GET /MCO/?mode=summary&reverse=0&reverse=1&npp=30&subtext=test",
i_conn=i_conn@entry=0) at src/elogd.c:29272
#10 0x00000000004ad385 in server_loop () at src/elogd.c:30286
#11 0x0000000000402e97 in main (argc=<optimized out>, argv=<optimized out>) at src/elogd.c:31314
Is it better to find problem ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan.
I installed debuginfo and ran elogd with gdb attached.
(gdb) backtrace
#0 0x00007fde38104337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fde38105a28 in __GI_abort () at abort.c:90
#2 0x00007fde38146e87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fde382593b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007fde3814f679 in malloc_printerr (ar_ptr=0x7fde38495760 <main_arena>, ptr=<optimized out>, str=0x7fde38259450 "free(): invalid next size (normal)", action=3)
at malloc.c:4967
#4 _int_free (av=0x7fde38495760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e756 in display_line ()
#6 0x00000000004824b0 in show_elog_list ()
#7 0x00000000004a69b1 in interprete ()
#8 0x00000000004a7023 in decode_get ()
#9 0x00000000004a954a in process_http_request ()
#10 0x00000000004ac9d3 in server_loop ()
#11 0x00000000004035a6 in main ()
Does it help you ?
| Stefan Ritt wrote: |
|
The only way for me is to reproduce it. Could be corrupted data file, or config file. Try with a new logbook with the same configuration, then add logbook files one by one until it happens.
Stefan
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
The problem seems to be specific with our logbooks. Maybe one of the file is corrupted or bad string encoding ?! But how to find the source of the problem ?
Any way to improve debug log ? (using debug level = 9 + logfile)
Also, do you have a procedure to debug ELOG with debuginfo package installed to have more informations in trace ?
.
Laurent
| Stefan Ritt wrote: |
|
Do you see the same problem on this server here?
If not, then it must be related to your specific configuration. You have to teach me how to reporduce the error in order to fix it. Best would be a minimal elogd.cfg which causes the error.
Stefan
| Laurent Jean-Rigaud wrote: |
|
OS: CentOS 7 x86_64 up2date with RPM 3.14 x86_64
- Connexion to ELOG
- open one logbook and click search,
- input "test" and click "Search" : BOOM
- Firefox can not connect anymore on ELOG.
The crash is met with several words now ("test", "post", "4.5", are the one i know). We use a mirror server and the problem appears also on it .
NB: the hangs appears on EL6 x86_64 server with customized RPM built by myself. I retry with 3.14 SRPMS avalaible from ELOG site with same results. Idem with last GIT version. So i tested on EL7 with official RPM x86_64 (not suitable for EL6 as it needs GLIBC_2.14) before to open this ticket.
Terminal traces on server side follows.
[root@localhost /]# /usr/local/sbin/elogd -c /usr/local/elog/elogd.cfg 2>&1
elogd 3.1.4 built Sep 26 2018, 13:14:57 revision 966e3dd
File "/var/run/elogd.pid" exists, using "/var/run/elogd.pid.8080" instead.
Falling back to default group "elog"
Falling back to default user "elog"
CKeditor detected
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
ImageMagick NOT detected. Image scaling will not work.
Indexing logbooks ... done
Server listening on port 8080 ...
*** Error in `/usr/local/sbin/elogd': free(): invalid next size (normal): 0x0000000001c59310 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7f45a898f679]
/usr/local/sbin/elogd[0x45e756]
/usr/local/sbin/elogd[0x4824b0]
/usr/local/sbin/elogd[0x4a69b1]
/usr/local/sbin/elogd[0x4a7023]
/usr/local/sbin/elogd[0x4a954a]
/usr/local/sbin/elogd[0x4ac9d3]
/usr/local/sbin/elogd[0x4035a6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f45a8930505]
/usr/local/sbin/elogd[0x4044d3]
======= Memory map: ========
00400000-004e3000 r-xp 00000000 fd:00 52822202 /usr/local/sbin/elogd
006e2000-006e3000 r--p 000e2000 fd:00 52822202 /usr/local/sbin/elogd
006e3000-007c4000 rw-p 000e3000 fd:00 52822202 /usr/local/sbin/elogd
007c4000-0173a000 rw-p 00000000 00:00 0
01a66000-01c8f000 rw-p 00000000 00:00 0 [heap]
7f459bdea000-7f459bdff000 r-xp 00000000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bdff000-7f459bffe000 ---p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bffe000-7f459bfff000 r--p 00014000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bfff000-7f459c000000 rw-p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459c000000-7f459c021000 rw-p 00000000 00:00 0
7f459c021000-7f45a0000000 ---p 00000000 00:00 0
7f45a010c000-7f45a0114000 r-xp 00000000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0114000-7f45a0313000 ---p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0313000-7f45a0314000 r--p 00007000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0314000-7f45a0315000 rw-p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0315000-7f45a0321000 r-xp 00000000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0321000-7f45a0520000 ---p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0520000-7f45a0521000 r--p 0000b000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0521000-7f45a0522000 rw-p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0522000-7f45a0528000 rw-p 00000000 00:00 0
7f45a0528000-7f45a6a52000 r--p 00000000 fd:00 50336362 /usr/lib/locale/locale-archive
7f45a6a52000-7f45a6ab2000 r-xp 00000000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6ab2000-7f45a6cb2000 ---p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb2000-7f45a6cb3000 r--p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb3000-7f45a6cb4000 rw-p 00061000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb4000-7f45a6cd8000 r-xp 00000000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6cd8000-7f45a6ed7000 ---p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed7000-7f45a6ed8000 r--p 00023000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed8000-7f45a6ed9000 rw-p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed9000-7f45a6edb000 rw-p 00000000 00:00 0
7f45a6edb000-7f45a6ef2000 r-xp 00000000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a6ef2000-7f45a70f1000 ---p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f1000-7f45a70f2000 r--p 00016000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f2000-7f45a70f3000 rw-p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f3000-7f45a70f7000 rw-p 00000000 00:00 0
7f45a70f7000-7f45a710d000 r-xp 00000000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a710d000-7f45a730c000 ---p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730c000-7f45a730d000 r--p 00015000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730d000-7f45a730e000 rw-p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730e000-7f45a7310000 rw-p 00000000 00:00 0
7f45a7310000-7f45a7313000 r-xp 00000000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7313000-7f45a7512000 ---p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7512000-7f45a7513000 r--p 00002000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7513000-7f45a7514000 rw-p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7514000-7f45a7522000 r-xp 00000000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7522000-7f45a7722000 ---p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7722000-7f45a7723000 r--p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7723000-7f45a7724000 rw-p 0000f000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7724000-7f45a7739000 r-xp 00000000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7739000-7f45a7938000 ---p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7938000-7f45a7939000 r--p 00014000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7939000-7f45a793a000 rw-p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a793a000-7f45a793c000 r-xp 00000000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a793c000-7f45a7b3c000 ---p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3c000-7f45a7b3d000 r--p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3d000-7f45a7b3e000 rw-p 00003000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3e000-7f45a7d74000 r-xp 00000000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7d74000-7f45a7f74000 ---p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f74000-7f45a7f90000 r--p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f90000-7f45a7f9d000 rw-p 00252000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f9d000-7f45a7fa1000 rw-p 00000000 00:00 0
7f45a7fa1000-7f45a7fd2000 r-xp 00000000 fd:00 321856 /usr/lib64/libk5crypto.so.3.1
Abandon [sorry, French locale]
Logbook contains 445 entries (shown in ELOG window), 384 files and 4 folders (2017 to 2020).
Thanks for help as i start to be annoying to unuse search function.
|
|
|
|
|
|
|
|
69111
|
Tue Feb 4 17:19:49 2020 |
| Stefan Ritt | stefan.ritt@psi.ch | Bug report | Linux | 3.14 RPM | Re: Elogd crashes on search | Looks like. Can you dig into the database file and have a look at that entry? Or send me the file containing that entry (together with your elogd.cfg)
Stefan
| Laurent Jean-Rigaud wrote: |
|
At #5, there is message_id=426 . You think it's this elog entries # 426 that is the source of crash ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
My previous dump is useless as your elog-debuginfo rpm is stripped.
So i rebuild elog w/ -g , reinstall it and activate core generation. I restst a search of "test" and get this in gdb corefile .
[root@localhost ~]# gdb /var/crash/core.elogd.11613.localhost.localdomain.1580820772
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 11613]
Reading symbols from /usr/local/sbin/elogd...Reading symbols from /usr/lib/debug/usr/local/sbin/elogd.debug...done.
done.
Missing separate debuginfo for
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b5/a5458535a1397fa6baaf5e8c13a6395426a1b2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/elogd -D -c /usr/local/elog/elogd.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f274640aa28 in __GI_abort () at abort.c:90
#2 0x00007f274644be87 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f274655e3b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f2746454679 in malloc_printerr (ar_ptr=0x7f274679a760 <main_arena>, ptr=<optimized out>,
str=0x7f274655e450 "free(): invalid next size (normal)", action=3) at malloc.c:4967
#4 _int_free (av=0x7f274679a760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e456 in display_line (lbs=0x222c968, message_id=426, number=<optimized out>,
mode=0x7fffde328c90 "summary", expand=1, level=0, printable=0, n_line=3, show_attachments=0,
show_att_column=1, date=0x7fffde328b50 "Wed, 04 Dec 2019 07:41:34 +0000",
in_reply_to=0x7fffde328ba0 "", reply_to=0x7fffde329600 "", n_attr_disp=14,
disp_attr=0x7fffde3792a0, disp_attr_link=0x7fffde329460, attrib=0x7fffde32fec0, n_attr=14,
text=0x2378d18 "<div style=\"margin: 0px; padding: 0px; text-indent: 0px;\">\r\n<table border=\"1\" cellpadding=\"1\" cellspacing=\"1\" style=\"height:230px; width:1308px\">\r\n\t<caption>\r\n\t<p style=\"text-align:left\"><strong>MAJ 2"..., show_text=1, attachment=0x7fffde32ccc0,
encoding=0x7fffde328bf0 "HTML", select=0, n_display=0x7fffde328a80, locked_by=0x7fffde328ec0 "",
highlight=0, re_buf=0x7fffde32b380, highlight_mid=0, absolute_link=0, draft=0x7fffde3290c0 "")
at src/elogd.c:18520
#6 0x0000000000482140 in show_elog_list (lbs=lbs@entry=0x222c968, past_n=past_n@entry=0,
last_n=last_n@entry=0, page_n=<optimized out>, page_n@entry=0, default_page=<optimized out>,
default_page@entry=1, info=info@entry=0x0) at src/elogd.c:21859
#7 0x00000000004a74a0 in interprete (lbook=lbook@entry=0x7fffde3c79e0 "MCO", path=<optimized out>,
path@entry=0x7fffde3c7520 "") at src/elogd.c:28454
#8 0x00000000004a7b13 in decode_get (logbook=logbook@entry=0x7fffde3c79e0 "MCO",
string=<optimized out>) at src/elogd.c:28494
#9 0x00000000004aa03a in process_http_request (request=<optimized out>,
request@entry=0x223e288 "GET /MCO/?mode=summary&reverse=0&reverse=1&npp=30&subtext=test",
i_conn=i_conn@entry=0) at src/elogd.c:29272
#10 0x00000000004ad385 in server_loop () at src/elogd.c:30286
#11 0x0000000000402e97 in main (argc=<optimized out>, argv=<optimized out>) at src/elogd.c:31314
Is it better to find problem ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan.
I installed debuginfo and ran elogd with gdb attached.
(gdb) backtrace
#0 0x00007fde38104337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fde38105a28 in __GI_abort () at abort.c:90
#2 0x00007fde38146e87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fde382593b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007fde3814f679 in malloc_printerr (ar_ptr=0x7fde38495760 <main_arena>, ptr=<optimized out>, str=0x7fde38259450 "free(): invalid next size (normal)", action=3)
at malloc.c:4967
#4 _int_free (av=0x7fde38495760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e756 in display_line ()
#6 0x00000000004824b0 in show_elog_list ()
#7 0x00000000004a69b1 in interprete ()
#8 0x00000000004a7023 in decode_get ()
#9 0x00000000004a954a in process_http_request ()
#10 0x00000000004ac9d3 in server_loop ()
#11 0x00000000004035a6 in main ()
Does it help you ?
| Stefan Ritt wrote: |
|
The only way for me is to reproduce it. Could be corrupted data file, or config file. Try with a new logbook with the same configuration, then add logbook files one by one until it happens.
Stefan
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
The problem seems to be specific with our logbooks. Maybe one of the file is corrupted or bad string encoding ?! But how to find the source of the problem ?
Any way to improve debug log ? (using debug level = 9 + logfile)
Also, do you have a procedure to debug ELOG with debuginfo package installed to have more informations in trace ?
.
Laurent
| Stefan Ritt wrote: |
|
Do you see the same problem on this server here?
If not, then it must be related to your specific configuration. You have to teach me how to reporduce the error in order to fix it. Best would be a minimal elogd.cfg which causes the error.
Stefan
| Laurent Jean-Rigaud wrote: |
|
OS: CentOS 7 x86_64 up2date with RPM 3.14 x86_64
- Connexion to ELOG
- open one logbook and click search,
- input "test" and click "Search" : BOOM
- Firefox can not connect anymore on ELOG.
The crash is met with several words now ("test", "post", "4.5", are the one i know). We use a mirror server and the problem appears also on it .
NB: the hangs appears on EL6 x86_64 server with customized RPM built by myself. I retry with 3.14 SRPMS avalaible from ELOG site with same results. Idem with last GIT version. So i tested on EL7 with official RPM x86_64 (not suitable for EL6 as it needs GLIBC_2.14) before to open this ticket.
Terminal traces on server side follows.
[root@localhost /]# /usr/local/sbin/elogd -c /usr/local/elog/elogd.cfg 2>&1
elogd 3.1.4 built Sep 26 2018, 13:14:57 revision 966e3dd
File "/var/run/elogd.pid" exists, using "/var/run/elogd.pid.8080" instead.
Falling back to default group "elog"
Falling back to default user "elog"
CKeditor detected
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
ImageMagick NOT detected. Image scaling will not work.
Indexing logbooks ... done
Server listening on port 8080 ...
*** Error in `/usr/local/sbin/elogd': free(): invalid next size (normal): 0x0000000001c59310 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7f45a898f679]
/usr/local/sbin/elogd[0x45e756]
/usr/local/sbin/elogd[0x4824b0]
/usr/local/sbin/elogd[0x4a69b1]
/usr/local/sbin/elogd[0x4a7023]
/usr/local/sbin/elogd[0x4a954a]
/usr/local/sbin/elogd[0x4ac9d3]
/usr/local/sbin/elogd[0x4035a6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f45a8930505]
/usr/local/sbin/elogd[0x4044d3]
======= Memory map: ========
00400000-004e3000 r-xp 00000000 fd:00 52822202 /usr/local/sbin/elogd
006e2000-006e3000 r--p 000e2000 fd:00 52822202 /usr/local/sbin/elogd
006e3000-007c4000 rw-p 000e3000 fd:00 52822202 /usr/local/sbin/elogd
007c4000-0173a000 rw-p 00000000 00:00 0
01a66000-01c8f000 rw-p 00000000 00:00 0 [heap]
7f459bdea000-7f459bdff000 r-xp 00000000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bdff000-7f459bffe000 ---p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bffe000-7f459bfff000 r--p 00014000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bfff000-7f459c000000 rw-p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459c000000-7f459c021000 rw-p 00000000 00:00 0
7f459c021000-7f45a0000000 ---p 00000000 00:00 0
7f45a010c000-7f45a0114000 r-xp 00000000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0114000-7f45a0313000 ---p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0313000-7f45a0314000 r--p 00007000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0314000-7f45a0315000 rw-p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0315000-7f45a0321000 r-xp 00000000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0321000-7f45a0520000 ---p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0520000-7f45a0521000 r--p 0000b000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0521000-7f45a0522000 rw-p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0522000-7f45a0528000 rw-p 00000000 00:00 0
7f45a0528000-7f45a6a52000 r--p 00000000 fd:00 50336362 /usr/lib/locale/locale-archive
7f45a6a52000-7f45a6ab2000 r-xp 00000000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6ab2000-7f45a6cb2000 ---p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb2000-7f45a6cb3000 r--p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb3000-7f45a6cb4000 rw-p 00061000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb4000-7f45a6cd8000 r-xp 00000000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6cd8000-7f45a6ed7000 ---p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed7000-7f45a6ed8000 r--p 00023000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed8000-7f45a6ed9000 rw-p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed9000-7f45a6edb000 rw-p 00000000 00:00 0
7f45a6edb000-7f45a6ef2000 r-xp 00000000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a6ef2000-7f45a70f1000 ---p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f1000-7f45a70f2000 r--p 00016000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f2000-7f45a70f3000 rw-p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f3000-7f45a70f7000 rw-p 00000000 00:00 0
7f45a70f7000-7f45a710d000 r-xp 00000000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a710d000-7f45a730c000 ---p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730c000-7f45a730d000 r--p 00015000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730d000-7f45a730e000 rw-p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730e000-7f45a7310000 rw-p 00000000 00:00 0
7f45a7310000-7f45a7313000 r-xp 00000000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7313000-7f45a7512000 ---p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7512000-7f45a7513000 r--p 00002000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7513000-7f45a7514000 rw-p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7514000-7f45a7522000 r-xp 00000000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7522000-7f45a7722000 ---p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7722000-7f45a7723000 r--p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7723000-7f45a7724000 rw-p 0000f000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7724000-7f45a7739000 r-xp 00000000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7739000-7f45a7938000 ---p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7938000-7f45a7939000 r--p 00014000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7939000-7f45a793a000 rw-p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a793a000-7f45a793c000 r-xp 00000000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a793c000-7f45a7b3c000 ---p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3c000-7f45a7b3d000 r--p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3d000-7f45a7b3e000 rw-p 00003000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3e000-7f45a7d74000 r-xp 00000000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7d74000-7f45a7f74000 ---p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f74000-7f45a7f90000 r--p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f90000-7f45a7f9d000 rw-p 00252000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f9d000-7f45a7fa1000 rw-p 00000000 00:00 0
7f45a7fa1000-7f45a7fd2000 r-xp 00000000 fd:00 321856 /usr/lib64/libk5crypto.so.3.1
Abandon [sorry, French locale]
Logbook contains 445 entries (shown in ELOG window), 384 files and 4 folders (2017 to 2020).
Thanks for help as i start to be annoying to unuse search function.
|
|
|
|
|
|
|
|
|
69112
|
Tue Feb 4 18:33:56 2020 |
| Laurent Jean-Rigaud | lollspam@free.fr | Bug report | Linux | 3.14 RPM | Re: Elogd crashes on search | Stefan,
I cut the log in two parts w/o modifying the content and the search runs. It seems that the size of this entrie 426 is closed to a limit (as during testing, i met a message after clicking save to recompile elog to increase a size of something), so it could be the problem.
I reduced the entrie size by extracting the last part in a new entrie and it seems to be OK.
The old size was 250099 bytes. New size is 240084.
I hope this will be OK.
Regards
| Stefan Ritt wrote: |
|
Looks like. Can you dig into the database file and have a look at that entry? Or send me the file containing that entry (together with your elogd.cfg)
Stefan
| Laurent Jean-Rigaud wrote: |
|
At #5, there is message_id=426 . You think it's this elog entries # 426 that is the source of crash ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
My previous dump is useless as your elog-debuginfo rpm is stripped.
So i rebuild elog w/ -g , reinstall it and activate core generation. I restst a search of "test" and get this in gdb corefile .
[root@localhost ~]# gdb /var/crash/core.elogd.11613.localhost.localdomain.1580820772
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 11613]
Reading symbols from /usr/local/sbin/elogd...Reading symbols from /usr/lib/debug/usr/local/sbin/elogd.debug...done.
done.
Missing separate debuginfo for
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b5/a5458535a1397fa6baaf5e8c13a6395426a1b2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/elogd -D -c /usr/local/elog/elogd.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) backtrace
#0 0x00007f2746409337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f274640aa28 in __GI_abort () at abort.c:90
#2 0x00007f274644be87 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f274655e3b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007f2746454679 in malloc_printerr (ar_ptr=0x7f274679a760 <main_arena>, ptr=<optimized out>,
str=0x7f274655e450 "free(): invalid next size (normal)", action=3) at malloc.c:4967
#4 _int_free (av=0x7f274679a760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e456 in display_line (lbs=0x222c968, message_id=426, number=<optimized out>,
mode=0x7fffde328c90 "summary", expand=1, level=0, printable=0, n_line=3, show_attachments=0,
show_att_column=1, date=0x7fffde328b50 "Wed, 04 Dec 2019 07:41:34 +0000",
in_reply_to=0x7fffde328ba0 "", reply_to=0x7fffde329600 "", n_attr_disp=14,
disp_attr=0x7fffde3792a0, disp_attr_link=0x7fffde329460, attrib=0x7fffde32fec0, n_attr=14,
text=0x2378d18 "<div style=\"margin: 0px; padding: 0px; text-indent: 0px;\">\r\n<table border=\"1\" cellpadding=\"1\" cellspacing=\"1\" style=\"height:230px; width:1308px\">\r\n\t<caption>\r\n\t<p style=\"text-align:left\"><strong>MAJ 2"..., show_text=1, attachment=0x7fffde32ccc0,
encoding=0x7fffde328bf0 "HTML", select=0, n_display=0x7fffde328a80, locked_by=0x7fffde328ec0 "",
highlight=0, re_buf=0x7fffde32b380, highlight_mid=0, absolute_link=0, draft=0x7fffde3290c0 "")
at src/elogd.c:18520
#6 0x0000000000482140 in show_elog_list (lbs=lbs@entry=0x222c968, past_n=past_n@entry=0,
last_n=last_n@entry=0, page_n=<optimized out>, page_n@entry=0, default_page=<optimized out>,
default_page@entry=1, info=info@entry=0x0) at src/elogd.c:21859
#7 0x00000000004a74a0 in interprete (lbook=lbook@entry=0x7fffde3c79e0 "MCO", path=<optimized out>,
path@entry=0x7fffde3c7520 "") at src/elogd.c:28454
#8 0x00000000004a7b13 in decode_get (logbook=logbook@entry=0x7fffde3c79e0 "MCO",
string=<optimized out>) at src/elogd.c:28494
#9 0x00000000004aa03a in process_http_request (request=<optimized out>,
request@entry=0x223e288 "GET /MCO/?mode=summary&reverse=0&reverse=1&npp=30&subtext=test",
i_conn=i_conn@entry=0) at src/elogd.c:29272
#10 0x00000000004ad385 in server_loop () at src/elogd.c:30286
#11 0x0000000000402e97 in main (argc=<optimized out>, argv=<optimized out>) at src/elogd.c:31314
Is it better to find problem ?
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan.
I installed debuginfo and ran elogd with gdb attached.
(gdb) backtrace
#0 0x00007fde38104337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007fde38105a28 in __GI_abort () at abort.c:90
#2 0x00007fde38146e87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fde382593b8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007fde3814f679 in malloc_printerr (ar_ptr=0x7fde38495760 <main_arena>, ptr=<optimized out>, str=0x7fde38259450 "free(): invalid next size (normal)", action=3)
at malloc.c:4967
#4 _int_free (av=0x7fde38495760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
#5 0x000000000045e756 in display_line ()
#6 0x00000000004824b0 in show_elog_list ()
#7 0x00000000004a69b1 in interprete ()
#8 0x00000000004a7023 in decode_get ()
#9 0x00000000004a954a in process_http_request ()
#10 0x00000000004ac9d3 in server_loop ()
#11 0x00000000004035a6 in main ()
Does it help you ?
| Stefan Ritt wrote: |
|
The only way for me is to reproduce it. Could be corrupted data file, or config file. Try with a new logbook with the same configuration, then add logbook files one by one until it happens.
Stefan
| Laurent Jean-Rigaud wrote: |
|
Hi Stefan,
The problem seems to be specific with our logbooks. Maybe one of the file is corrupted or bad string encoding ?! But how to find the source of the problem ?
Any way to improve debug log ? (using debug level = 9 + logfile)
Also, do you have a procedure to debug ELOG with debuginfo package installed to have more informations in trace ?
.
Laurent
| Stefan Ritt wrote: |
|
Do you see the same problem on this server here?
If not, then it must be related to your specific configuration. You have to teach me how to reporduce the error in order to fix it. Best would be a minimal elogd.cfg which causes the error.
Stefan
| Laurent Jean-Rigaud wrote: |
|
OS: CentOS 7 x86_64 up2date with RPM 3.14 x86_64
- Connexion to ELOG
- open one logbook and click search,
- input "test" and click "Search" : BOOM
- Firefox can not connect anymore on ELOG.
The crash is met with several words now ("test", "post", "4.5", are the one i know). We use a mirror server and the problem appears also on it .
NB: the hangs appears on EL6 x86_64 server with customized RPM built by myself. I retry with 3.14 SRPMS avalaible from ELOG site with same results. Idem with last GIT version. So i tested on EL7 with official RPM x86_64 (not suitable for EL6 as it needs GLIBC_2.14) before to open this ticket.
Terminal traces on server side follows.
[root@localhost /]# /usr/local/sbin/elogd -c /usr/local/elog/elogd.cfg 2>&1
elogd 3.1.4 built Sep 26 2018, 13:14:57 revision 966e3dd
File "/var/run/elogd.pid" exists, using "/var/run/elogd.pid.8080" instead.
Falling back to default group "elog"
Falling back to default user "elog"
CKeditor detected
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
Falling back to default group "elog"
Falling back to default user "elog"
ImageMagick NOT detected. Image scaling will not work.
Indexing logbooks ... done
Server listening on port 8080 ...
*** Error in `/usr/local/sbin/elogd': free(): invalid next size (normal): 0x0000000001c59310 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7f45a898f679]
/usr/local/sbin/elogd[0x45e756]
/usr/local/sbin/elogd[0x4824b0]
/usr/local/sbin/elogd[0x4a69b1]
/usr/local/sbin/elogd[0x4a7023]
/usr/local/sbin/elogd[0x4a954a]
/usr/local/sbin/elogd[0x4ac9d3]
/usr/local/sbin/elogd[0x4035a6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f45a8930505]
/usr/local/sbin/elogd[0x4044d3]
======= Memory map: ========
00400000-004e3000 r-xp 00000000 fd:00 52822202 /usr/local/sbin/elogd
006e2000-006e3000 r--p 000e2000 fd:00 52822202 /usr/local/sbin/elogd
006e3000-007c4000 rw-p 000e3000 fd:00 52822202 /usr/local/sbin/elogd
007c4000-0173a000 rw-p 00000000 00:00 0
01a66000-01c8f000 rw-p 00000000 00:00 0 [heap]
7f459bdea000-7f459bdff000 r-xp 00000000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bdff000-7f459bffe000 ---p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bffe000-7f459bfff000 r--p 00014000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459bfff000-7f459c000000 rw-p 00015000 fd:00 84 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f459c000000-7f459c021000 rw-p 00000000 00:00 0
7f459c021000-7f45a0000000 ---p 00000000 00:00 0
7f45a010c000-7f45a0114000 r-xp 00000000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0114000-7f45a0313000 ---p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0313000-7f45a0314000 r--p 00007000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0314000-7f45a0315000 rw-p 00008000 fd:00 1598592 /usr/lib64/libnss_sss.so.2
7f45a0315000-7f45a0321000 r-xp 00000000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0321000-7f45a0520000 ---p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0520000-7f45a0521000 r--p 0000b000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0521000-7f45a0522000 rw-p 0000c000 fd:00 22487 /usr/lib64/libnss_files-2.17.so
7f45a0522000-7f45a0528000 rw-p 00000000 00:00 0
7f45a0528000-7f45a6a52000 r--p 00000000 fd:00 50336362 /usr/lib/locale/locale-archive
7f45a6a52000-7f45a6ab2000 r-xp 00000000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6ab2000-7f45a6cb2000 ---p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb2000-7f45a6cb3000 r--p 00060000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb3000-7f45a6cb4000 rw-p 00061000 fd:00 22554 /usr/lib64/libpcre.so.1.2.0
7f45a6cb4000-7f45a6cd8000 r-xp 00000000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6cd8000-7f45a6ed7000 ---p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed7000-7f45a6ed8000 r--p 00023000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed8000-7f45a6ed9000 rw-p 00024000 fd:00 40935 /usr/lib64/libselinux.so.1
7f45a6ed9000-7f45a6edb000 rw-p 00000000 00:00 0
7f45a6edb000-7f45a6ef2000 r-xp 00000000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a6ef2000-7f45a70f1000 ---p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f1000-7f45a70f2000 r--p 00016000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f2000-7f45a70f3000 rw-p 00017000 fd:00 22495 /usr/lib64/libpthread-2.17.so
7f45a70f3000-7f45a70f7000 rw-p 00000000 00:00 0
7f45a70f7000-7f45a710d000 r-xp 00000000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a710d000-7f45a730c000 ---p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730c000-7f45a730d000 r--p 00015000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730d000-7f45a730e000 rw-p 00016000 fd:00 22497 /usr/lib64/libresolv-2.17.so
7f45a730e000-7f45a7310000 rw-p 00000000 00:00 0
7f45a7310000-7f45a7313000 r-xp 00000000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7313000-7f45a7512000 ---p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7512000-7f45a7513000 r--p 00002000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7513000-7f45a7514000 rw-p 00003000 fd:00 62649 /usr/lib64/libkeyutils.so.1.5
7f45a7514000-7f45a7522000 r-xp 00000000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7522000-7f45a7722000 ---p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7722000-7f45a7723000 r--p 0000e000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7723000-7f45a7724000 rw-p 0000f000 fd:00 321861 /usr/lib64/libkrb5support.so.0.1
7f45a7724000-7f45a7739000 r-xp 00000000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7739000-7f45a7938000 ---p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7938000-7f45a7939000 r--p 00014000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a7939000-7f45a793a000 rw-p 00015000 fd:00 40937 /usr/lib64/libz.so.1.2.7
7f45a793a000-7f45a793c000 r-xp 00000000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a793c000-7f45a7b3c000 ---p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3c000-7f45a7b3d000 r--p 00002000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3d000-7f45a7b3e000 rw-p 00003000 fd:00 22475 /usr/lib64/libdl-2.17.so
7f45a7b3e000-7f45a7d74000 r-xp 00000000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7d74000-7f45a7f74000 ---p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f74000-7f45a7f90000 r--p 00236000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f90000-7f45a7f9d000 rw-p 00252000 fd:00 303887 /usr/lib64/libcrypto.so.1.0.2k
7f45a7f9d000-7f45a7fa1000 rw-p 00000000 00:00 0
7f45a7fa1000-7f45a7fd2000 r-xp 00000000 fd:00 321856 /usr/lib64/libk5crypto.so.3.1
Abandon [sorry, French locale]
Logbook contains 445 entries (shown in ELOG window), 384 files and 4 folders (2017 to 2020).
Thanks for help as i start to be annoying to unuse search function.
|
|
|
|
|
|
|
|
|
|
69113
|
Tue Feb 11 12:12:49 2020 |
| Stefan Ritt | stefan.ritt@psi.ch | Bug report | Linux | 3.14 RPM | Re: Elogd crashes on search | Thanks for the detailed investiations and report. Finally I could reproduce the problem by having messages with a text body size close to 250000 bytes (some internal limit). Never thought that someone really has the patience to write 250'000 chars in a single message, but I guess you did some copy/paste from a large file. Thought in such cases people use attachments. Nevertheless, I fixed an internal memory allocation problem, now it shoudl be fine to have such large messages. Change is committed.
Stefan
| Laurent Jean-Rigaud wrote: |
|
Stefan,
I cut the log in two parts w/o modifying the content and the search runs. It seems that the size of this entrie 426 is closed to a limit (as during testing, i met a message after clicking save to recompile elog to increase a size of something), so it could be the problem.
I reduced the entrie size by extracting the last part in a new entrie and it seems to be OK.
The old size was 250099 bytes. New size is 240084.
I hope this will be OK.
Regards
|
|
|
69114
|
Wed Feb 12 13:19:31 2020 |
 | Stefan Ritt | stefan.ritt@psi.ch | Info | All | 3.1.4 | ELOG Null Pointer Dereference Denial-of-Service Vulnerability | An ELOG vulnerability has been reported, thanks to Asif Akbar of Trend Micro Security Researchworking with Trend Micro's Zero Day Initiative:
https://www.zerodayinitiative.com/advisories/ZDI-20-252/
The issue has been fixed in the current release 3.1.4-033e292 and in the RPM http://elog.psi.ch/elog/download/RPMS/elog-latest.x86_64.rpm
Best,
Stefan
|
|
Draft
|
Wed Feb 19 17:43:34 2020 |
 | Gino Guenzburger | gino.guenzburger@empa.ch | Bug report | All | 3.1.4 | Re: Find cannot find values with brackets | Hi Stefan
we are running elog
| Stefan Ritt wrote: |
|
I‘m happy to merge the PR after a quick test next week.
Stefan
| Sebastian Schenk wrote: |
|
For demonstration, I created https://elog.psi.ch/elogs/Linux+Demo/8
The Find search for category aaa(bb) does not give results.
A quick filter corrects the value to aaa\(bb) and delivers results.
I made a simple fix and submitted it as PR to the bitbucket repository.
|
|
|
|
69116
|
Thu Feb 20 16:54:32 2020 |
| VUIIS SysAdmin | vuiis-sysadmin@vumc.org | Question | Linux | elog.x86_64 | ELOG reverts to demo configuration | I have been using elog for over 10 years. Suddenly my elog installation has changed (probably from a recent update. My /etc/elogd.cfg has changed to the original demo configuration. Even after changing it to my configuration from backup none of the entries for the logbooks apper even though the tabs for the logboos are ther all logbooks are empty. I have looced at the logbook files and the entries seem to be there but are not showing on the web interface. Has something changed? Is there a new location for the logbooks and other files? Is there a change from the 32-bit to the 64-bit version that will cause this. I am running CentOS 7 fully patched and updated. Will there be a CentOS 8 compatible version? |
|