Demo Discussion
Forum Config Examples Contributions Vulnerabilities
  Discussion forum about ELOG, Page 627 of 807  Not logged in ELOG logo
ID Date Icon Author Author Email Category OSdown ELOG Version Subject
  69627   Wed Jan 25 14:36:33 2023 Warning Tamas Galtgal@km3net.deBug reportLinux3.1.4-3Invalid Content-Length in header when running behind a load balancer

I am still struggling to get ELOG running behind a load balancer and hope to get some advice here. As already reported in https://elog.psi.ch/elogs/Forum/69542 I observed an infinite loop of redirects when prompted to log in and using a non-empty password file. Without a password, the service worked as expected. This was with version 3.1.3.

With the new version 3.1.4-3, I get another error: "Invalid Content-Length in header" when I click on "submit" of a new post. Viewing the logbooks works fine. The instance is currently live and running here: https://elog.test.km3net.de but I might change it anytime due to debugging etc.

This is a kind of difficult thing to debug (I spent the whole day and no progress). The only thing I've found was this post: https://techcommunity.microsoft.com/t5/iis-support-blog/invalid-content-length/ba-p/3038724 where it seems that some responses are not RFC conform and were rejected in the load-balancer.

The load balancer I use is HAProxy, the same as in my old setup where I got the infinite redirects, and I can't find any setting which would work. To my understanding, the most basic setup should work just fine. The SSL termination is on the load-balancer side so ELOG doesn't even have to know anything about it. The configuration is below. I am running a single instance, so there is not even replication with session keep-alive via cookies or anything fancy.

I also want to mention that I am runnin around 30 different services behind the load balancer and none of them are having any issues with the SSL termination or the load-balancing itself, that's why assume that something in ELOG is either non-conform or buggy.

Any thoughts? I'd really like to use the same infrastructure for the ELOG service as for every other service (automatic certificate renewal via letsencrypt, load-balancing, easy movement to other nodes, SSL termination etc.), to minimise the complexity of our Docker Swarm system.

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

 

Btw. I am running ELOG with -v but I don't see any error whatsoever in the logs:

km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "ios_specific_templates_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_anonymous_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_user_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "logged_out_marketing_header_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3437 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes

Attachment 1: Screenshot_2023-01-25_at_14.46.05.png
Screenshot_2023-01-25_at_14.46.05.png
  69628   Wed Jan 25 17:41:30 2023 Question Giuseppe Cucinottagiuseppe.cucinotta@unifi.itQuestionLinux3.1.3ssl certificate

We obtained a certificate from let's encrypt in order to replace the self signed certificate provided with elog. We copied the new certificates replacing the older server.crt

The problem is that when restarted elog raises an error related to the fact it is looking for server.crt and it doesn't find it anymore.

I searched in elog config file in order to find a way to indicate the new certificate but I didn't find how to manage this issue.

Any suggestion?

  69629   Wed Jan 25 18:41:27 2023 Question Tamas Galtgal@km3net.deQuestionLinux3.1.4-3Upload via command line through Apache reverse proxy and load balancer

After fiddling around I managed to get ELOG working behind the load balancer HAProxy by stacking ELOG together with an Apache reverse proxy in a Docker stack. I am currently pretty convinced that something with the HTTP communication is somehow faulty in ELOG and Apache is more forgiving than HAProxy, since the configuration is the same as without Apache. So putting ELOG behind an Apache and then Apache behind the HAProxy is working.

For the sake of completeness, here is the HAProxy configuration:

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_apache:80 check resolvers docker init-addr libc,none

and here is the Apache httpd.conf:

Listen 80

LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
<IfModule !mpm_prefork_module>
        #LoadModule cgid_module modules/mod_cgid.so
</IfModule>
<IfModule mpm_prefork_module>
        #LoadModule cgi_module modules/mod_cgi.so
</IfModule>
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so


ServerAdmin email
ServerName elog.test.km3net.de

ErrorLog /proc/self/fd/2

LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    CustomLog /proc/self/fd/1 common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "logs/access_log" combined
</IfModule>

<IfModule headers_module>
    #
    # Avoid passing HTTP_PROXY environment to CGI's on this or any proxied
    # backend servers which have lingering "httpoxy" defects.
    # 'Proxy' request header is undefined by the IETF, not listed by IANA
    #
    RequestHeader unset Proxy early
</IfModule>

 

<VirtualHost *:80>
    ServerName elog.test.km3net.de
    #ProxyPreserveHost On
    ProxyPass / http://elog:8080/
    ProxyPassReverse / http://elog:8080/

    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteRule /(.*)           ws://elog:8080/$1 [P,L]
    RewriteCond %{HTTP:Upgrade} !=websocket [NC]
    RewriteRule /(.*)           http://elog:8080/$1 [P,L]

    ErrorLog /apache/error.log
    CustomLog /apache/access.log combined
    TransferLog /apache/transfer.log
</VirtualHost>

Long story short: I am still not able to upload anything from the command line. So something like

elog -v -h elog.test.km3net.de -p 443 -l "Individual Logbooks" -v -m elog_test.txt  -n 0 -a author="Whoever" -a Subject="Upload Test" -u USER PWD -s

gives this:

root@b9db27a421e1:/# elog -v -h elog.test.km3net.de -p 443 -l "Individual Logbooks" -v -m elog_test.txt  -n 0 -a author="Whoever" -a Subject="Upload Test" -u USER PWD -s
Successfully connected to host elog.test.km3net.de, port 443
Possibly invalid certificate, continue on your own risk!
Request sent to host:
POST /Individual+Logbooks/ HTTP/1.0
Content-Type: multipart/form-data; boundary=---------------------------66D92EF0673838014927FA6E
Host: elog.test.km3net.de:443
User-Agent: ELOG
Content-Length: 977


Content sent to host:
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="cmd"

Submit
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="unm"

USER
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="upwd"

PWD_HASH
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="exp"

Individual Logbooks
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="encoding"

ELCode
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="author"

Whoever
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="Subject"

Upload Test
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="Text"

foo

---------------------------66D92EF0673838014927FA6E

Response received:
HTTP/1.1 503 Service Unavailable
content-length: 107
cache-control: no-cache
content-type: text/html
connection: close

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

Error transmitting message

Is this command line interface even able to communicate through a(n Apache) reverse proxy or does it need to communicate with elogd directly?

  69630   Wed Jan 25 19:51:29 2023 Reply Tamas Galtgal@km3net.deBug reportLinux3.1.4-3Re: Invalid Content-Length in header when running behind a load balancer

I put the ELOG service behind an Apache reverse proxy which is now sitting behind the HAProxy. It works like this, but it's just a workaround. I am pretty sure that ELOG has problems to communicate with HAProxy correctly and it seems that Apache is more forgiving. So that the chain HAProxy -> Apache -> ELOG and vice versa is working.

If anyone manages to figure out what's wrong, I am happy to get rid of the extra reverse proxy layer!

Tamas Gal wrote:

I am still struggling to get ELOG running behind a load balancer and hope to get some advice here. As already reported in https://elog.psi.ch/elogs/Forum/69542 I observed an infinite loop of redirects when prompted to log in and using a non-empty password file. Without a password, the service worked as expected. This was with version 3.1.3.

With the new version 3.1.4-3, I get another error: "Invalid Content-Length in header" when I click on "submit" of a new post. Viewing the logbooks works fine. The instance is currently live and running here: https://elog.test.km3net.de but I might change it anytime due to debugging etc.

This is a kind of difficult thing to debug (I spent the whole day and no progress). The only thing I've found was this post: https://techcommunity.microsoft.com/t5/iis-support-blog/invalid-content-length/ba-p/3038724 where it seems that some responses are not RFC conform and were rejected in the load-balancer.

The load balancer I use is HAProxy, the same as in my old setup where I got the infinite redirects, and I can't find any setting which would work. To my understanding, the most basic setup should work just fine. The SSL termination is on the load-balancer side so ELOG doesn't even have to know anything about it. The configuration is below. I am running a single instance, so there is not even replication with session keep-alive via cookies or anything fancy.

I also want to mention that I am runnin around 30 different services behind the load balancer and none of them are having any issues with the SSL termination or the load-balancing itself, that's why assume that something in ELOG is either non-conform or buggy.

Any thoughts? I'd really like to use the same infrastructure for the ELOG service as for every other service (automatic certificate renewal via letsencrypt, load-balancing, easy movement to other nodes, SSL termination etc.), to minimise the complexity of our Docker Swarm system.

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

 

Btw. I am running ELOG with -v but I don't see any error whatsoever in the logs:

km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "ios_specific_templates_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_anonymous_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_user_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "logged_out_marketing_header_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3437 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes

 

  69631   Wed Jan 25 21:44:51 2023 Reply Laurent Jean-Rigaudlollspam@free.frQuestionLinux3.1.3Re: ssl certificate

Hi Giuseppe,

The new certificate files should be copy under ssl folder (/usr/local/elog/ssl or /usr/share/elog/ssl by example, closed to templates and script directories) in place of the embedded (autosigned) certificate files enclosed with ELOG source.

It seems that there is no parameter to set a custom path.

SSL = <0 | 1>
Turn on Secure Socket Layer transport. If SSL is on, one can connect via https://... to the elogd daemon. If the URL = directive is used, make sure to use https://... instead of http://... there. The ELOG distribution contains a simple self-signed certificate in the ssl subdirectory. One can replace this certificate and key with a real ceritficate to avoid browser pop-up windows warning about the self-signed certificate. The default for this option is 0.

 

 

Giuseppe Cucinotta wrote:

We obtained a certificate from let's encrypt in order to replace the self signed certificate provided with elog. We copied the new certificates replacing the older server.crt

The problem is that when restarted elog raises an error related to the fact it is looking for server.crt and it doesn't find it anymore.

I searched in elog config file in order to find a way to indicate the new certificate but I didn't find how to manage this issue.

Any suggestion?

 

  69632   Wed Jan 25 22:22:07 2023 Reply Giuseppe Cucinottagiuseppe.cucinotta@unifi.itQuestionLinux3.1.3Re: ssl certificate

Hi Laurent,

thanks very much! Probably I've copied the certificate in the wrong directory. I'll try ASAP

Laurent Jean-Rigaud wrote:

Hi Giuseppe,

The new certificate files should be copy under ssl folder (/usr/local/elog/ssl or /usr/share/elog/ssl by example, closed to templates and script directories) in place of the embedded (autosigned) certificate files enclosed with ELOG source.

It seems that there is no parameter to set a custom path.

SSL = <0 | 1>
Turn on Secure Socket Layer transport. If SSL is on, one can connect via https://... to the elogd daemon. If the URL = directive is used, make sure to use https://... instead of http://... there. The ELOG distribution contains a simple self-signed certificate in the ssl subdirectory. One can replace this certificate and key with a real ceritficate to avoid browser pop-up windows warning about the self-signed certificate. The default for this option is 0.

 

 

Giuseppe Cucinotta wrote:

We obtained a certificate from let's encrypt in order to replace the self signed certificate provided with elog. We copied the new certificates replacing the older server.crt

The problem is that when restarted elog raises an error related to the fact it is looking for server.crt and it doesn't find it anymore.

I searched in elog config file in order to find a way to indicate the new certificate but I didn't find how to manage this issue.

Any suggestion?

 

 

  69636   Tue Jan 31 17:40:22 2023 Warning Frank HeyrothnoBug reportLinux3.1.5-1*.cxx files missing in elog-3.1.5-1.tar.gz
I downloaded the last elog-3.1.5-1.tar.gz to install elog.

All *.cxx files are missing.
  69637   Wed Feb 1 11:13:21 2023 Warning James Smallcombesmallcombe.james@jaea.go.jpQuestionLinux3.1.4CKeditor Settings Cant Be Changed

I wanted to change some CKeditor settings so tried modifying elog/scripts/ckeditor to no avail.

I wiped elog/scripts/ and dropped a fresh download of CKeditor4, with only the basic extensions. But when I open the elog it still shows the full toolbar, with elog default style and with all extensions operational.

If I leave elog/scripts empty, I get "CKeditor NOT detected" when starting elogd and the HTML option is empty and shows nothing, all as expected.

Does anyone understand this? Is there some CKeditor configuration file elog is defering to that I've overlooked? I have tried system wide seaches just in case.

ELOG V3.1.5-3fb85fa6