Demo Discussion
Forum Config Examples Contributions Vulnerabilities
  Discussion forum about ELOG, Page 96 of 806  Not logged in ELOG logo
ID Date Icon Authordown Author Email Category OS ELOG Version Subject
  69550   Tue Jul 19 12:57:37 2022 Reply Tamas Galtgal@km3net.deQuestionLinux3.1.3Re: Too many redirects when running behind load balancer?

Yes, I used the empty `passwd` file from example. When I then click on one of the logbooks, I get to the page where I can register a user (see attached screenshot). After clickin on "Save" for the user registration, I again get the redirect error. Once there is a registered user (i.e. a non-empty password file) the redirect issue is persistent. Any idea where the problem might be? I just emptied the password file again, so you can have a one-shot, if you like.

Btw. I have SSL termination in the load balancer, so ELOG does not need to do any SSL related things (the swarm is in a locally isolated network, so all internal communication between the load balancer and the swarm machines are safe). Maybe that's the issue? On the other hand, the main page loads fine and uses SSL termination too, so I don't know, maybe there is logic behind the authentication which collides with the SSL termination.

Stefan Ritt wrote:

Yes I see the redirects. You say with the example logbook it works, right? Is it the password protection which triggers the problem or anything else? Does it work if you take out the password protection? The key is to identify which setting in your config file triggers the problem, so you can bracket the problem down between the example logbook and your logbook definition.

 

Attachment 1: Screenshot_2022-07-19_at_13.02.19.png
Screenshot_2022-07-19_at_13.02.19.png
  Draft   Fri Jan 20 14:08:25 2023 Question Tamas Galhe iQuestionLinux3.1.3Re: Too many redirects when running behind load balancer?

The issue is still present and now it's quite urgent to move this last service into the Swarm. Does anyone maybe have an idea what's wrong? To sum up: if there is a non-empty password file, the login page chokes in an infinite loop of redirects. I am using the same HAProxy load balancer configuration as for all the other services (running Apache, NGINX, GitLab, XWiki, etc.):

backend be_elog.km3net.de
    mode http

    option forwardfor except 127.0.0.1
    http-request add-header X-Forwarded-Proto https if { ssl_fc }

    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

Tamas Gal wrote:

Yes, I used the empty `passwd` file from example. When I then click on one of the logbooks, I get to the page where I can register a user (see attached screenshot). After clickin on "Save" for the user registration, I again get the redirect error. Once there is a registered user (i.e. a non-empty password file) the redirect issue is persistent. Any idea where the problem might be? I just emptied the password file again, so you can have a one-shot, if you like.

Btw. I have SSL termination in the load balancer, so ELOG does not need to do any SSL related things (the swarm is in a locally isolated network, so all internal communication between the load balancer and the swarm machines are safe). Maybe that's the issue? On the other hand, the main page loads fine and uses SSL termination too, so I don't know, maybe there is logic behind the authentication which collides with the SSL termination.

Stefan Ritt wrote:

Yes I see the redirects. You say with the example logbook it works, right? Is it the password protection which triggers the problem or anything else? Does it work if you take out the password protection? The key is to identify which setting in your config file triggers the problem, so you can bracket the problem down between the example logbook and your logbook definition.

 

 

  69623   Fri Jan 20 14:11:52 2023 Question Tamas Galtgal@km3net.deQuestionLinux3.1.3Re: Too many redirects when running behind load balancer?

The issue is still present and now it's quite urgent to move this last service into the Swarm. Does anyone maybe have an idea what's wrong? To sum up: if there is a non-empty password file, the login page chokes in an infinite loop of redirects. I am using the same HAProxy load balancer configuration as for all the other services (running Apache, NGINX, GitLab, XWiki, etc.):

backend be_elog.km3net.de
    mode http

    option forwardfor except 127.0.0.1
    http-request add-header X-Forwarded-Proto https if { ssl_fc }

    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

Tamas Gal wrote:

Yes, I used the empty `passwd` file from example. When I then click on one of the logbooks, I get to the page where I can register a user (see attached screenshot). After clickin on "Save" for the user registration, I again get the redirect error. Once there is a registered user (i.e. a non-empty password file) the redirect issue is persistent. Any idea where the problem might be? I just emptied the password file again, so you can have a one-shot, if you like.

Btw. I have SSL termination in the load balancer, so ELOG does not need to do any SSL related things (the swarm is in a locally isolated network, so all internal communication between the load balancer and the swarm machines are safe). Maybe that's the issue? On the other hand, the main page loads fine and uses SSL termination too, so I don't know, maybe there is logic behind the authentication which collides with the SSL termination.

Stefan Ritt wrote:

Yes I see the redirects. You say with the example logbook it works, right? Is it the password protection which triggers the problem or anything else? Does it work if you take out the password protection? The key is to identify which setting in your config file triggers the problem, so you can bracket the problem down between the example logbook and your logbook definition.

 

 

  69627   Wed Jan 25 14:36:33 2023 Warning Tamas Galtgal@km3net.deBug reportLinux3.1.4-3Invalid Content-Length in header when running behind a load balancer

I am still struggling to get ELOG running behind a load balancer and hope to get some advice here. As already reported in https://elog.psi.ch/elogs/Forum/69542 I observed an infinite loop of redirects when prompted to log in and using a non-empty password file. Without a password, the service worked as expected. This was with version 3.1.3.

With the new version 3.1.4-3, I get another error: "Invalid Content-Length in header" when I click on "submit" of a new post. Viewing the logbooks works fine. The instance is currently live and running here: https://elog.test.km3net.de but I might change it anytime due to debugging etc.

This is a kind of difficult thing to debug (I spent the whole day and no progress). The only thing I've found was this post: https://techcommunity.microsoft.com/t5/iis-support-blog/invalid-content-length/ba-p/3038724 where it seems that some responses are not RFC conform and were rejected in the load-balancer.

The load balancer I use is HAProxy, the same as in my old setup where I got the infinite redirects, and I can't find any setting which would work. To my understanding, the most basic setup should work just fine. The SSL termination is on the load-balancer side so ELOG doesn't even have to know anything about it. The configuration is below. I am running a single instance, so there is not even replication with session keep-alive via cookies or anything fancy.

I also want to mention that I am runnin around 30 different services behind the load balancer and none of them are having any issues with the SSL termination or the load-balancing itself, that's why assume that something in ELOG is either non-conform or buggy.

Any thoughts? I'd really like to use the same infrastructure for the ELOG service as for every other service (automatic certificate renewal via letsencrypt, load-balancing, easy movement to other nodes, SSL termination etc.), to minimise the complexity of our Docker Swarm system.

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

 

Btw. I am running ELOG with -v but I don't see any error whatsoever in the logs:

km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "ios_specific_templates_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_anonymous_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_user_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "logged_out_marketing_header_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3437 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes

Attachment 1: Screenshot_2023-01-25_at_14.46.05.png
Screenshot_2023-01-25_at_14.46.05.png
  69629   Wed Jan 25 18:41:27 2023 Question Tamas Galtgal@km3net.deQuestionLinux3.1.4-3Upload via command line through Apache reverse proxy and load balancer

After fiddling around I managed to get ELOG working behind the load balancer HAProxy by stacking ELOG together with an Apache reverse proxy in a Docker stack. I am currently pretty convinced that something with the HTTP communication is somehow faulty in ELOG and Apache is more forgiving than HAProxy, since the configuration is the same as without Apache. So putting ELOG behind an Apache and then Apache behind the HAProxy is working.

For the sake of completeness, here is the HAProxy configuration:

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_apache:80 check resolvers docker init-addr libc,none

and here is the Apache httpd.conf:

Listen 80

LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authn_core_module modules/mod_authn_core.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule filter_module modules/mod_filter.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule env_module modules/mod_env.so
LoadModule headers_module modules/mod_headers.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
<IfModule !mpm_prefork_module>
        #LoadModule cgid_module modules/mod_cgid.so
</IfModule>
<IfModule mpm_prefork_module>
        #LoadModule cgi_module modules/mod_cgi.so
</IfModule>
LoadModule dir_module modules/mod_dir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so


ServerAdmin email
ServerName elog.test.km3net.de

ErrorLog /proc/self/fd/2

LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    CustomLog /proc/self/fd/1 common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "logs/access_log" combined
</IfModule>

<IfModule headers_module>
    #
    # Avoid passing HTTP_PROXY environment to CGI's on this or any proxied
    # backend servers which have lingering "httpoxy" defects.
    # 'Proxy' request header is undefined by the IETF, not listed by IANA
    #
    RequestHeader unset Proxy early
</IfModule>

 

<VirtualHost *:80>
    ServerName elog.test.km3net.de
    #ProxyPreserveHost On
    ProxyPass / http://elog:8080/
    ProxyPassReverse / http://elog:8080/

    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteRule /(.*)           ws://elog:8080/$1 [P,L]
    RewriteCond %{HTTP:Upgrade} !=websocket [NC]
    RewriteRule /(.*)           http://elog:8080/$1 [P,L]

    ErrorLog /apache/error.log
    CustomLog /apache/access.log combined
    TransferLog /apache/transfer.log
</VirtualHost>

Long story short: I am still not able to upload anything from the command line. So something like

elog -v -h elog.test.km3net.de -p 443 -l "Individual Logbooks" -v -m elog_test.txt  -n 0 -a author="Whoever" -a Subject="Upload Test" -u USER PWD -s

gives this:

root@b9db27a421e1:/# elog -v -h elog.test.km3net.de -p 443 -l "Individual Logbooks" -v -m elog_test.txt  -n 0 -a author="Whoever" -a Subject="Upload Test" -u USER PWD -s
Successfully connected to host elog.test.km3net.de, port 443
Possibly invalid certificate, continue on your own risk!
Request sent to host:
POST /Individual+Logbooks/ HTTP/1.0
Content-Type: multipart/form-data; boundary=---------------------------66D92EF0673838014927FA6E
Host: elog.test.km3net.de:443
User-Agent: ELOG
Content-Length: 977


Content sent to host:
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="cmd"

Submit
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="unm"

USER
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="upwd"

PWD_HASH
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="exp"

Individual Logbooks
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="encoding"

ELCode
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="author"

Whoever
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="Subject"

Upload Test
---------------------------66D92EF0673838014927FA6E
Content-Disposition: form-data; name="Text"

foo

---------------------------66D92EF0673838014927FA6E

Response received:
HTTP/1.1 503 Service Unavailable
content-length: 107
cache-control: no-cache
content-type: text/html
connection: close

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

Error transmitting message

Is this command line interface even able to communicate through a(n Apache) reverse proxy or does it need to communicate with elogd directly?

  69630   Wed Jan 25 19:51:29 2023 Reply Tamas Galtgal@km3net.deBug reportLinux3.1.4-3Re: Invalid Content-Length in header when running behind a load balancer

I put the ELOG service behind an Apache reverse proxy which is now sitting behind the HAProxy. It works like this, but it's just a workaround. I am pretty sure that ELOG has problems to communicate with HAProxy correctly and it seems that Apache is more forgiving. So that the chain HAProxy -> Apache -> ELOG and vice versa is working.

If anyone manages to figure out what's wrong, I am happy to get rid of the extra reverse proxy layer!

Tamas Gal wrote:

I am still struggling to get ELOG running behind a load balancer and hope to get some advice here. As already reported in https://elog.psi.ch/elogs/Forum/69542 I observed an infinite loop of redirects when prompted to log in and using a non-empty password file. Without a password, the service worked as expected. This was with version 3.1.3.

With the new version 3.1.4-3, I get another error: "Invalid Content-Length in header" when I click on "submit" of a new post. Viewing the logbooks works fine. The instance is currently live and running here: https://elog.test.km3net.de but I might change it anytime due to debugging etc.

This is a kind of difficult thing to debug (I spent the whole day and no progress). The only thing I've found was this post: https://techcommunity.microsoft.com/t5/iis-support-blog/invalid-content-length/ba-p/3038724 where it seems that some responses are not RFC conform and were rejected in the load-balancer.

The load balancer I use is HAProxy, the same as in my old setup where I got the infinite redirects, and I can't find any setting which would work. To my understanding, the most basic setup should work just fine. The SSL termination is on the load-balancer side so ELOG doesn't even have to know anything about it. The configuration is below. I am running a single instance, so there is not even replication with session keep-alive via cookies or anything fancy.

I also want to mention that I am runnin around 30 different services behind the load balancer and none of them are having any issues with the SSL termination or the load-balancing itself, that's why assume that something in ELOG is either non-conform or buggy.

Any thoughts? I'd really like to use the same infrastructure for the ELOG service as for every other service (automatic certificate renewal via letsencrypt, load-balancing, easy movement to other nodes, SSL termination etc.), to minimise the complexity of our Docker Swarm system.

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

 

Btw. I am running ELOG with -v but I don't see any error whatsoever in the logs:

km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "ios_specific_templates_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_anonymous_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_user_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "logged_out_marketing_header_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3437 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes

 

  68912   Tue Mar 19 06:13:03 2019 Question Takashi Ichiharaichihara@ribf.riken.jpQuestionLinux3.1.4 Reverse proxy setting of Elog for Apache httpd 2.4 so that changing password windows works ?

In CentOS 7.6 + Apache httpd-2.4.6 + ELOG V3.1.4 environment with Reverse Proxy setting of

/etc/httpd/conf/httpd.conf
 :
ProxyRequests Off
RedirectMatch ^/elog$ /elog/
<Location /elog/>
ProxyPass                      http://mmm.riken.jp:3333/ 
ProxyPassReverse         http://mmm.riken.jp:3333/
ProxyPassReverseCookiePath   /      /elog/
</Location>
:
It almost works fine.  But Changing password has problem. 

When clicking the "Forget password?" link in login window and displaying "Entering your user name or email address"
and entering it, an email is sent to the user:

----
This is an automatically generated account recovery email for host mmmm.riken.jp.
Please click on following link to recover your account:

http://mmm.riken.jp/elog/?redir=%3Fcmd%3DChange+password%26oldpwd%3DIHCPHXNTMJGEYDKY&uname=test&upassword=IHCPHXNTMJGEYDKY

ELOG Version 3.1.4
---

Clicking the URL above in the Firefox Browser results invalid URL: 

http://mmm.riken.jp/?cmd=Change%20password&oldpwd=IHCPHXNTMJGEYDKY

These parameters does not pass to the elog by the Reverse Proxy setting above.

 While accessing the URL of (native elog port: 3333)

http://mmm.riken.jp:3333/?cmd=Change%20password&oldpwd=IHCPHXNTMJGEYDKY

displays the normal page for Changing password windows for the user.

Is there any suggestions for the Reverse Proxy setting in Apache httpd 2.4 
so that the changing password windows works fine ?

Thank you for any suggestions.

  68920   Tue Mar 26 06:41:21 2019 Smile Takashi Ichiharaichihara@ribf.riken.jpQuestionLinux3.1.4 Re: Reverse proxy setting of Elog for Apache httpd 2.4 so that changing password windows works ?

The problem was resolved. I forgot to do this statement...

https://elog.psi.ch/elog/adminguide.html
Because elogd uses links to itself (for example in the email notification and the redirection after a submit), it has to know under which URL it is running. If you run it under a proxy, you have to add the line:

    URL = http://your.proxy.host/subdir/

into elogd.cfg.


After inserting the URL in elogd.cfg, elog works correctly with Reverse Proxy Setting of Apache 2.4.

Takashi Ichihara wrote:

In CentOS 7.6 + Apache httpd-2.4.6 + ELOG V3.1.4 environment with Reverse Proxy setting of

/etc/httpd/conf/httpd.conf
 :
ProxyRequests Off
RedirectMatch ^/elog$ /elog/
<Location /elog/>
ProxyPass                      http://mmm.riken.jp:3333/ 
ProxyPassReverse         http://mmm.riken.jp:3333/
ProxyPassReverseCookiePath   /      /elog/
</Location>
:
It almost works fine.  But Changing password has problem. 

When clicking the "Forget password?" link in login window and displaying "Entering your user name or email address"
and entering it, an email is sent to the user:

----
This is an automatically generated account recovery email for host mmmm.riken.jp.
Please click on following link to recover your account:

http://mmm.riken.jp/elog/?redir=%3Fcmd%3DChange+password%26oldpwd%3DIHCPHXNTMJGEYDKY&uname=test&upassword=IHCPHXNTMJGEYDKY

ELOG Version 3.1.4
---

Clicking the URL above in the Firefox Browser results invalid URL: 

http://mmm.riken.jp/?cmd=Change%20password&oldpwd=IHCPHXNTMJGEYDKY

These parameters does not pass to the elog by the Reverse Proxy setting above.

 While accessing the URL of (native elog port: 3333)

http://mmm.riken.jp:3333/?cmd=Change%20password&oldpwd=IHCPHXNTMJGEYDKY

displays the normal page for Changing password windows for the user.

Is there any suggestions for the Reverse Proxy setting in Apache httpd 2.4 
so that the changing password windows works fine ?

Thank you for any suggestions.

 

ELOG V3.1.5-3fb85fa6