Demo Discussion
Forum Config Examples Contributions Vulnerabilities
  Discussion forum about ELOG  Not logged in ELOG logo
icon4.gif   Invalid Content-Length in header when running behind a load balancer, posted by Tamas Gal on Wed Jan 25 14:36:33 2023 Screenshot_2023-01-25_at_14.46.05.png
    icon2.gif   Re: Invalid Content-Length in header when running behind a load balancer, posted by Tamas Gal on Wed Jan 25 19:51:29 2023 
Message ID: 69630     Entry time: Wed Jan 25 19:51:29 2023     In reply to: 69627
Icon: Reply  Author: Tamas Gal  Author Email: tgal@km3net.de 
Category: Bug report  OS: Linux  ELOG Version: 3.1.4-3 
Subject: Re: Invalid Content-Length in header when running behind a load balancer 

I put the ELOG service behind an Apache reverse proxy which is now sitting behind the HAProxy. It works like this, but it's just a workaround. I am pretty sure that ELOG has problems to communicate with HAProxy correctly and it seems that Apache is more forgiving. So that the chain HAProxy -> Apache -> ELOG and vice versa is working.

If anyone manages to figure out what's wrong, I am happy to get rid of the extra reverse proxy layer!

Tamas Gal wrote:

I am still struggling to get ELOG running behind a load balancer and hope to get some advice here. As already reported in https://elog.psi.ch/elogs/Forum/69542 I observed an infinite loop of redirects when prompted to log in and using a non-empty password file. Without a password, the service worked as expected. This was with version 3.1.3.

With the new version 3.1.4-3, I get another error: "Invalid Content-Length in header" when I click on "submit" of a new post. Viewing the logbooks works fine. The instance is currently live and running here: https://elog.test.km3net.de but I might change it anytime due to debugging etc.

This is a kind of difficult thing to debug (I spent the whole day and no progress). The only thing I've found was this post: https://techcommunity.microsoft.com/t5/iis-support-blog/invalid-content-length/ba-p/3038724 where it seems that some responses are not RFC conform and were rejected in the load-balancer.

The load balancer I use is HAProxy, the same as in my old setup where I got the infinite redirects, and I can't find any setting which would work. To my understanding, the most basic setup should work just fine. The SSL termination is on the load-balancer side so ELOG doesn't even have to know anything about it. The configuration is below. I am running a single instance, so there is not even replication with session keep-alive via cookies or anything fancy.

I also want to mention that I am runnin around 30 different services behind the load balancer and none of them are having any issues with the SSL termination or the load-balancing itself, that's why assume that something in ELOG is either non-conform or buggy.

Any thoughts? I'd really like to use the same infrastructure for the ELOG service as for every other service (automatic certificate renewal via letsencrypt, load-balancing, easy movement to other nodes, SSL termination etc.), to minimise the complexity of our Docker Swarm system.

backend be_elog.km3net.de
    mode http
    server-template km3net-elog- 1 km3net-elog_elog:8080 check resolvers docker init-addr libc,none

 

Btw. I am running ELOG with -v but I don't see any error whatsoever in the logs:

km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "ios_specific_templates_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_anonymous_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_group_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_trait"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "rl_user_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Received unknown cookie "logged_out_marketing_header_id"
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3437 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET / HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 120 bytes
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | GET /demo/ HTTP/1.1
km3net-elog_elog.1.fm8i1eia9l9t@ecap-s021    | Returned 3518 bytes

 

ELOG V3.1.5-fe60aaf