Damn do I feel stupid, and VNLib.Net.Http needs a little more logging. For over a year my experimental webservers have been dropping transport connections randomly behind my load balancer. Literally a config issue, I limited the number of headers received too low and it was dropping the connection (my preference as per the RFC). So I've been serving 502s randomly to clients who were sending over 15 http request headers... I finally took the time to catch it. It was a config issue, not a bug XD
Discussion
Why 502? That suggests server error vs "silly client sent too many headers" error. Always good to log and investigate 5xx
The upstream/backend server was capped to 15. When a request is considered "malformatted" I drop the transport and free resources. The LB saw the connection drop and considered it a gateway error. So that's correct.
Ah, that makes sense. If the LB is HTTP aware, you want to make sure that the app server isn't more restrictive. Didn't the LB retry the request until it ran out of retries?
These situations are somewhat dangerous because client requests will eventually cause the LB to think servers are unhealthy until the whole cluster is down. If an attacker notices this, it's an easy DoS
I felt like the appropriate way to handle "mal-formatted" requests was to drop the transport to be safe, however yeah, it was more restrictive. During my migration I added a couple more headers and it threw it over the limit.
I just use RR upstreams in nginx so it just keeps trying, but I can definitely see where that could be an issue in another configuration.
Very large tech companies still get this wrong so 🤷🏻♂️