There is a spiral of death that makes servers to exit load balancer after a spike of requests.
Even if the request rate goes back to normal, servers might not recover for a long time.
What is the solution to have resiliency and reliability?

The problem

First, let's understand the problem. Let's imagine we have 5 servers, that can each serve a maximum of 100 requests/s. The maximum throughput that this cluster can serve is 5*100 = 500 requests/second.

What happens when we get a burst above that limit, for example 600 requests/second? In that moment, servers will receive around 600/5= 120 requests/second, that is 20% higher than their serving capacity. Some requests will fail for sure.

Among those requests, there are load balancer health-checks. Depending on your settings, there is a high chance to lose 2 consecutive healthchecks. In that moment one server will be considered unhealthy and will be temporarily taken out of the load balancing.

What happens with the other servers? Well, if 120 requests/second was too much, the remaining servers will start receiving 600/4=150 requests/second. In that moment, the chance for a second server to lose enough health-checks is close to certainty. So, in a short period, most servers will be removed from the load balancer.

Indeed, after like a minute, one of the servers will go back in the load balancer - as it finished its pending requests and it started answering healthchecks. However, not all servers are getting back to the load balancer in the same time. If only 3 servers are back into load balancer, they will be still overwhelmed by requests, and they will be soon removed again from the load balancer pool.

Even if the traffic goes below the maximum capacity of the cluster (500 req/s), the spiral of death will continue. If, for example, traffic goes back to 400 requests/second, but only 3 servers are back into load balancer, they will be overwhelmed and they to go out of the pool again.

The only solution for this problem is to reduce the incoming traffic of the cluster below 100 requests/second, so that first server can go back in the pool without being overwhelmed. Then the rest of the servers will soon join the pool. After all 5 servers are in the pool, the cluster can serve up top 500 request/second again.

The mitigation (fast solution)

You can set your health-checks to remove a server from load balancer after 3-5 lost requests. The chance to lose so many requests consecutively is close to zero even for 50% of requests that are lost. Note that an overwhelmed server will also respond to healthchecks very slow. You must also adjust the healthchecks timeouts to be higher than the server timeout for requests.

This mitigation will allow a fast recovery when the traffic goes back to normal, and might even serve a higher percent of requests. For the example above, the server might still serve around 80% of the requests as long as all servers stay in the load balancer. However, the response time might be too high for the clients to not timeout and retry - that increases the load.

Note that, if you set a high bar for the healthchecks to fail, you might keep a failing server in service for a longer time. However, this is often prefered to the spiral of death that I depicted above. You can test what value is acceptable for you, probably the sweet spot is between 3 and 5.

Remember, if servers start getting out of the load balancer, the battle is already lost. You must prevent that! Ideally your should have load test clients that can test how your servers and healthchecks are working under high pressure.

The full solution

Reliability of a serving cluster is a fine tuning process. However, there are some counter-intuitive principles that I discovered after years of testing various solutions.

Fail fast! Return 429 immediately for requests over capacity!

If you try to serve all requests, the latency will increase. The clients might timeout before your answer anyway. The server itself might become unreliable because of the queues and high memory pressure. The load will increase in a spiral due to retries.

It is better to serve only the requests that can be served in reasonable time, and reject the extra load with HTTP 429 (“Too Many Requests"). You can even set a Retry-After header, however not all clients will obey it.

The objective is to not lose server resources with requests that are doomed to fail. You "cut the leg to save the rest of the body". It sounds harsh for clients, but this is the higher good for most of the clients.

DO NOT set a very small request timeout. Once a request started, it will continue to consume capacity (for example database) even if you returned timeout to the client. This would create many fire-and-forget requests that can kill your overloaded backend - that is probably in high stress already. Don't do that! Keep the timeout just a little smaller than your clients use, so that you can still serve whoever is patient enough.

But how to decide what requests to reject?

A naive approach is to benchmark the capacity of each server, and reject any throughput that is over that capacity. While this might help sometimes, it has a big problem: you don't know exactly the limit of your servers. Maybe a downstream service answers with higher latency, so it will increase the queues on your servers. Maybe you go out of working threads because of that.

Server concurrency limit


The best solution is to count concurrency instead of requests/second. If you know you have 512 working threads, you should not accept more than 500 concurrent requests (keep some threads to reject requests). If you see that you already have 500 concurrent requests, any additional requests should be rejected with HTTP 429. That simple. This will keep your server serving requests at its maximum capacity and with reasonable latency. Any time a request finishes, a new request can be accepted to be served.

You can implement such concurrent requests counter in a Java Servlet Filter, but be careful on synchronization and not forget to decrement even on exceptions. Frameworks like Resilience4j might help.

Client/IP concurrency limits

The above solution will reduce your downtime a lot. If you want to increase the resiliency even better, the effort is a little higher and I will only touch on the methods.

Sometimes the problematic requests comes from a specific IP or there is a specific client that is producing the abnormal load. Would be fair to reject only his requests, right?

The better solution is to create concurrency buckets for clients and connecting IPs. If you only need 100 concurrent requests to serve regular traffic, even a limit as high as 400 for a client can reserve 100 threads for normal traffic of other clients. Probably a limit like 100 requests per client or IP.

Just set a reasonable concurrency limit per IP and per Client like 100 out of 500. In this way, no single client or specific client IP can block all threads. Chances are that the other clients can still be served on the remaining capacity. You need 5 problematic clients or IPs to fill the server's capacity.

You can also create buckets per requests type, like limit 100 for fast requests and limit 400 for slow requests. If you have many buckets, it is ok for the sum of the limits to be over 500. The important thing is that no single bucket can put the server on its knees.

The smaller bucket limit should be no smaller than the normal concurrency, like 50 threads. You want the limit to be 2-10 times higher than the normal concurrency to accommodate higher latency than normal. Otherwise you risk to limit clients even when there is enough capacity but a little execution delay.

If you use such buckets, a single type of request or a single client that are sending long-running requests will not block other buckets that can be served at normal latency.

In this way, you will avoid the single client of failure, and also single client IP of failure.

DO NOT try to keep the health-checks out of those concurrency limits. It happened to me for some requests to get stuck indefinitely due to a bug. If health-check if not part of the limits, the server will never be declared unhealthy and be replaced. Serve the health-check under the same limits, just increase the number of fails to declare the server unhealthy (see above).

Bulkheads - downstream concurrency limits

It is often the case that some requests can be served from cache, while others cannot be served due to a downstream system. To fail randomly the requests will also fail some requests that could have been served. The bulkhead pattern assures that no downstream service can "sink down" the whole server.

Once you created buckets for incoming requests, you can create buckets also for downstream services. You can configure concurrency limits for each downstream service. The same with input concurrency limits, you have to fail the requests before being executed, throwing an exception. DO NOT queue the requests. You can map a specific exception to HTTP 429 in the error handlers.

If you have 500 threads and p99 is 100 concurrent requests, you can allocate up to 400 requests to the most problematic downstream system. In this way, you keep 100 requests to serve the requests not needing this system. And those 100 threads will be enough 99% of the time (as in p99).

You can allocate less concurrent requests (like 100) for downstream services that are not expected to have high latency, like Redis. For a SQL database you have a number of concurrent requests to share. You might want to set lower than 100 concurrent toward SQL database to protect it from overload. Just benchmark what is the minimum concurrency that serves maximum traffic and multiply by 2-10.

On short:

  • keep the servers from exiting the load balancer under high load

  • reject fast requests after a certain server concurrency limit

  • reject fast requests after a certain Client/IP concurrency limit

  • create bulkheads toward downstream systems, so no slow service can compromise the whole server

The above are methods to keep a service resilient and assure a high SLA - even if the component parts have a smaller SLA.
Of course, the load balancer is itself a single point of failure, there are some methods also for this.

I hope this will help you to create more resilient systems!