Load Balancing

Picture a tea shop with one counter and one person serving. On a quiet morning it is fine. At 8 a.m. the queue reaches the street, the person at the counter is sweating, and half the customers give up and walk away.

The owner opens three more counters. That helps, but only if somebody stands at the door and tells each customer which counter to go to. Without that person, everyone still crowds the first counter because it is the one they can see.

That person at the door is a load balancer. Your servers are the counters. This note is about how the person at the door decides, what happens when a counter shuts down, and what happens if the person at the door is the one who falls sick.

What a load balancer does

A load balancer is a piece of software or hardware that sits in front of a group of identical servers. Clients talk to the load balancer. The load balancer picks one server, passes the request along, and passes the answer back.

Clients only know one address, the address of the load balancer. They have no idea there are three servers behind it, or thirty. That is the whole trick, and it gives you four things for free.

• Traffic is shared, so no single server melts.
• You can add or remove servers without telling any client.
• A dead server stops getting traffic, so users do not see errors (more on this under health checks).
• You can deploy a new version one server at a time without any downtime.

Many load balancers also do a few side jobs, like handling HTTPS so the servers behind do not have to, compressing responses, or limiting how fast one client can send requests. Do not worry about those for now. The main job is picking a server.

One more thing before we go on. A load balancer only helps if the servers are interchangeable. If server 1 holds something that server 2 does not know (a user's login session, an uploaded file on its local disk), then sending the next request to server 2 breaks things. We will come back to this when we talk about sticky sessions.

Layer 4 and layer 7

You will hear "L4 load balancer" and "L7 load balancer" a lot. The numbers come from a model that describes how network traffic is built in layers. You do not need the whole model. You only need to know two of the layers.

Layer 4 is the transport layer. At this level all that exists is an IP address, a port number and a stream of bytes (TCP or UDP). Layer 7 is the application layer. At this level you can see the actual HTTP request: the URL, the headers, the cookies, the body.

Think of a post office sorting letters. An L4 balancer is a clerk who only reads the address on the envelope and drops it in the right bin. Quick, and he never opens anything. An L7 balancer is a clerk who opens the envelope, reads the letter, and decides based on what it says. Slower, but he can make much smarter choices.

A small example of why L7 matters. Say your site has a normal API and also a video upload service. They run on different groups of servers. An L7 balancer can look at the path and send /upload to the upload group and everything else to the API group, all behind one domain. An L4 balancer cannot do that, because it never sees the path.

My honest advice: for a normal web app, start with L7. The extra cost is tiny for most traffic, and the routing and visibility you get are worth it. Pick L4 when you are moving something that is not HTTP, when you are at truly huge volume, or when you want a fast, dumb front layer that spreads traffic over a fleet of L7 balancers. Big systems often do exactly that: L4 in front, L7 behind it.

Algorithms

The algorithm is just the rule the balancer uses to pick a server. There are four you must know, and a couple of extras worth a sentence.

Round robin

Go down the list one by one and loop back to the start. With three servers, requests go 1, 2, 3, 1, 2, 3, and so on.

Say six requests arrive. Each server gets exactly two. Simple, fair on paper, and needs no memory of what is happening on the servers.

It is the right choice when the servers are the same size and the requests are all about the same amount of work. Think of an API where every call takes roughly 20 ms.

It goes wrong when requests are not equal. If one request asks for a huge report that takes 10 seconds and the next 99 are tiny, round robin does not care. It will keep handing new work to the server that is stuck on the report, because it is "its turn".

Least connections

The balancer keeps a count of how many requests each server is working on right now, and sends the new one to the server with the fewest.

Say server A has 12 open requests, B has 3 and C has 7. The next request goes to B. If A is stuck on slow work, its count stays high and it naturally gets fewer new requests.

This is the right choice when requests take very different amounts of time, or when connections stay open a long time, like chat or file downloads. It is usually the better default than round robin for real traffic. The cost is that the balancer has to track the counts, which is cheap but not free.

Weighted

Sometimes servers are not the same. Say you have two big machines and one small old one. You give the big ones a weight of 3 and the small one a weight of 1. Out of every 7 requests, the big ones take 3 each and the small one takes 1.

Weights can be combined with either of the above: weighted round robin, or weighted least connections. Use it when you have mixed hardware, or when you are slowly sending traffic to a new version. Giving the new server a weight of 1 against 9 means about 10 percent of users test it first.

IP hash

Take the client's IP address, run it through a hash function, and use the result to pick a server. A hash function turns any input into a number, and the same input always gives the same number. So the same client always lands on the same server.

server = hash(client IP) mod number of servers

Use it when you want the same user to keep hitting the same server without cookies, for example a server that keeps a warm cache for each user. It also works for L4 balancers, since the IP is all they can see.

There are two problems. First, many users can share one IP (a whole office behind one router), so one server gets a lump of traffic. Second, and worse, if you add or remove a server, the "mod number of servers" changes, and almost every client is suddenly sent somewhere new. The next note, on consistent hashing, is about fixing exactly that.

A few others

Random just picks any server, and with enough traffic it evens out. Least response time sends traffic to whichever server has been answering fastest lately. Both exist in most products, and you can mention them if the interviewer asks "what else is there". Do not spend time on them.

Which one to say out loud

If nobody tells you anything about the traffic, say "least connections, because I do not know that requests are equal" and move on. That sounds like someone who has thought about it.

Health checks

Servers die. Memory runs out, a deploy goes wrong, a machine is turned off by accident. If the balancer keeps sending requests to a dead server, one third of your users (with three servers) get errors.

So the balancer keeps checking on each server. There are two ways.

An active check means the balancer sends a small request on its own, every few seconds, for example GET /health. If the server answers with 200, it is fine. If not, it is suspect.

A passive check means the balancer just watches the real traffic. If a server starts returning errors or timing out on actual user requests, the balancer marks it as bad.

Most setups use active checks, and some add passive ones on top because they notice trouble between two active checks.

One more detail that people forget. A single failed check should not remove a server, because one packet can get lost for no reason. Typical settings are something like "check every 5 seconds, take the server out after 3 failures in a row, put it back after 2 successes in a row". That means a dead server is out within about 15 seconds. You can make it faster, but the faster you go the more false alarms you get.

What should /health check?

The mistake almost everyone makes here is putting a database check inside the health endpoint. It feels responsible. "If the database is down, the server is not healthy."

Now picture the database slowing down for ten seconds. Every server fails its health check at the same time. The balancer removes all of them. Now you have zero servers, and users see a total outage, even though the servers could have served cached pages or a friendly error.

I would keep the health endpoint shallow: "the process is up and can answer". Check deeper things in your monitoring, not in the check that decides whether a server lives or dies. If you do want a deep check, at least make the balancer keep a minimum number of servers in, even if they all look bad.

Draining

When you deploy, you do not want to cut off requests that are half done. The balancer can mark a server as "draining": it stops sending new requests, waits for the running ones to finish (say up to 30 seconds), and only then is the server stopped. Cloud balancers call this connection draining or deregistration delay. Mention it in an interview when you talk about deployments, it shows you know how it works in practice.

Sticky sessions

A session is the memory a server keeps about a logged in user: who they are, what is in their cart. Say your server keeps that memory in its own RAM. If the user's first request goes to server 1 and the second goes to server 2, server 2 has never heard of them and asks them to log in again.

The quick fix is sticky sessions (also called session affinity). The balancer remembers that this user belongs to server 1 and keeps sending them there. It does this either with a cookie that the balancer sets, or with IP hash.

It works, and that is why people use it. But I would avoid it if you can, for four reasons.

• Load gets uneven. One user who does a lot of work keeps hammering the same server, and the balancer is not allowed to move them.
• When that server dies, every user stuck to it loses their session, and they all get kicked out at once.
• Adding a server does not help old users. They stay stuck to the old servers, and the new one only gets newcomers.
• Deploys get painful. Restarting a server logs out everyone who was on it.

The better design is to make the servers stateless. Keep the session somewhere every server can reach, like Redis or a database, or put the session in a signed token that the client sends back each time. Then any server can handle any request and the balancer can use plain least connections.

There are cases where stickiness is fine. A WebSocket connection stays open to one server for its whole life anyway. Some old applications cannot be changed. And sometimes you want a user's requests to hit a server that already has their data warm in a local cache. In those cases, use it on purpose and know what you are paying.

The load balancer is a single point of failure

Here is the awkward part. We put a load balancer in front of the servers so that no single server can take us down. But now every request goes through the load balancer. If it dies, everything is down, and we are worse off than before. Back to the tea shop: if the person at the door goes home sick, nobody gets into any counter.

The fix is the same as for the servers: have more than one. There are two common ways.

Active and passive pair

You run two balancers. One is active and does all the work. The other is passive and just waits. They send each other a small "I am alive" message every second or so, called a heartbeat.

Clients are not given the address of either machine. They are given a shared address, a virtual IP, that points to whoever is active. If the passive one stops hearing the heartbeat, it takes over the virtual IP and starts serving. Failover takes a few seconds. Tools like keepalived do this on normal Linux machines.

The downside is that the standby sits idle, and that is half your money doing nothing.

Active and active

Both balancers take traffic at the same time. Clients are spread over them using DNS (the name returns two addresses) or a trick called anycast, where several machines in different places announce the same IP and the network sends each client to the closest one. If one balancer dies, the other still works and takes the whole load. It needs to be big enough to do that.

In practice, if you are on a cloud provider, you do not build any of this. A managed load balancer from AWS, Google Cloud or Azure is already spread over several machines and several data centres, and you just pay for it. In an interview it is enough to say "the load balancer is made redundant, either an active passive pair with a virtual IP or several active ones behind DNS, and in the cloud it is a managed service that already does this".

DNS and global load balancing

Everything so far is about balancing inside one place, one data centre or one region. When you have users all over the world, you also need to decide which region they go to.

DNS is the phone book of the internet. A client asks "what is the address of shop.example.com" and DNS answers with an IP. The cheapest form of balancing is to answer with different IPs.

Plain DNS round robin returns a list of addresses and rotates the order. It is cheap and dumb: it has no idea if any of those addresses is dead.

Better is geo DNS (or latency based DNS). The DNS service looks at where the question came from and answers with the closest region. A user in Mumbai gets the address of your Singapore region, a user in Berlin gets Frankfurt. Many DNS services also run health checks, and stop returning the address of a region that has gone down. That is how you survive a whole region failing.

The gotcha is caching. DNS answers carry a TTL (time to live), which tells everyone how long they may remember the answer. Phones, routers and your users' providers all cache. If the TTL is 1 hour and a region dies, some users keep going to the dead region for up to an hour. So for records that need quick failover, people set a short TTL, like 30 to 60 seconds. The cost is more DNS traffic.

So a large system usually has two levels.

DNS picks the region. The balancer inside that region picks the server.

Interview talking points

When you draw the box labelled "load balancer", say a few things out loud and you will sound like someone who has used one.

1. What it does: "It spreads requests over identical stateless servers and removes dead ones."
2. L4 or L7, and why: "L7, because I want path based routing and to read headers."
3. The algorithm: "Least connections, since request times vary."
4. How it finds dead servers: "Active health checks on a shallow endpoint."
5. How it stays alive itself: "A pair with a virtual IP, or the cloud managed one."

If you are asked where the load balancer goes, the honest answer is "more than one place". One in front of the web servers, maybe another in front of an internal service group, and DNS in front of the regions.

Common slips to avoid. Do not draw one balancer and forget that it can fail. Do not say "sticky sessions" as the answer to sessions without saying what it costs. Do not put a database check in the health endpoint. And do not call it a "reverse proxy" and "load balancer" as if they are rivals: a load balancer is a kind of reverse proxy that has more than one place to send the request.

Interview questions to check yourself

"What is the difference between a layer 4 and a layer 7 balancer?"

L4 only sees IP, port and bytes, so it is fast but simple. L7 reads the HTTP request, so it can route by path, header or cookie.

"Requests on your service vary from 5 ms to 8 seconds. Round robin or least connections?"

Least connections. Round robin keeps handing work to a server that is stuck on a slow one.

"Why do sticky sessions hurt?"

Load gets uneven, a server crash logs out everyone stuck to it, new servers do not help old users, and deploys are painful. Keep sessions in a shared store instead.

"How can a load balancer be a single point of failure, and how do you fix it?"

Every request goes through it, so if it dies all is down. Run two, as an active passive pair sharing a virtual IP or as two active ones behind DNS, or use a managed cloud balancer.

"Why is a database check inside the health endpoint a risky idea?"

If the database slows down, every server fails the check together, the balancer removes them all, and a small problem becomes a full outage.