Scaling out with a load balancer

When one machine cannot handle all the requests, you can use a bigger machine or several machines. Several machines need a load balancer.

On enlistment day you type the address of the online enlistment system and the page loads. A classmate types the same address on another laptop, and so do thousands of other students. A site with that many visitors usually runs on several machines behind that one address.

Students' browsersSend requests
One addressLoad balancer
Application machinesServers
Think first. Your site runs on two machines, and users type one web address. What must happen so that every request reaches one of the two machines?
Students should arrive at one address that answers for all the machines, which is the load balancer. The browser never learns how many machines there are. Ask what happens if that single address itself fails.

There are two ways to get more capacity. Read the two cards side by side.

Scale up

Buy a bigger machine. It has a size limit, and one failure takes the whole site down.

Scale out

Run several copies behind a load balancer. Each copy must be able to answer any request.

More detail

Scaling up (vertical scaling)

Add more RAM and more processor cores to the single machine. Server memory covers the memory side. This needs no code changes, so it is the first thing to try. The largest machine you can rent has a fixed size, and the price rises faster than the capacity. One machine is also a single point of failure: if it breaks, the whole site is down.

Scaling out (horizontal scaling)

These are identical copies of the machine from Server capacity, with a load balancer in front deciding which copy gets each request. There is no fixed upper limit, and one machine failing no longer takes the site down. The application has to meet one condition: every copy must be able to answer any request, because you no longer control which copy receives it.

A load balancer has one public address and passes each request to one machine behind it. Press Play. Machine E is four times slower than the others, so watch its in-progress count, then switch to Least connections.

Load balancer
accepts every request and forwards each one to one machine
How it chooses a machine
More detail

A load balancer is a reverse proxy. It has one public address, accepts each connection from a client and opens a second connection to one of the backends, which are the machines behind it. Clients never see the backends. Every few seconds it sends each backend a health check, which is a small request to see whether the backend answers. It stops sending work to any backend that does not answer.

A session is the record that keeps a user logged in. Sticky sessions send the same client to the same machine every time, which is needed if a machine keeps that record in its own memory. Switch it on, send traffic, then take E down.

Requests sent
0
Difference in counts
—
Sessions lost
0

A bottleneck limits everything else. The load is 400 requests per second. Add machines, then a cache and read replicas, and watch the bottleneck move.

Clients
400rps
→
Load balancer
LBround robin
→
Application machines
80rps maximum
→
Cache
—hit rate
→
Database
300qps maximum
0read replicas
Served
0rps
System maximum
0rps
Turned away
0rps

You can now find the part that limits a system and say what to add.

More detail

The system starts at 400 requests per second (rps). Each application machine is a copy of the machine in Server capacity. The database is measured in queries per second (qps). A queue of background jobs lets a request finish before slow work, such as sending an email, is done. An application that keeps no user data in the machine is called stateless, and scaling out depends on that. Turn on the last switch with several machines and see what happens.

Database load
0qps