Server capacity: workers, queues and requests per second

A server can finish only so many requests per second. The limit depends on the number of workers and on the time each request takes.

A concert ticket sale opens at noon. You load the site and a waiting page appears, asking you to stay on it. A few minutes later the page lets you in, or it shows an error. Thousands of people loaded the same site in the same minute.

BrowsersRequests
Waiting requestsQueue
The ticket site's serverWorkers
Think first. If a server has four workers and each request takes 200 milliseconds, how many requests can it finish in one second?
The answer is 4 workers divided by 0.2 seconds, which is 20 requests per second. Watch for students who multiply the two numbers. Ask what they expect p95 latency to do when arrivals reach 30 per second.

Throughput cannot pass the maximum rate under Machine and load. Press Play, then raise the arrival rate above that maximum. Watch the queue and the p95 latency.

Waiting in queue Completed Rejected
Throughput
0rps
Queue depth
0
p95 latency
0ms
Worker utilization
0%
IdlePress Play to send requests to the server.
Machine and load
Maximum rate = workers ÷ handler time.
4 ÷ 0.05 s = 80 rps
More detail

Add a memory limit

Each request being handled or waiting uses RAM. Lower Total RAM or raise RAM per request, then press Play. The Memory gauge shows when the limit is reached.

Memory
0MB
Rejected
0

Definitions

Latency is the time from a request arriving to its answer being finished. Utilization is the arrival rate divided by the most the workers can finish per second. When arrivals exceed that maximum, the queue grows, latency keeps rising and throughput stops increasing. The workers here are the same separate processes as in Server memory. Several machines are covered in Load balancers and scaling.

Every request that is being handled or waiting also uses RAM. When the machine runs out of RAM, the operating system stops processes. Extra workers then make things worse, because each one needs RAM too.

You can now find a server's maximum rate and name what limits it.

More detail

Test the memory limit

Go back to "Find the maximum rate", open its More detail and lower Total RAM.

The smallest web server

Below is the smallest possible web server, written in Python with no framework. It waits for a connection, prints the request it receives and sends back a short answer. To try it, save it as a file, run it, then run the curl command from the first comment in a second terminal window.