Server capacity: workers, queues and requests per second
A server can finish only so many requests per second. The limit depends on the number of workers and on the time each request takes.
A concert ticket sale opens at noon. You load the site and a waiting page appears, asking you to stay on it. A few minutes later the page lets you in, or it shows an error. Thousands of people loaded the same site in the same minute.
Throughput cannot pass the maximum rate under Machine and load. Press Play, then raise the arrival rate above that maximum. Watch the queue and the p95 latency.
4 ÷ 0.05 s = 80 rps
More detail
Add a memory limit
Each request being handled or waiting uses RAM. Lower Total RAM or raise RAM per request, then press Play. The Memory gauge shows when the limit is reached.
Definitions
Latency is the time from a request arriving to its answer being finished. Utilization is the arrival rate divided by the most the workers can finish per second. When arrivals exceed that maximum, the queue grows, latency keeps rising and throughput stops increasing. The workers here are the same separate processes as in Server memory. Several machines are covered in Load balancers and scaling.
Every request that is being handled or waiting also uses RAM. When the machine runs out of RAM, the operating system stops processes. Extra workers then make things worse, because each one needs RAM too.
You can now find a server's maximum rate and name what limits it.
More detail
Test the memory limit
Go back to "Find the maximum rate", open its More detail and lower Total RAM.
The smallest web server
Below is the smallest possible web server, written in Python with no framework. It waits for a connection, prints the request it receives and sends back a short answer. To try it, save it as a file, run it, then run the curl command from the first comment in a second terminal window.