Every internet application starts with a simple question: how do two programs talk to each other? This post covers five essential networking topics every system designer must know. You will start with Client & Server — the foundation of every web and mobile application. Then you will learn how machines find and reach each other through IP Address & DNS — the addressing and naming system of the internet. From there you will explore HTTP / HTTPS, the communication protocol that clients and servers use to exchange data, followed by TCP vs UDP, the two transport protocols that trade reliability against speed. Finally, you will understand Latency & Throughput — the two performance metrics that determine how fast and how much your system can handle. Each topic is explained with real-world analogies, step-by-step examples, and clear diagrams — starting from zero.

The client-server model is where every system design journey begins. Before you can understand load balancers, databases, CDNs, or APIs, you need to understand this one idea: a client asks, a server answers. In this section, we will build that understanding from the ground up — with analogies, step-by-step examples, and clear diagrams — so that every concept that follows makes intuitive sense.

Imagine you type netflix.com into your browser. Within milliseconds, your browser has contacted Netflix's servers, authenticated your account, fetched a personalised list of movies, and started streaming a video — all without you doing anything beyond pressing Enter. That entire sequence is the client-server model in action.

Your browser is the client — the program that asks for something. Netflix's backend systems are the servers — the programs that listen for requests and send back replies. Every time you open a website, send a WhatsApp message, order on Amazon, or call an Uber, this exact exchange is happening behind the scenes.

Every single system design problem starts with the same question: how does the user's device communicate with the backend? Whether you are designing Instagram (2 billion users), Uber (150 million users), or a simple URL shortener, the answer always begins with the client-server model. Everything else — load balancers, databases, caches, CDNs — exists to make this basic exchange faster, more reliable, and capable of handling millions of users at once.

• Without understanding clients and servers, you cannot reason about how requests reach your system.
• Load balancers only make sense when you understand that many clients send requests to multiple servers.
• CDNs only make sense when you understand that static files are served from servers closer to the client.
• Microservices only make sense when you understand that a server can itself be a client to another server.

Foundation first: In system design, strong answers always start with the simple request-response path and then evolve. Never jump straight to "Kafka, Redis, sharding, microservices" — always start with "a client sends a request to a server."

Think of a restaurant. You walk in, sit at a table, and look at the menu. When you are ready, you call the waiter over and place your order. The waiter goes to the kitchen, the kitchen prepares your food, and the waiter brings it back to your table.

Notice a key point: you never go to the kitchen yourself. You send a request through the waiter, and the waiter brings back the result. This is exactly how a client and server communicate — the client never directly touches the database or business logic; it only talks to the server.

Remember: A server is a program, not a physical machine. Your laptop can run a server. One physical machine can run dozens of server programs at the same time on different ports.

Let us walk through exactly what happens when you type amazon.com in your browser and press Enter. This is the most important request path to understand in system design.

Key insight: The entire exchange above — from Step 1 to Step 7 — typically happens in under 500 milliseconds. For large-scale systems like Amazon, this same process happens for millions of users simultaneously, which is why concepts like load balancers, caches, and CDNs become necessary.

"Client" and "server" are roles, not fixed things. The same program can be a server to some callers and a client to others. Here are the most common types you will encounter in system design.

The "server as client" pattern: In modern microservices architectures, almost every service acts as both a server (to the services that call it) and a client (to the services it calls). For example, Instagram's Feed Service is a server to the mobile app, but it's a client to the User Service, Media Service, and Recommendation Service.

The diagram below shows the core client-server request-response cycle — a client sends a request, the server processes it and queries the database, and the response travels back. This is the fundamental pattern behind every internet application.

Reading the diagram: The client sends an HTTP Request ① to the server. The server queries ② the database for the data it needs, the database returns ③ the result, and the server sends an HTTP Response ④ back to the client. Every internet interaction follows this four-step cycle.

The client-server model is the default choice for virtually every internet application. You should use it whenever you have a centralized resource to share, business logic to protect, or data that needs to be consistent across users.

Rule of thumb: If you are designing any application for users — social media, e-commerce, banking, streaming, messaging — use client-server. If you are designing a decentralised protocol or file-sharing network, consider Peer-to-Peer. In practice, 99% of system design problems use client-server.

When you open Instagram on your phone and scroll through your feed, here is what happens behind the scenes:

Notice: Your phone (one client) communicated with five different server types — Load Balancer, API Server, Cache Server, Database Server, and CDN Server — all within a single feed load. This is how real large-scale systems work: many specialised servers working together to serve one client request.

New terms above? Load Balancer, Cache Server, and CDN will each get their own dedicated post in Phase 2 of this series. For now, just notice that a single client request touches multiple server types — that is the key insight from this example.

• Client initiates, Server responds — the client always makes the first move; the server waits and reacts.
• A server is a program, not a physical machine — it can run on any hardware, including your laptop.
• A server can be a client — in microservices, services call each other; roles are relative, not fixed.
• Multiple server types work together — a single user request typically touches several specialised servers: application server, database, and more.
• Always start simple — Client → Server → Database is the baseline; add complexity only when justified by scale or requirements.

🍕 Challenge: Design a food delivery app

You are designing a system like Uber Eats or DoorDash. Think through the following:

• What are the different types of clients in your system? (Hint: there is more than one kind of user.)
• What are the different types of servers you would need? List at least four.
• Draw a simple diagram showing how a customer places an order — trace the request from the customer's phone to the restaurant and back.
• What happens if your main application server goes down while someone is placing an order?

In the cloud, a "server" is any service that receives and processes requests. The three main ways to run server code on any cloud platform are:

Simplest AWS picture: A browser (client) sends a request → EC2 instance or Lambda function (server) receives and processes it → sends a response back. That is the client-server model running in the cloud.

Every device on the internet has a unique numeric address — an IP address — just like every house has a street address. But humans don't think in numbers. We use friendly names like youtube.com. DNS is the system that bridges this gap, translating the names we type into the addresses machines actually use. In this section you will learn what IP addresses are, how public and private addresses differ, how DNS resolves names step by step, and why both concepts are foundational to every system design decision you will make.

Imagine you type youtube.com into your browser. You know the name — but your computer does not know where YouTube's servers are physically located on the internet. It needs a numeric address. An IP address is that numeric address: a unique identifier assigned to every device connected to a network, from your laptop to YouTube's servers.

But here is the challenge: IP addresses look like 142.250.80.14. No human is going to memorise that. So the internet uses a naming system called DNS — Domain Name System — that automatically translates youtube.com into 142.250.80.14 every time you press Enter. Without IP addresses, devices cannot communicate. Without DNS, humans cannot use the internet practically.

IP addresses and DNS are not optional infrastructure — they are the foundation on which every internet system runs. Cloudflare's public DNS resolver (1.1.1.1) alone handles over 1 trillion DNS queries per month. Google's DNS (8.8.8.8) processes billions of queries daily. Every website visit, API call, and app request begins with a DNS lookup.

• In system design, DNS is how traffic is routed to the right servers — load balancers, CDN edge nodes, and multi-region endpoints all use DNS.
• When you add a new server or replace a failed one, you update a DNS record — not every client application.
• Private vs public IP addressing determines what parts of your system are reachable from the internet — a critical security decision.
• DNS TTL directly controls how quickly your system can recover from failures and how smoothly you can migrate servers.

Key insight: DNS is where system design meets the internet. Every load balancer, CDN, and API gateway in this series is ultimately reached through a DNS record. Understanding DNS now means every future topic will make more sense.

Think of a city's postal system. Every building has a street address (the IP address) — a precise numeric location that delivery services use to physically find it. But people don't walk around saying "I'm going to 221B Baker Street" — they say "I'm going to Sherlock Holmes' house." The phonebook or directory is what translates that name into the actual address.

Just as you would look up a name in a phonebook to find the phone number, your browser looks up a domain name in DNS to find the IP address — every single time, unless the answer is already cached.

Here is the exact sequence of events when you type youtube.com in your browser and press Enter. This process completes in milliseconds, but involves up to 9 steps behind the scenes.

Fast path: Steps ④–⑥ are skipped whenever a cached answer exists — which is most of the time for popular domains. Caching is what makes DNS fast enough to be invisible to users.

The diagram below shows the full DNS resolution journey — from your browser typing a domain name to connecting to the actual server.

Reading the diagram: Your browser ① asks the DNS Resolver for youtube.com. The Resolver doesn't know the answer, so it asks ② Root DNS, which points it to the .com TLD servers ③④. The TLD points it to YouTube's own nameservers ⑤, which return ⑥ the final IP address with a TTL of 300 seconds ⑦. The Resolver caches the answer and returns it ⑧. Your browser then connects directly to YouTube's server ⑨.

Golden rule: In production systems, only your load balancers, CDNs, and API gateways have public IPs. Everything behind them — databases, caches, internal services — uses private IPs and is never exposed to the internet.

When you open the Instagram app from Tokyo, here is exactly how DNS and IP addressing route your request to the nearest server:

New term above? Step ③ uses GeoDNS — DNS that returns different IPs based on where the user is located, routing them to the nearest data center. This will be covered in full when we reach Data Centers & Multi-Region in Phase 2.

• IP address is the unique numeric identifier of any device on a network — machines use it to reach each other.
• IPv4 (4.3B addresses, largely exhausted) vs IPv6 (virtually unlimited) — new infrastructure should prefer IPv6.
• Public IPs face the internet; private IPs are for internal communication and should never be exposed directly.
• DNS translates human-readable domain names into IP addresses through a 4-level hierarchy: Resolver → Root → TLD → Authoritative.
• TTL controls how long DNS answers are cached — low TTL for fast changes, high TTL for fewer queries.
• DNS records (A, CNAME, MX, TXT, NS) each serve a specific purpose — A records map domains to IPs, CNAME creates aliases, MX handles email.

🌍 Challenge: Design a global web application

Your company is launching a web application with servers in 3 regions: US East, Europe (Frankfurt), and Asia Pacific (Tokyo). Answer the following:

• European users should connect to Frankfurt servers, Asian users to Tokyo servers. How do you configure DNS to achieve this?
• Your TTL is set to 86400 seconds (24 hours). Your primary server fails. How long before users fail over to the backup? What should you have done differently?
• Your backend databases must never be reachable from the internet. How do you configure IP addressing to enforce this?

DNS management and IP routing are provided as managed services on every major cloud platform:

AWS-first picture: youtube.com is managed in Route 53. Route 53 returns different IPs based on the user's region (latency-based routing). Each region's load balancer has a public IP; backend servers use private IPs inside a VPC.

You now know that DNS translates youtube.com into an IP address — but what happens next? Once your browser has the server's address, it needs a common language to ask for data and receive responses. That language is HTTP. When that communication is encrypted, it becomes HTTPS. In this section you will learn how HTTP requests and responses are structured, the five HTTP methods every engineer must know, what status codes mean, why HTTP is stateless, and why HTTPS is non-negotiable in production systems.

Imagine you search for "laptop" on Amazon. Your browser sends a precisely structured message: GET /search?q=laptop HTTP/1.1. That is an HTTP request. Amazon's server processes it and sends back an HTTP response with product data. Every web page you visit, every API call your app makes, every file you download — all of it travels as HTTP or HTTPS.

HTTP (HyperText Transfer Protocol) defines how clients and servers communicate — what a request looks like, what a response contains, and what each side can expect. HTTPS is HTTP with TLS encryption so no one can intercept or read the data in transit.

When system designers draw an arrow between a client and a server — that arrow IS HTTP/HTTPS. Every REST API, web application, mobile app, and most microservice-to-microservice calls use HTTP as the communication protocol.

• HTTP methods (GET, POST, PUT, PATCH, DELETE) are how you design clean, predictable APIs that developers can understand instantly.
• Status codes (200, 404, 500) are how clients know whether a request succeeded or failed — without them, every error looks the same.
• HTTP is stateless — every request must carry its own authentication. This single property shapes how you design sessions and scalability in every distributed system.
• HTTPS is non-negotiable in production: passwords, payment details, tokens, and personal data must always be encrypted in transit.

In system design: Always say "clients communicate over HTTPS" — never draw an arrow without knowing that arrow means an HTTP/HTTPS call. This shows you understand both the protocol and the security requirement.

Think of HTTP like placing a phone order at a restaurant. There is a structured format both sides agree on: you say what you want (request), the restaurant confirms and gives you the result (response). Both sides follow the same script — that script is the protocol.

An HTTP exchange has two halves: a request (client → server) and a response (server → client). Each has a defined, structured format that every client and server in the world understands.

Every HTTP request has three parts: a request line (method + URL + HTTP version), headers (metadata), and an optional body (data for POST/PUT/PATCH). Here is a real search request to Amazon:

In plain English: "Hey Amazon (Host), please give me (GET) the search results for 'laptop' (/search?q=laptop). Here is my login token (Authorization). I want the response as JSON (Accept)."

The server replies with a status line (version + status code + text), headers, and a body containing the actual data returned.

In plain English: "Request successful (200 OK). Here is the data as JSON (Content-Type). You can cache this for 60 seconds (Cache-Control)."

Key insight: The request line tells the server WHAT to do. The headers add context (who you are, what format you accept). The body carries data (only in POST/PUT/PATCH). The response status code tells you the outcome before you even read the body.

HTTP has several key building blocks: methods (action to perform), status codes (what happened), headers (metadata), body/payload (data), the critical stateless property, and the HTTPS/TLS security layer. Each is explained below.

GET — Read data. Fetches data without changing anything on the server. Safe to repeat — refreshing a page just sends the same GET request again.

POST — Create new data. Sends data in the body to create something new. Repeating a POST order request creates two separate orders — not idempotent like GET.

PUT — Replace entire resource. Replaces the full resource with a new version. You must send ALL fields — any field not included is removed.

PATCH — Update part of a resource. Updates only the fields you send. More efficient than PUT when you only need to change one or two fields.

PUT vs PATCH: PUT = replace the whole object (must send everything). PATCH = change only what you specify (send only changed fields). In practice, PATCH is used far more often because it is safer and more efficient.

DELETE — Remove a resource. Permanently removes the identified resource.

Status codes are three-digit numbers in every HTTP response. They tell the client immediately — before reading the body — whether the request succeeded or failed. Memorise these eight codes: they cover 90% of what you will encounter in real systems.

401 vs 403: 401 = "I don't know who you are — login first." 403 = "I know who you are, but you're not allowed to do this." A request with no token → 401. A normal user trying an admin action → 403.

Headers are key-value pairs that carry metadata. Think of them like labels on a package — the package contains the main item (the body), but the labels tell the receiver what type of item it is, who sent it, and how it should be handled.

Here is what a real POST request with authentication headers looks like:

The body is the actual data content. GET and DELETE requests usually have no body — the URL carries all the information. POST, PUT, and PATCH carry data in the body — this is how you send new or updated data to the server.

In modern APIs, the body is almost always JSON because it is readable by both humans and machines. Example login request body:

And the server's response body (after placing an order):

This single property shapes every scalability decision you will make: HTTP is stateless. The server does not automatically remember anything about a previous request. Every request is treated as completely independent.

Real-world analogy: Imagine calling a customer support center. Every time you call, a different agent answers. That agent has no memory of your previous calls — you must re-identify yourself every time: "Hi, my name is james, customer ID 12345, calling about order ORD-123." HTTP works exactly the same — every request must carry enough information for the server to understand who you are and what you are allowed to do.

Because the server remembers nothing, the client includes an authentication token, cookie, or session ID in every request header:

Why statelessness is great for scalability:

• Any server in a cluster can handle any request — the request contains all the information the server needs
• Load balancers can route requests to any available server — no "sticky sessions" needed
• If a server crashes, another server picks up the next request with no data loss
• Auto-scaling works cleanly — new servers are immediately ready to handle requests

Without HTTPS: Anyone between the client and server — on the same Wi-Fi, at the ISP, or a malicious middle actor — can read everything: passwords, tokens, credit card numbers, personal messages. This is called a man-in-the-middle attack. HTTPS makes all of this data completely unreadable to anyone who intercepts it.

TLS (Transport Layer Security) is the security layer under HTTPS. You may hear "SSL" — that is the older name; modern systems use TLS. TLS provides three guarantees for every HTTPS connection:

The TLS handshake (happens automatically in milliseconds before the first HTTP request):

In production: TLS is usually terminated at the load balancer or CDN layer — not at the backend server. The load balancer handles TLS encryption/decryption, and backend servers receive unencrypted HTTP on the internal private network (protected by private IPs and firewall rules). This is called TLS termination.

The diagram below shows the difference between HTTP and HTTPS, and the structure of the request-response cycle.

Reading the diagram: HTTP sends data as plaintext — anyone who intercepts the traffic between client and server can read passwords, tokens, and personal data. HTTPS wraps the same HTTP communication in TLS encryption — the data is unreadable to anyone except the intended client and server.

Golden rule: Use HTTP (not HTTPS) only for local development or internal service-to-service calls inside a private VPC. Every external-facing endpoint — login, API, CDN, admin panel — must use HTTPS.

When you place a food order on Uber Eats, here are the HTTP calls happening behind the scenes:

Notice: All five HTTP methods appear in a single user session. Each call has the right method for the action — GET for reading, POST for creating, PATCH for partial update, DELETE for removal. This is clean REST API design.

• HTTP is the protocol defining how clients and servers communicate — every web request is an HTTP request-response pair.
• HTTPS = HTTP + TLS encryption — always use HTTPS in production for any data that matters.
• 5 methods: GET (read) · POST (create) · PUT (replace) · PATCH (partial update) · DELETE (remove). Use the right one for each action.
• Status codes: 2xx success · 3xx redirect · 4xx client error · 5xx server error. Return meaningful codes — never 200 for everything.
• HTTP is stateless — every request is independent. Authentication tokens or cookies must be included with every request that needs them.
• REST APIs are built on HTTP — resources are URLs, actions are methods, results are status codes.

🍕 Challenge: Design a food delivery app REST API

For each of the following actions, choose the correct HTTP method, design the endpoint URL, and state the expected success status code:

• Browse available restaurants near the user
• Place a new food order
• Change the delivery address on an existing order
• Cancel an order before it is picked up
• A user tries to cancel an order that doesn't exist — what status code should the server return?

In cloud production systems, HTTP/HTTPS traffic flows through these managed services:

AWS flow: Client → Route 53 (DNS) → CloudFront (CDN + HTTPS) → Application Load Balancer → EC2/Lambda (backend). ACM automatically provides and renews the TLS certificate for CloudFront and ALB — no manual certificate management needed.

You now know that HTTP/HTTPS is the language clients and servers use to communicate. But how does that data actually travel across the internet? That is the job of the transport layer, and there are two main protocols to choose from: TCP (reliable, ordered, slower) and UDP (fast, lightweight, no guarantees). Every system design decision involving real-time communication — video calls, online gaming, live location tracking — ultimately comes down to choosing between these two.

Imagine you are on a Zoom call. At the same moment, your browser downloads your bank statement. Both use the internet, but they behave very differently: the Zoom video stream keeps going even if a few frames are lost — your call stays smooth. But your bank statement absolutely cannot have a single byte missing or corrupted — every number must be exact.

This difference comes down to TCP vs UDP. TCP (Transmission Control Protocol) is the careful, reliable choice — it guarantees every byte arrives in order. UDP (User Datagram Protocol) is the fast, lightweight choice — it sends data as quickly as possible without waiting for confirmations.

Understanding where TCP and UDP sit in the network stack is essential:

Every system you design has components that communicate over a network. The choice of TCP vs UDP directly affects reliability, latency, and user experience. Getting this wrong can mean lost payments, broken file downloads, or laggy video calls.

• HTTP/HTTPS (every web page and REST API) runs on TCP — reliable delivery is non-negotiable for web content.
• DNS lookups commonly use UDP — queries are tiny and speed matters more than retrying.
• Zoom, Google Meet, and Discord voice use UDP-based protocols — a lost video frame is better ignored than waited for.
• WhatsApp text messages use TCP — but WhatsApp voice/video calls switch to UDP-based transport.
• Modern HTTP/3 uses QUIC over UDP — an attempt to get TCP-like reliability with UDP-like speed.

Core decision: Use TCP when correctness matters more than speed. Use UDP when speed matters more than perfect delivery.

TCP is like sending an important contract via registered mail with tracking and signature confirmation. The courier confirms delivery, tracks every step, resends if something goes missing, and ensures pages arrive in the right order. Slower — but nothing is lost.

UDP is like a sports commentator shouting live updates. They keep talking regardless of whether every word reaches every listener — some words may be lost to background noise, but the commentary stays current and keeps moving forward.

Step 1: The 3-Way Handshake — Before any data is sent, TCP establishes a connection:

Only after all three steps does data transfer begin. This adds one round-trip of latency before any content is sent.

Step 2: Ordered Delivery — TCP numbers every packet. Even if they arrive out of order, TCP reorders them before handing data to the application:

Step 3: Retransmission — If a packet is lost, TCP detects it and requests a resend. The application waits until the complete data arrives:

Head-of-Line Blocking: Because TCP delivers data IN ORDER, one missing packet blocks all later packets from being delivered — even if they've already arrived. Like a queue where one person drops something and nobody behind them can move forward until it's picked up.

No handshake — UDP just sends packets immediately. No connection setup, no waiting:

No ordering, no retransmission — if a packet is lost, UDP ignores it and keeps going. The application receives whatever arrives, in whatever order:

Where each protocol sits in real stacks:

HTTP/3 & QUIC: HTTP/3 runs on QUIC which is built on top of UDP. QUIC adds reliability features similar to TCP (ordering, retransmission) but solves TCP's head-of-line blocking problem. For most system design discussions, remember: classic HTTP = TCP, HTTP/3 = QUIC/UDP.

One of the most useful ways to decide between TCP and UDP is to ask: "If a packet is lost, can the application continue correctly — or does it break?"

Two questions to decide:
① Do I need reliable, ordered delivery? → Yes → TCP
② Can I tolerate some packet loss in exchange for lower latency? → Yes → UDP

The diagram below compares the TCP and UDP flows side by side — showing the handshake, ordered delivery, and retransmission of TCP versus the fire-and-forget simplicity of UDP.

Reading the diagram: TCP (blue) requires a 3-step handshake before any data, confirms every packet, and resends losses. UDP (orange) just fires packets one after another with no confirmation — faster, but Packet 3 being lost is simply ignored.

Ask two questions: Do I need every byte to arrive correctly? and Can I tolerate losing some data if it means lower latency?

Rule of thumb: If missing data would cause a bug, security issue, or incorrect result → TCP. If missing data just causes a tiny visual glitch or the data is immediately superseded anyway → UDP.

The same application often uses BOTH TCP and UDP for different features. Here are four concrete examples showing exactly which protocol is chosen and why.

For online payment, correctness is more important than speed. A payment request contains critical data:

You do not want this data to be lost, duplicated, corrupted, or delivered out of order. A missing packet could mean the wrong amount is charged or the transaction is never recorded. A tiny delay is perfectly acceptable. An incorrect payment is not.

Payment always uses TCP (HTTPS): correctness > speed. Every byte must arrive in order.

This is the most important distinction to understand — and one that trips up beginners. Not all video is the same.

Key takeaway: YouTube uses TCP because buffering tolerates short delays. Zoom uses UDP because a 200ms delay waiting to retransmit an old video frame makes the conversation choppy and unusable.

In an online game, your character's position changes many times per second. The server must know where every player is at every moment:

If the position update from 10:00:01.020 is lost, it is pointless to wait for it — by the time it is retransmitted, the position at 10:00:01.040 is already more accurate. Waiting for the old packet (TCP behaviour) would cause lag and make the game feel sluggish. Instead, games use UDP: if a position update is lost, just use the next one that arrives.

Online gaming uses UDP because old state (old position) is immediately superseded. Low latency and smooth experience matter far more than perfect delivery of every packet.

A single chat app uses different protocols for different features — a perfect illustration of how real systems mix TCP and UDP:

Key insight: WhatsApp uses TCP for everything that must not be lost (text, files, login) and UDP-based protocols for everything where latency matters more than perfection (voice, video). One app — both protocols — different features.

• TCP = reliable, ordered, connection-oriented. Use for payments, APIs, file transfers, login — anything where missing data = broken functionality.
• UDP = fast, lightweight, connectionless. Use for live video/audio, online gaming, DNS — anything where speed matters and old data is worthless.
• TCP 3-way handshake (SYN → SYN-ACK → ACK) establishes a connection before data is sent, adding one round-trip of latency.
• Head-of-line blocking is TCP's key limitation — one lost packet stalls all later packets until it is retransmitted.
• HTTP/HTTPS runs on TCP. DNS uses UDP. HTTP/3 uses QUIC over UDP — reliability with less head-of-line blocking.
• One system can use both TCP and UDP for different features — WhatsApp uses TCP for text, UDP-based for voice/video.

🚗 Challenge: Design the Uber App — Choose TCP or UDP for each feature

For each feature below, choose TCP or UDP and explain why:

• User login and signup
• Booking a ride (request + confirmation)
• Processing payment at the end of a ride
• Live driver location updates (shown on the map every second)
• In-app chat between rider and driver

TCP and UDP are protocols, not cloud services — but cloud load balancers and gateways handle them differently. Here are the cloud services relevant to TCP vs UDP routing:

AWS mental model: Web/API traffic (HTTP/HTTPS over TCP) → Application Load Balancer. Real-time or raw TCP/UDP (gaming, VoIP, custom protocols) → Network Load Balancer. DNS queries (UDP) → Route 53.

You have now learned how data is found (DNS), how it is communicated (HTTP/HTTPS), and how it is transported (TCP/UDP). The final question in this networking foundation is: how fast does the system respond, and how much work can it handle at once? These are the two most important performance metrics in system design — latency (speed for one user) and throughput (capacity for many users). Every performance decision you make as an engineer comes down to these two concepts.

Imagine you are designing a system like Amazon. A user types "laptop" in the search box and presses Enter. Two critically important questions arise immediately:

Latency is the time it takes for a single request to travel from the client to the server and come back with a response — the user's waiting time. If this takes 200 ms, that is the latency. Throughput is how much work the system can handle per unit of time — how many requests per second it can process. A system can have good latency for individual users but still fail during peak traffic if throughput is too low.

Latency and throughput are not just academic concepts — they directly affect users and business outcomes. Studies show that a 100ms increase in latency reduces Amazon sales by 1% and a 1-second delay causes a 7% drop in conversions. Google found that 53% of mobile users abandon a page that takes longer than 3 seconds to load.

• Latency determines whether your app feels responsive or sluggish — it is what the user directly experiences.
• Throughput determines whether your system survives peak traffic — a sale event, a viral post, or a breaking news moment.
• p99 latency matters more than average — if 1% of requests to Amazon are slow and Amazon serves 10 million requests/day, that is 100,000 slow experiences daily.
• Every system design decision — caching, CDN, database indexing, load balancing — ultimately improves latency, throughput, or both.

Remember: Latency = speed for one user. Throughput = capacity for many users. A system can be fast for individual requests but still collapse under heavy load — or handle massive traffic but feel sluggish for each user.

Think of a highway between Tokyo and Osaka:

A highway may let thousands of cars through per hour (high throughput) — but if there is a traffic jam, each car still takes longer to reach its destination (high latency). Similarly, your system can handle many requests per second while some individual requests are slow. These two dimensions are independent but related.

Latency is measured from the moment the client sends a request to the moment it receives a complete response:

Each component adds to the total latency. A slow database query, a distant server, or a large response all increase the time the user waits.

This is one of the most important concepts in production systems — and one that beginners consistently get wrong.

Suppose your system has an average latency of 100ms. That sounds good. But what if some users experience 5 seconds? Average latency hides these slow users. This is why production systems use percentile latency:

Why p99 matters: If your system serves 10 million requests per day and p99 = 2 seconds, that means 100,000 users per day are experiencing 2-second wait times. Average latency of 100ms looks great on the dashboard — but 100,000 users are having a bad experience. Always monitor p95 and p99, not just average.

Throughput is measured as the number of operations completed per unit of time:

Understanding why latency is high is the first step to fixing it. These are the six most common causes:

1. Network Distance — If a user in Japan makes a request to a server in the US, the data travels thousands of kilometres. Each kilometre adds latency. Speed of light in fibre optics is about 200,000 km/s — a round trip Japan→US→Japan adds ~150ms just for travel.

Fix: Deploy regional servers, use CDN edge nodes, use DNS-based geographic routing.

2. Slow Database Queries — A backend can respond in milliseconds, but if the database takes 2 seconds to run a query, that 2 seconds is the bottleneck.

Fix: Add indexes, optimize queries, use caching, use read replicas, use search engines (Elasticsearch) for complex searches.

3. Too Many Service Calls (Microservices) — In microservices, one user request may trigger a chain of calls to many internal services. Each call adds latency.

Fix: Reduce unnecessary calls, run independent calls in parallel, cache frequently needed data, avoid chatty communication patterns.

4. Server Overload — If a server receives more requests than it can handle, requests queue up. Users at the back of the queue wait longer.

Fix: Add more servers (horizontal scaling), load balancing, auto-scaling, queue-based processing for heavy tasks.

5. Large Response Size — Returning too much data takes longer to send over the network.

Fix: Pagination, compression (gzip/Brotli), CDN for static content, return only required fields, efficient data formats.

6. Cold Starts (Serverless) — In serverless systems (AWS Lambda), if a function hasn't run recently, the cloud provider must spin up a new instance. This startup delay — a "cold start" — can add hundreds of milliseconds.

Fix: Keep critical functions warm, use provisioned concurrency, use always-running services for latency-sensitive paths.

The diagram below shows latency (time for one request end-to-end) and throughput (multiple requests handled per second) as distinct but related dimensions.

Reading the diagram: Latency (blue) is the journey of ONE request through the network, server, and database — every hop adds time. Throughput (green) is many users hitting a load balancer that distributes work across multiple servers — adding more servers increases capacity.

Different system features have different performance priorities. Always ask: does this feature need a fast response for one user, or does it need to handle many users simultaneously, or both?

Key rule: Real-time user-facing features need low latency. Background and batch operations need high throughput. Features serving many users simultaneously need both.

Notice the pattern: Every large system has latency-critical features (things users wait for) and throughput-critical features (things that happen at massive scale). Knowing which is which is how you make the right architectural decisions.

Improving latency and throughput can sometimes conflict. Understanding these trade-offs is essential for making the right system design decision.

Golden rule: Identify the bottleneck first. Adding more app servers when the database is the bottleneck does not help. Profile before optimizing — measure which component contributes most to latency, then fix that specific component.

• Latency = time for one request to complete. Throughput = how many requests the system handles per second. They are different dimensions.
• Always measure percentile latency (p50, p95, p99), not just average. p99 can reveal thousands of users having a bad experience that averages hide.
• Main causes of high latency: network distance, slow DB queries, too many service calls, server overload, large responses, cold starts.
• Reduce latency with: caching, CDN, regional servers, DB indexing, async processing, parallel service calls, compression.
• Improve throughput with: horizontal scaling, load balancing, caching, DB scaling (replicas/sharding), message queues, reducing per-request work.
• Caching improves both latency and throughput — it is one of the most powerful tools in system design.

🍕 Challenge: Food Delivery App Performance

For each feature below, decide whether Latency, Throughput, or Both are the primary concern. Then suggest one technique to improve that dimension:

• User searches nearby restaurants
• User places an order
• System sends order confirmation email
• Driver location updates every second
• Payment processing

Every major cloud platform has services specifically designed to reduce latency and increase throughput:

AWS mental model for performance: Reduce latency → CloudFront (global) + ElastiCache (data) + Route 53 latency routing (regional). Increase throughput → Auto Scaling + Load Balancer + SQS (absorb spikes) + Read Replicas (database). Monitor both → CloudWatch metrics + X-Ray traces.