Almost every system design drawing has a box near the front labelled "API Gateway". Beginners copy it onto the board because everyone else does. Then the interviewer asks "what does it actually do?" and the answer falls apart.
This note is about getting that answer right. We start with the plain idea of a proxy, because a gateway is just a proxy with a job description.
What a proxy is
A proxy is a middleman. Instead of two programs talking to each other directly, they talk through a third one. The proxy receives a request, may look at it or change it, forwards it on, then passes the answer back.
The only question is which side the middleman works for. That gives you two kinds.
Forward proxy
A forward proxy sits in front of the clients. It speaks for them.
Think of a personal assistant who makes phone calls for you. The people they call hear the assistant, not you. And the assistant can refuse to call certain numbers.
A forward proxy works the same way. Your laptop sends its web request to the proxy, and the proxy requests the website for you. The website sees the proxy's address, not yours.
Where you meet them:
- Office or school networks, to block certain sites or log who visited what.
- VPNs, which hide your real address and make you look like you are somewhere else.
- Caching for a group of users, so a big download is fetched once and shared.
- Web scraping tools that rotate through many addresses (and that is why many sites block them).
The server does not choose a forward proxy. The client side does.
Reverse proxy
A reverse proxy sits in front of the servers. It speaks for them.
This one is the reception desk at a company. Visitors never walk straight into the offices. They talk to reception, who sends them to the right person, turns away the ones who should not come in, and keeps the layout of the building private.
From the outside, the user thinks they are talking to shop.example.com. In reality they talk to the proxy, and behind it there might be ten servers. The user never learns how many, or where.
What a reverse proxy usually does:
- Sends requests to one of many backend servers (this includes load balancing).
- Handles HTTPS, so the backends only need to speak plain HTTP inside your private network. This is called TLS termination. TLS is the encryption behind the padlock in the browser.
- Caches common responses.
- Compresses responses, for example with gzip.
- Hides your internal servers and their addresses.
Nginx, HAProxy and Envoy are common examples. Many CDNs are, under the hood, a reverse proxy spread across the world.
A one line way to remember the difference: a forward proxy hides the clients from the servers, and a reverse proxy hides the servers from the clients.
From reverse proxy to API gateway
Say you run a food delivery app that started as one program. Over time you split it into services: users, restaurants, orders, payments, reviews. Each has its own address and its own team.
Now the mobile app needs to talk to all of them. Without anything in front, the phone has to know five addresses, log in to each, and deal with five sets of errors. And every team writes their own code for checking login tokens and limiting abusive users. Five teams, five slightly different versions of the same code, and one of them has a bug.
An API gateway solves this by being a single front door. The app calls one address. The gateway looks at the request, does the shared chores, and sends it to the right service.
It is a reverse proxy that understands your API. A plain reverse proxy mostly looks at addresses and ports. A gateway looks at the request itself: the path, the headers, who is calling.
What an API gateway usually does
Here are the usual jobs. Not every gateway does all of them, and you should be able to say which ones you actually need.
Authentication. The gateway checks that the caller is who they say, typically by checking a token (like a JWT, which is a signed piece of text that proves you logged in). If it is bad or missing, the request is rejected at the door with a 401, and the services behind it never see it. Many teams check the token at the gateway and also let each service read the user id from a header the gateway adds.
Rate limiting. The gateway counts how many requests each user or API key sends, and rejects extra ones with 429 Too Many Requests. For example, 100 requests per minute per user. This protects you from bugs, abuse and one noisy customer using everything up. It is easy to do here because the gateway sees all traffic in one place.
Routing. /orders/* goes to the orders service, /users/* goes to the users service. It can also route by version (/v2/ to the new service) or by header, which is handy for slowly moving 5 percent of traffic to a new version.
Request shaping. This means changing requests and responses on the way. Adding a header, removing a field the client should not see, turning an old URL format into a new one, or combining the answers of several services into one response so the phone makes one call instead of four. Keep this light. The more logic you cram here, the more it becomes a hidden second application.
Logging and metrics. Since every request passes through, it is a natural place to record who called what, how long it took, and what status came back. Many gateways also attach a request id that follows the request through every service, which makes debugging much easier.
Other things you may see: TLS termination, response caching, request size limits, CORS headers, retries and timeouts, and blocking known bad addresses.
What does not belong in a gateway: business rules. "Is this coupon valid for this restaurant?" is the orders service's job. If you find your gateway running that kind of logic, it is getting too big.
Gateway, load balancer and service mesh
These three get mixed up all the time, because all of them are "a thing that sits in the path and forwards requests". The difference is what they know and where they sit.
| Load balancer | API gateway | Service mesh | |
|---|---|---|---|
| Sits where | In front of a group of identical servers | At the edge, in front of all your services | Next to every service, handling traffic between services |
| Main job | Spread requests across many copies of one service | One front door with shared chores (auth, limits, routing) | Safe, observable service to service calls |
| Knows about | Which servers are healthy | Your API paths, callers and plans | Every service and its connections |
| Traffic direction | Mostly outside to inside | Outside to inside ("north to south") | Inside to inside ("east to west") |
| Typical features | Health checks, round robin, least connections | Auth, rate limits, routing, request shaping | Mutual TLS, retries, timeouts, tracing between services |
| Examples | AWS ALB, Nginx, HAProxy | Kong, AWS API Gateway, Apigee | Istio, Linkerd |
| Do you need it early | Yes, as soon as you have two servers | When you have several services or outside developers | Rarely, only with many services and a platform team |
A way to picture it. The load balancer is the person at a restaurant door who sends each new group to a free table. The gateway is the host who also checks your reservation, turns away people who are not allowed in, and knows which section each group should go to. The service mesh is the set of rules and walkie talkies used by waiters between the kitchen, the bar and the tables, and it has nothing to do with customers walking in.
Important: they are not rivals. A real system often has all of them. The gateway sits behind a load balancer (so the gateway itself has several copies), and each service behind the gateway has its own load balancing. In many products one tool does two jobs, for example a cloud load balancer that also supports path routing. In an interview, do not get stuck on product names. Say what job each box does.
A service mesh is the one most people are quickest to over use. Every service gets a small proxy running beside it (called a sidecar), and those proxies handle encryption between services, retries, and metrics, so the application code does not need to. It is useful when you have dozens of services and want the same rules everywhere. It also adds moving parts, more memory, and a lot of new things to learn. For a team with four services, I would skip it until it hurts.
A tiny rate limit example
Rate limiting sounds abstract until you put numbers on it. A common method is the token bucket. Each user has a bucket that holds up to 10 tokens. Every request takes one token. The bucket gets 1 new token every second, up to the maximum of 10.
So a user can burst 10 requests at once, then settles to 1 request per second. If they send 15 requests in the first second, the first 10 pass and the last 5 get 429. Five seconds later they have 5 tokens again and can send 5 more.
The gateway stores each user's token count and last refill time. With several gateway copies, that pair has to live in a shared place such as Redis. If every copy kept its own count, a user hitting 4 copies would get 4 times the allowed rate.
When you reject, tell the caller when to come back:
HTTP/1.1 429 Too Many Requests
Retry-After: 3Good clients read that and wait. This pairs nicely with the retry advice from the idempotency and retries note.
A routing table, as a gateway sees it
Most gateways are configured with a list of rules. In plain words, a small one looks like this:
| Path starts with | Goes to | Extra rule |
|---|---|---|
/users | users service | Token required |
/restaurants | restaurants service | Public, cached for 30 seconds |
/orders | orders service | Token required, 20 requests per minute per user |
/payments | payments service | Token required, 5 requests per minute per user |
/v2/orders | new orders service | Only for 5 percent of users |
Each row is one decision the gateway makes without calling any service. That is why it is fast, and also why this table is a good size to stay at. Once rows turn into long if and else logic, you are building an application inside a proxy.
The gateway as a bottleneck and a single point of failure
Putting everything through one door has a price.
It is a single point of failure. If the gateway is down, every service behind it is unreachable, even if all of them are healthy. A single bad deploy of the gateway can take down the whole product.
It can be a bottleneck. Every request pays the cost of one extra hop, and the gateway has to handle the total traffic of the whole system. If it is slow, everything is slow.
It can become a dumping ground. Over time, teams add "just one more rule" and the gateway becomes a big shared program that everyone is afraid to touch, and every change needs the gateway team. That is a team bottleneck, which is as real as the technical one.
What to do about it:
- Run several copies of the gateway across availability zones, behind a load balancer, so losing one machine is not an outage.
- Keep it stateless. Anything it needs to remember (like rate limit counters) goes into a shared store like Redis, so any copy can handle any request.
- Keep it thin. Auth, limits, routing, logging. Leave business logic to the services.
- Set timeouts and a circuit breaker for each backend, so one slow service does not use up all the gateway's connections.
- Watch its latency and error rate closely. Its numbers are the numbers of your whole product.
- Use a managed gateway from a cloud provider if you can. Running your own gateway fleet is real work.
If rate limiting needs a shared counter and Redis is down, decide in advance what happens. "Fail open" means allow requests when the limiter is broken. "Fail closed" means block them. For most public APIs, failing open is the safer choice, since a broken limiter should not take your whole site offline. For something like a login endpoint under attack, you may prefer to fail closed. Whichever you choose, say it out loud.
Backend for frontend
Sometimes one gateway for everybody is not a good fit. A phone app wants small responses with just a few fields, because of slow networks and small screens. A web dashboard wants a lot of data on one page. A smart TV app wants something else again.
Backend for frontend, shortened to BFF, means one small gateway per kind of client. The mobile BFF is built by the mobile team and shapes responses for phones. The web BFF is built by the web team and shapes them for the dashboard. Both call the same core services behind them.
The win is that each team can change its BFF without asking anyone, and each client gets exactly the data shape it likes. The cost is that you now run several of them, and some code gets copied. I would only do this when the clients really want different things and different teams own them. With one web app and one mobile app that need roughly the same data, one gateway is fine.
Service discovery in plain words
The gateway has to know where to send /orders traffic. But in a modern system, the orders service might be running on 6 machines today, 9 tomorrow, and the addresses change every time one is replaced. Nobody can write those addresses by hand in a config file.
Service discovery is the phone directory that keeps up with this. Services register themselves ("orders service, now at 10.0.4.17:8080, healthy") and the directory removes entries that stop answering health checks. When someone needs the orders service, they ask the directory and get a list of live addresses.
There are two common styles:
- Server side. The caller asks a load balancer or the gateway, and that component looks up the live addresses and picks one. The caller knows nothing. Kubernetes Services work this way: you call a stable name, and Kubernetes forwards to a healthy pod.
- Client side. The caller itself asks the directory, gets the full list, and chooses. This saves a hop, but every client needs the logic.
Common tools: DNS based discovery, Consul, etcd, and the built in discovery in Kubernetes and cloud platforms. In an interview, "services register with a registry, and the gateway looks up healthy instances there" is enough. Do not try to explain how Consul agrees on its data unless asked.
A request, end to end
Putting it together, here is a phone app placing an order.
Notice how little the gateway does. It checks, looks up, forwards, logs. The real work happens in the orders service.
Interview talking points
When you draw a gateway box, say why it is there in one or two sentences, then say what it does not do.
"Clients talk to a single API gateway. It checks the auth token, applies rate limits per user, and routes each path to the right service, and it logs every request with an id. It does not hold business logic. The gateway runs as several stateless copies behind a load balancer, with rate limit counters in Redis, so it is not a single point of failure."
If the system is small, you can also say: "With only two services, I would skip a gateway and use a load balancer with path routing. I would add a real gateway when we have more services or outside API users." That kind of sentence shows judgement, which interviewers like more than a full box diagram.
Things people get wrong here:
- Drawing the gateway and the load balancer as the same box without saying so. Say which one does what.
- Putting business logic in the gateway.
- Forgetting that the gateway itself needs more than one copy.
- Adding a service mesh to a three service design.
Draw the order from outside to inside as: user, CDN (for static files), load balancer, API gateway, services. Each does a different job, and in a short interview it is fine to merge the load balancer and gateway into one labelled box, as long as you say you are doing it.
Interview questions to check yourself
"What is the difference between a forward proxy and a reverse proxy?"
A forward proxy sits in front of clients and hides them from servers. A reverse proxy sits in front of servers and hides them from clients.
"Name four jobs an API gateway usually does. Authentication, rate limiting, routing to services, request shaping (such as changing headers or merging responses) and logging. Any four of these is fine."
"How is a service mesh different from an API gateway?"
A gateway handles traffic coming in from outside to your services. A mesh handles traffic between services inside, using a small proxy next to each service.
"Why is a gateway a single point of failure, and how do you reduce that risk?"
All traffic goes through it, so if it is down everything is down. Run several stateless copies in different zones behind a load balancer and keep shared state, like rate limit counters, in Redis.
"When would you use backend for frontend instead of one gateway?"
When different clients, such as phone and web, need very different data and different teams own them. Each gets its own small gateway shaped for its needs.