System-Design

Your checkout service calls the payment gateway. One night the gateway has a bad deploy and starts timing out. Every synchronous checkout request thread now remains occupied for the full 10-second client timeout. Threads pile up, the pool exhausts, health checks fail, and the load balancer starts draining your perfectly healthy checkout pods. Nothing about checkout is broken — it just never stopped calling a dependency that was already dead. ...

It’s 3 AM again. A customer placed an order, payment went through, and inventory reservation failed — the warehouse is out of stock. The customer has been charged for something that will never ship. The order row exists and the customer was charged, but nothing will ship — and no single transaction can unwind it, because there is no @Transactional that spans your order service, your payment service, and your inventory service. Each one already committed its own local transaction. ...

It’s 3 AM. A customer paid, the orders row is in your database, and the OrderCreated event never reached Kafka. The warehouse never shipped. The email never sent. Somewhere in your service there is code that looks like this: @Transactional public Order placeOrder(PlaceOrder cmd) { Order order = orderRepository.save(new Order(cmd)); // write #1 kafkaTemplate.send("orders", order.getId(), toJson(order)); // write #2 — outside the safety net return order; } Two writes. No shared atomicity. If the broker is down, if the pod gets killed, if the send times out after the commit — the database and the downstream world silently disagree, and you get to find out from an angry customer. This is the dual-write problem, and the fix is the transactional outbox pattern. ...

“Design a distributed cache.” You say hash(key) % N. The interviewer nods, then asks: “A node dies at 3 AM. What happens?” If your answer is “we rehash everything,” you’ve just told them your cache has a planned outage every time the cluster changes shape. This post builds the real answer — a working consistent hash ring in Java, with virtual nodes and tests — and the lineage and follow-ups that turn it into a senior-level answer. ...