When event-driven architecture makes sense

Event-driven architecture has a reputation problem: it is either sold as the default way to build anything on AWS, or dismissed as overengineering. Both takes are wrong. EDA is a specific trade — you give up simple causality and immediate consistency, and in exchange you get decoupling and durable retry semantics. Whether that trade is good depends entirely on the shape of your workload. Here is how we decide, after several years of running both styles in production.

The real win is retries, not diagrams

The textbook pitch for EDA is loose coupling: producers do not know about consumers, teams ship independently, new subscribers attach without touching the publisher. That is real, but it is not the thing you feel day to day. The thing you feel is what happens when a downstream dependency fails at 3 a.m.

In a synchronous chain, a failure anywhere means the caller gets an error and someone has to decide what to do — retry, apologize to the user, or drop the work on the floor. In an event-driven chain, the failed message sits in SQS, gets retried on a schedule you configured, and after N attempts lands in a dead-letter queue where you can inspect and redrive it. The work is durable by default. That single property — failure becomes a queue depth metric instead of a lost request — is worth more than every architecture diagram benefit combined.

Where it hurts

Be honest about the costs, because they are not small.

• Debugging distributed flows. A request that used to be one stack trace is now five Lambda invocations stitched together by EventBridge rules. Without correlation IDs propagated through every event and decent tracing, answering "what happened to order 4812" takes an afternoon.
• Eventual consistency surprises. The user creates a record, the read model updates 400 ms later, and the very next page load shows stale data. Product managers file this as a bug every single time.
• Ordering. SQS standard queues do not guarantee order, EventBridge does not either. FIFO queues exist but cap throughput per message group and push complexity into your partition key choice. If your domain genuinely needs strict ordering, EDA fights you.
• Local development and testing get harder. Reproducing a five-hop event flow on a laptop is nothing like curl against one endpoint.

Rules of thumb

Our working heuristics, which have held up well:

• If the caller needs the answer to proceed — validation, pricing, auth checks — stay synchronous. Wrapping a request/response interaction in events just adds latency and failure modes.
• If the work can tolerate seconds of delay and must never be lost — notifications, billing events, data ingestion, third-party syncs — go event-driven. This is the sweet spot.
• If one action fans out to many independent reactions, use EventBridge. That is what a bus is for.
• If you have a multi-step process with branching, human-visible state, or compensation logic, reach for Step Functions rather than choreographing it through raw events. Explicit state machines are debuggable; implicit ones are archaeology.

Non-negotiables: idempotency and DLQs

Two things are not optional in any event-driven system we ship. First, idempotent consumers. SQS and EventBridge deliver at-least-once, which means duplicates are a certainty, not an edge case. Every handler must be safe to run twice — a conditional write on a request ID in DynamoDB is usually enough. Second, a dead-letter queue on every consumer, with an alarm on its depth. An event-driven system without DLQs does not fail loudly; it silently loses work, and you find out from a customer.

A worked example: ingestion pipeline

A client sends us bursts of tracking events — quiet for hours, then 5,000 per minute. The synchronous version is an API Gateway endpoint that validates, enriches, and writes to DynamoDB in one Lambda. It works until a burst arrives: Lambda concurrency spikes, DynamoDB throttles, the enrichment call to a third party times out, and every one of those failures surfaces as a 5xx to the client, who now has to implement retries on their side.

The event-driven version: the endpoint does schema validation only and drops the payload onto SQS, returning 202 in under 50 ms. Lambda workers consume the queue at a concurrency we control, do the enrichment and the DynamoDB writes with idempotency keys, and failed batches retry automatically before landing in a DLQ. Bursts become queue depth. Throttling becomes backpressure. The third-party outage last quarter delayed processing by eleven minutes and lost nothing. Nobody got paged.

Note what made this work: the client did not need a synchronous answer beyond "received". If they had needed the enriched result in the response, this design would have been wrong, and no amount of queueing would fix it.

The bottom line

Default to request/response for anything the caller waits on. Go event-driven when work is deferrable and loss is unacceptable. When you do, treat idempotency and dead-letter queues as part of the definition of done, propagate correlation IDs from day one, and use Step Functions when the flow has real state. EDA is not a maturity level to ascend to — it is a tool with a sharp edge, and it is best used where the trade actually pays.