Serverless on AWS in Production — What Nobody Tells You Until It Hurts

Someone on your team shipped a Lambda behind an API Gateway. It cost eleven cents last month, and now the team says "we're going serverless." Six months later you have forty functions. Three DLQs fill up quietly. The CloudWatch bill is bigger than the compute. The incident channel is full of the phrase "but it worked in the demo."

None of that is Lambda's fault. It's the tax on a truth nobody says out loud. Serverless doesn't remove the distributed system — it removes the servers you used to debug it on. Let's talk about what actually happens in production.

1. Serverless is a boundary, not a runtime

The mental model that gets people in trouble is "serverless = Lambda = a function." The useful model is different. Serverless is a collection of managed integration points, and Lambda is just the glue that runs your code between them. The interesting engineering is never inside a single handler — it's in the seams.

The pieces you actually compose with:

Notice the column that repeats: at-least-once. That word is the whole ballgame. Every managed integration in AWS will, under the right timing, hand your code the same event twice. Design around that and serverless is wonderful. Ignore it and you'll ship double-charged customers.

2. A reference architecture that isn't a toy

Here's an order-processing system — the kind of thing that's genuinely a good fit for serverless because traffic is spiky and the work is event-shaped. I'll use it as the running example for every failure mode below.

Read the shape, not the boxes. The synchronous path (client → API GW → Lambda → DynamoDB) is deliberately short: validate, write once, publish an event, return 202. Everything expensive or flaky — charging a card, reserving stock, sending email — happens behind a queue, asynchronously. A failure there is a retry instead of a 500 in the user's face.

3. The failure modes nobody demos

Partial failure is the default, not the exception

In the diagram above, EventBridge fans one OrderPlaced event to four consumers. Payment succeeds. Inventory succeeds. Notifications succeed. Fulfillment's Lambda throws because a downstream API had a bad 30 seconds. Now you have an order that's paid for, decremented from stock, the customer got a "thanks!" email — and nothing is shipping.

There is no transaction across those four consumers. There cannot be. This is life beyond distributed transactions in its purest form. Each consumer is its own scope of atomicity, and the only tools you have are retries, idempotency, and compensation. If your design assumes all four either happen or none do, it's already wrong.

Retries mean idempotency isn't optional

Every async invocation in AWS retries. Async Lambda invokes retry twice on error (three attempts total) and then stop. SQS redelivers until the consumer deletes the message or the maxReceiveCount is hit. EventBridge retries for up to 24 hours. So the same create-order or charge-card handler will run twice on the same input. The only defense is making the operation idempotent.

Two patterns carry almost everything:

Conditional writes for the "create once" case — let DynamoDB reject the duplicate instead of trusting your code:

// create-order: safe to run twice with the same orderId
await ddb.putItem({
  TableName: "orders",
  Item: marshall({ orderId, status: "PLACED", amount, createdAt }),
  // fails with ConditionalCheckFailedException on a retry — that's success, not error
  ConditionExpression: "attribute_not_exists(orderId)",
}).catch((e) => {
  if (e.name !== "ConditionalCheckFailedException") throw e;
  // duplicate delivery — the order already exists, so we're done
});

An idempotency table for side effects you can't make naturally idempotent (charging a card, calling a partner API). Take an idempotency key from the message, record it before the side effect, and short-circuit if you've seen it:

// charge-card: the payment provider gets called AT MOST once per orderId
const key = `charge#${orderId}`;
try {
  await ddb.putItem({
    TableName: "idempotency",
    Item: marshall({ pk: key, status: "IN_PROGRESS", ttl: now + 86400 }),
    ConditionExpression: "attribute_not_exists(pk)",
  });
} catch (e) {
  if (e.name === "ConditionalCheckFailedException") return; // already charged (or charging)
  throw e;
}
const receipt = await paymentProvider.charge(orderId, amount); // the non-idempotent part
await ddb.updateItem({ /* set status = DONE, store receipt */ });

Poison messages: the message that kills the batch

SQS delivers to Lambda in batches (up to 10 messages, or up to 10,000 with a batch window). Here's the trap in the naive setup: if one message in a batch of ten throws, the whole batch goes back on the queue. The nine good messages get reprocessed. If that one bad message carries a malformed body and always throws — a "poison" message — it retries forever. It drags its nine innocent neighbors through reprocessing every time, until it finally hits maxReceiveCount and the whole batch lands in the DLQ. You lose good messages to one bad one.

The fix is ReportBatchItemFailures — tell Lambda exactly which messages failed so it retries only those:

exports.handler = async (event) => {
  const batchItemFailures = [];
  for (const record of event.Records) {
    try {
      await process(JSON.parse(record.body));
    } catch (err) {
      // report ONLY this message as failed; the rest are deleted from the queue
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }
  return { batchItemFailures };
};

DLQs vs failure destinations — they are not the same thing

People say "DLQ" to mean "where failed stuff goes," but AWS has two different mechanisms and they capture different things:

• SQS redrive to a DLQ — for the SQS→Lambda path. After maxReceiveCount receives, the message moves to the dead-letter queue. You get the original payload. This is what you want for queue consumers.
• Lambda failure destinations — for async invokes (EventBridge, S3, direct async). On final failure, Lambda sends a rich record (request, response, error, stack) to a target — SQS, SNS, EventBridge, or another Lambda. You get the invocation context, not just the payload. Strictly better than the old "on-failure DLQ" for async, which only gave you the event.

Ordering: you probably don't have it

Standard SQS and EventBridge make no ordering guarantee. If your OrderPlaced and OrderCancelled events can race, a naive consumer will happily cancel-then-place and leave an order live that the customer cancelled. Options, in order of cost:

• Don't need order? Make handlers commutative — apply a version number or timestamp and ignore stale updates. Cheapest and most reliable.
• Need per-entity order? SQS FIFO with MessageGroupId = orderId gives ordering within a group and dedup — but caps throughput and reduces consumer parallelism per group.
• Need a strict sequence of steps? That's not a queue problem, it's a workflow. Use Step Functions.

4. Cold starts, honestly

Cold starts are the most over-discussed and least-measured topic in serverless. Here's the honest version.

A cold start is init + your code's module-load time, paid once when Lambda spins a new execution environment. Rough numbers on a warm-ish account:

When cold starts matter: low-traffic synchronous APIs, where each new concurrent request may hit a cold env. They also matter for spiky traffic that forces many simultaneous new environments, and for heavy runtimes. When they don't: steady-traffic functions, where envs stay warm, and anything async. Nobody is watching a spinner behind a queue, so a 2s cold start is invisible.

Your levers, cheapest first:

• Trim the package. Import only what you use. const { DynamoDBClient } = require("@aws-sdk/client-dynamodb"), not the whole SDK. This alone often halves a Node cold start. Free.
• SnapStart — Lambda snapshots an initialized environment and restores from it. Originally Java-only. Now also Python and .NET. Turns multi-second JVM starts into ~sub-second restores. Free to enable (you pay a small caching cost). Watch for uniqueness bugs: anything randomized at init (seeds, connection IDs) is frozen in the snapshot and restored identically across environments.
• Provisioned concurrency — keep N environments warm and pre-initialized. Kills cold starts entirely for that N. But you pay for it 24/7 whether traffic comes or not. This knob quietly converts serverless back into "servers you rent by the hour." Use it for a known, latency-sensitive endpoint. Drive it with Application Auto Scaling on a schedule. Don't sprinkle it everywhere.

5. Concurrency and scaling traps

Concurrency is an account-wide budget

The default account concurrency limit is 1,000 across all functions in a region. AWS caps burst too — you get an initial burst (500–3,000 depending on region) and then scale by 1,000 concurrent invocations per minute per function. This has a nasty consequence: one runaway function can starve every other function in the account.

SQS batch size × concurrency is a multiplication you must do on purpose

The SQS→Lambda scaler will start consumers up to your concurrency limit, each pulling batches. If each of 1,000 concurrent Lambdas opens a database connection, you now have 1,000 connections hitting a database that maxes out at 100. The queue that you built to protect the database becomes the mechanism that DDoSes it — faster than any human traffic could.

Downstream connection exhaustion — why RDS Proxy exists

This is the single most common way serverless meets a relational database and loses. Lambda scales by process count. Every concurrent invocation is its own process with its own connection. Postgres and MySQL are the opposite of Lambda. Connections are expensive, each costs memory, and there's a hard ceiling (see why your connection pool is probably wrong). 1,000 Lambdas want 1,000 connections. Your db.r6g.large wants maybe 100.

RDS Proxy exists precisely for this impedance mismatch. It sits between Lambda and the database and maintains a warm pool. The proxy multiplexes thousands of short-lived Lambda connections onto a small number of real database connections. It also handles failover faster. If you put Lambda in front of RDS/Aurora at any real concurrency, you need it. Or you need DynamoDB, which has no connections at all and is the reason it's the "native" serverless database.

6. Observability: the actually-hard part

Here's what nobody tells you: the hardest thing about serverless isn't building it, it's debugging it at 2 a.m. A request flows client → API Gateway → Lambda → EventBridge → SQS → Lambda → Step Functions. AWS scatters a single logical "order failed" across seven CloudWatch log groups, and no thread connects them. There's no server to SSH into. There's no single stack trace. There's just log soup.

The three things that make it survivable:

1. A correlation ID on everything. Generate one at the edge (or accept the API Gateway request ID), then thread it through every event payload, every log line, every message attribute. Without it, correlating logs across services is manual archaeology.

// structured logging — JSON, not console.log("thing happened")
const log = (level, msg, extra = {}) =>
  console.log(JSON.stringify({
    level, msg, correlationId, orderId,
    requestId: context.awsRequestId,
    ...extra,
  }));

log("info", "charge.started", { amount });
// → CloudWatch Logs Insights: filter @message like /correlationId":"abc-123/
//   now you can follow ONE order across every function it touched

2. Structured JSON logs. CloudWatch Logs Insights can query JSON fields. It cannot usefully query console.log("failed for user " + id). Log objects, not sentences. This is the difference between a 30-second query and a 40-minute grep.

3. Distributed tracing. X-Ray (or OpenTelemetry via the ADOT layer) draws the actual service map and shows you where the latency and errors live across the hops. Turn it on before you need it. You can't trace a request that already failed. It is the only tool that turns "the order is slow" into "the fulfillment Step Function waits 4 seconds on the inventory Lambda's cold start."

7. The cost math — and where it flips

The serverless pitch is "pay only for what you use, zero when idle." That's true, and it's genuinely unbeatable at low and spiky volume. The lie of omission is that per-request pricing has a crossover point where a steadily-loaded container gets cheaper — and it's lower than people expect.

The rough shape: Lambda bills per request + per GB-second of duration. A Fargate/ECS task bills per hour whether it's busy or not. So:

• Spiky, low-average, bursty traffic → serverless wins, often by a lot. You pay nothing between spikes. A container sits idle and burns money.
• High, steady, predictable traffic → containers win. Once a Fargate task is ~70%+ utilized around the clock, its per-request cost drops below Lambda's. The crossover is often somewhere in the millions-of-requests-per-day, steady range. Measure it for your duration and memory. A 2 GB / 800 ms Lambda flips much sooner than a 128 MB / 50 ms one.

8. Local development and testing — what actually works

The dream is "run the whole stack on my laptop." The reality is different. Heavy local emulation (LocalStack for everything, SAM local for complex event flows) drifts from real AWS behavior. It drifts in exactly the places that matter — IAM, throttling, eventual consistency, event timing. So it gives you false confidence. Here's what holds up in practice:

• Test handlers as pure functions. Structure code so the AWS wiring is thin and the logic is a plain function you can unit-test with no cloud at all. Handler parses the event → calls doTheThing(input) → formats the response. doTheThing is 90% of your logic and needs zero emulation. This is the highest-leverage testing decision you'll make.
• Integration-test against real AWS. Every developer (and CI) gets an ephemeral stack in a real account. Deploy is fast. The behavior is actually AWS — real IAM denials, real eventual consistency, real throttles. This catches the class of bug emulators cannot.
• Use emulators narrowly. LocalStack is genuinely fine for a single DynamoDB table in a fast unit test. It is a trap as a substitute for an integration environment.

9. When NOT to go serverless

The honest part. Serverless is a fantastic default for event-driven, spiky, glue-heavy workloads. It's the wrong tool when:

• Steady high throughput. If you run flat-out 24/7, you pay serverless's per-request premium for elasticity you never use. A right-sized Fargate/ECS or EKS deployment is cheaper and simpler. (See the crossover in §7.)
• Long-running or compute-heavy jobs. Lambda caps at 15 minutes. Video transcoding, big ML inference, long ETL — that's Fargate, Batch, or EC2, not Lambda. Shoehorning it in leads to grotesque Step Function fan-out just to dodge a timeout.
• Sustained low latency with zero tolerance for cold starts. If a P99.9 cold start is unacceptable and provisioned concurrency for your whole surface would cost more than just running warm containers, run the containers.
• Chatty inter-service calls / heavy connection reuse. Your workload may be dozens of calls to the same database, or a persistent connection (WebSockets with server-side state, gRPC streaming). A stateful container that keeps warm connections is a better fit than 1,000 stateless processes fighting over RDS Proxy.
• You want portability. Deep serverless is deep AWS lock-in — EventBridge rules, Step Functions ASL, DynamoDB access patterns don't port. A container does. Sometimes that trade is worth it. Make it on purpose.

The takeaway

After enough production time, the serverless lessons compress into a few opinions I'd defend:

• Serverless doesn't remove complexity, it relocates it. The complexity moves out of your servers and into the seams between managed services. The seams are harder to see, because there's no box to log into. You traded ops toil for distributed-systems reasoning. That's often a good trade. It is never a free one.
• Idempotency is the price of admission, not an optimization. Everything retries. If you can't run a handler twice safely, you don't have a serverless system, you have a time bomb with good uptime.
• Every DLQ needs an alarm, every async path needs a failure destination, and every SQS mapping needs ReportBatchItemFailures. These three settings are the difference between "resilient" and "silently losing data."
• Observability is the real cost, in dollars and in effort. Budget for correlation IDs and structured logs on day one, and budget for the CloudWatch bill — both will surprise you if you don't.
• Do the cost math with your own numbers. "Serverless is cheaper" and "serverless is more expensive" are both true, on different sides of a crossover point. That point depends on your traffic shape, your memory, and your duration. Measure it. Don't argue it.
• Pick serverless for the shape of the work, not the hype. Spiky and event-driven? Reach for it. Steady and heavy? Reach for a container. The senior move is knowing you're allowed to use both in the same system — and usually should.

Serverless in production is not "no servers." It's "someone else's servers, your distributed system, and a bill that tells the truth about your architecture." Design for the failures nobody demos, and it's one of the best tools AWS has ever shipped.