AWS Lambda Under the Hood: Firecracker, Cold Starts, and the Concurrency Model Nobody Reads the Docs For

Tags: AWS · Lambda · Firecracker · Cold Starts · Concurrency · SnapStart · Serverless · Reading time: ~20 min · Category: AWS → Scaling & Infra

Most people treat Lambda as a black box: you upload a zip, it runs, you get billed by the millisecond. That model is fine until the day p99 latency triples for no reason you can see in your code. Or until your function starts throwing TooManyRequestsException at a fraction of the concurrency you thought you had. At that point you have to open the black box. What’s inside is one of the more interesting pieces of systems engineering AWS has ever shipped. It is a fleet of bare-metal machines that slice themselves into tens of thousands of hardware-isolated microVMs. They boot a Linux guest in single-digit milliseconds, and they freeze your process mid-flight to reuse it later. This post is the opened box.

I’m going to assume you’ve written a Lambda before and skip the “what is serverless” tour entirely. Instead: how Lambda routes an invocation, what the INIT phase really does, and why some runtimes eat 800 ms of it. Then how concurrency scaling ramps and throttles, and the freeze/thaw lifecycle you can exploit for connection reuse. Then the VPC networking history that used to add ten seconds to a cold start. Last, the retry rules that differ in ways that will silently double-process your payments if you don’t know them. Then the part nobody writes: when Lambda is the wrong answer.

The physical machine: Firecracker and the Worker fleet

Underneath every invocation is a plain EC2 bare-metal instance AWS calls a Worker. Lambda does not run your code in a container in the Docker sense, and it has not since 2018. It runs your code inside a Firecracker microVM. Firecracker is a purpose-built virtual machine monitor (VMM) that AWS wrote in Rust and open-sourced, and it sits on top of KVM. Firecracker exists for one hard requirement. AWS must run untrusted code from thousands of different accounts on the same physical host. The security boundary must be strong enough that a kernel exploit in one function cannot reach another customer’s data.

Containers alone could not give AWS that. A container is a process with namespaces and cgroups. It shares the host kernel, and the host kernel is a very large attack surface. A traditional VM gives you a real hardware boundary. But it boots in seconds, and it carries hundreds of MB of memory overhead per instance. That cost is economically impossible when you pack hundreds of tiny functions onto one host. Firecracker sits in between. It is a stripped-down VMM, and it emulates only the four or five devices a serverless guest actually needs. Those devices are virtio-net, virtio-block, a serial console, and a one-button keyboard controller for clean shutdown. It emulates nothing else — no BIOS, no PCI, no USB, no VGA.

Here is the key mental shift. An execution environment is the unit you think of as “my warm Lambda”. It is one Firecracker microVM, pinned to one function version, and it handles exactly one invocation at a time. It is not a server handling concurrent requests. If ten requests arrive at once, you need ten microVMs. That single fact drives the entire concurrency model, and half the production surprises in this post trace back to it.

How an invocation is actually routed

When a request lands, it does not go straight to your code. It flows through a control plane. That control plane exists to answer two questions: is there a warm microVM leased to this function version that is idle right now? And if not, where do I make one?

The Placement service is the interesting part. It solves a bin-packing problem under constraints. The service packs microVMs densely onto Workers to keep utilization high, and that density is how Lambda stays cheap. It also obeys the isolation rules. Most important, it avoids too many microVMs from the same function on the same Worker, so a single bad host cannot remove all your capacity. When no warm slot exists, placement leases a slice of a Worker, and the cold start clock starts.

On the warm path, all of this collapses. The environment is already leased, already has your code, already ran INIT. The frontend hands the event to an idle microVM. You pay only the network hop plus your handler time — typically low single-digit milliseconds of platform overhead. This is why the exact same function shows a bimodal latency distribution: a tight warm cluster and a long cold tail. If you have ever stared at a latency histogram with two humps and blamed the network, that second hump is cold starts.

Cold starts, phase by phase

“Cold start” is a fuzzy word for a very specific sequence. When placement decides to build a new environment, three things happen in order. Only the middle one is code you wrote:

1. Environment creation. Placement leases a Worker slot, Firecracker creates the microVM, the guest kernel boots. AWS absorbs this time. It is not on your bill, but it is on your latency.
2. Code download & runtime bootstrap. Lambda pulls your deployment package (zip or container image) and makes it available to the microVM. The language runtime process then starts. Package size matters here — a 250 MB zip is not free to stage.
3. INIT: your initialization code. Everything outside your handler function runs exactly once per environment. That means imports, static initializers, SDK client construction, reading a config from Parameter Store, and opening a DB connection. Lambda bills this as init duration, and this is where you have leverage.

Here is the number that changes how you write functions once you understand it: init code gets up to 10 seconds of “free,” fully-provisioned CPU. Historically, Lambda did not meter the CPU during INIT the way it meters handler CPU. So move heavy one-time setup into module scope rather than inside the handler. That means deserializing a model, JIT warmup, or building a connection pool. This choice is almost always correct. The setup runs once per environment, and every warm invocation that environment ever serves shares the cost.

Why some runtimes are dramatically slower to start

The runtime and your code do work before the first request, and that work dominates cold start cost. The amount varies widely by language, and the ordering is not subtle:

War story. A team I worked with put a Spring Boot app behind API Gateway on Lambda because “it’s just a REST controller.” Warm, it answered in 30 ms. Cold, it took 6.5 seconds. Spring’s component scan and bean wiring ran on every fresh environment. Their traffic was spiky enough that a meaningful fraction of requests hit cold environments. Their fix wasn’t “tune Spring”. They stopped pretending that Spring was a good fit. A framework built to start once and run for months does not suit a runtime that starts thousands of times a day. They moved the classpath scan off the startup path with Spring Cloud Function and functional bean registration. That knocked it to ~2.5 s. Then they added provisioned concurrency for the baseline. The real lesson: cold start cost is a property of your startup work, and heavyweight frameworks make that work enormous.

SnapStart: cheat the boot with a memory snapshot

AWS’s answer to the JVM problem is SnapStart. Lambda does not boot fresh every time. It runs your INIT once at publish time, then takes a Firecracker snapshot. That snapshot is a full serialized image of the microVM’s memory and CPU state. Lambda encrypts it and caches it. On a cold invocation it restores from that snapshot instead of re-running INIT. A restore of a pre-warmed memory image costs far less than class loading and framework wiring. Teams routinely see JVM cold starts drop from multiple seconds to a couple hundred milliseconds.

Provisioned concurrency: pay to keep them warm

SnapStart makes cold starts cheaper. Provisioned concurrency (PC) makes them disappear. You tell Lambda “keep N environments initialized and ready,” and it pre-runs INIT on N microVMs and holds them hot. Invocations up to N never see a cold start. Beyond N, Lambda sends the extra invocations to regular on-demand concurrency and its cold starts, unless you have also capped things.

The trap with PC is to treat it as a fire-and-forget setting. Traffic patterns move. A PC of 50 provisioned for last quarter’s load is 50 idle environments you pay for at 3 a.m. Drive it with Application Auto Scaling on a schedule or a utilization target. Always keep on-demand concurrency available as a spillover, so a traffic spike past your PC ceiling degrades to “some cold starts” rather than throttling.

The concurrency model, precisely

This is where most production incidents actually originate, and where the docs are easy to misread. Start from the invariant: concurrency = number of invocations executing simultaneously. One microVM serves one invocation. So concurrency is literally a count of busy microVMs.

By Little’s Law, your steady-state concurrency is request_rate × average_duration. 100 req/s at 200 ms average = 20 concurrent executions. 100 req/s at 2 s average = 200. This is the single most useful formula for capacity-planning Lambda. It is also why slow functions are expensive in ways the per-ms price does not show. They hold environments hostage.

The three flavors of concurrency

How scaling actually ramps — burst limits and throttling

Lambda does not hand you your full account limit instantly. A burst allowance comes first. A function can immediately scale by a fixed increment, per-function and tied to a rolling window. Beyond that burst, concurrency grows at a bounded rate until it either meets demand or reaches the account limit. Modern Lambda scales each function by up to 1,000 additional concurrent executions every 10 seconds, roughly +1,000 per 10 s per function. That is far better than the old shared-token behavior. But it is still not infinite and still not instantaneous.

When demand outruns the ramp, you get throttling, and what happens next depends entirely on invocation type:

• Synchronous (API Gateway, ALB, direct SDK invoke): the caller gets a 429 TooManyRequestsException immediately. Retrying is your problem. If your API Gateway integration does not handle it, that is a 5xx to the user.
• Asynchronous (S3 events, SNS, EventBridge): the event went to an internal queue first. Lambda retries throttled events automatically with backoff for up to ~6 hours before it stops. Throttling here is mostly invisible — until the queue grows and end-to-end latency balloons.
• Poll-based / stream (SQS, Kinesis, DynamoDB Streams): the event source mapping (a Lambda-managed poller) controls the rate. Throttling appears as slower draining and, for ordered streams, head-of-line blocking on the shard.

The execution environment lifecycle: freeze, thaw, reuse

After your handler returns, Lambda does not destroy the environment. Lambda freezes it. The freeze pauses the entire microVM and suspends its process mid-memory-state. Lambda keeps the environment around to serve the next invocation for that function version. When the next request arrives, Lambda thaws it: unpaused, memory intact, your global state exactly as you left it. Lambda eventually reaps idle environments. There is no published TTL, and the delay is deliberately variable. Think minutes, not seconds, but do not depend on a number.

This freeze/thaw model is the single biggest performance lever you have. It is entirely about what you put in global scope vs. handler scope:

import os
import boto3
import psycopg2

# ─── GLOBAL SCOPE: runs once per environment, survives freeze/thaw ───
# Reused across every warm invocation this microVM ever serves.
_secrets = boto3.client("secretsmanager")
_db_creds = _secrets.get_secret_value(SecretId=os.environ["DB_SECRET"])
_conn = psycopg2.connect(**parse(_db_creds))   # opened ONCE, kept warm
_conn.autocommit = True

def handler(event, context):
    # ─── HANDLER SCOPE: runs every invocation ───
    # Reuse the warm connection. Do NOT open a new one here.
    with _conn.cursor() as cur:
        cur.execute("SELECT id FROM orders WHERE user_id = %s", (event["user"],))
        return {"orders": cur.fetchall()}

Open the DB connection in global scope, and thousands of invocations share one connection. Open it inside the handler and you pay the TCP + TLS + auth handshake every single request. Worse, you can exhaust your database’s connection limit. 500 concurrent Lambdas that each open a connection are 500 connections your Postgres box did not agree to.

/tmp as a warm cache

Every environment gets writable /tmp (default 512 MB, configurable up to 10 GB), and — like global variables — it survives freeze/thaw. That makes it a per-environment disk cache. Downloading a 200 MB ML model from S3? Cache it in /tmp on first invocation and check for it on subsequent ones — warm invocations skip the download entirely.

import os, boto3

MODEL_PATH = "/tmp/model.bin"
_s3 = boto3.client("s3")

def _ensure_model():
    if not os.path.exists(MODEL_PATH):        # cold env → download once
        _s3.download_file("my-bucket", "model.bin", MODEL_PATH)
    return load(MODEL_PATH)                    # warm env → already on disk

def handler(event, context):
    model = _ensure_model()
    return model.predict(event["features"])

VPC networking: the ten-second cold start that used to ruin lives

This one deserves its own section. It is the biggest architectural change AWS ever made to Lambda. If you learned Lambda before ~2019, your instincts here are wrong.

The old world (pre-2019). To reach private resources, such as an RDS instance or an ElastiCache cluster, you had to attach a Lambda to your VPC. Back then a VPC attachment meant one thing. On every cold start, Lambda created an Elastic Network Interface (ENI) in your subnet and attached it to the microVM. ENI creation is slow, because it involves the EC2 control plane, and it added up to 10 seconds to cold starts. Worse, each concurrent execution consumed an ENI, so high-concurrency VPC functions could exhaust the IP space in a subnet. VPC Lambda was borderline unusable for latency-sensitive work, and the standard advice was “don’t put Lambda in a VPC unless you absolutely must.”

The new world (Hyperplane). AWS re-architected it on Hyperplane, the same internal NAT/network-virtualization fabric that powers NLB and PrivateLink. Now Lambda creates an ENI once per function+subnet+security-group combination. All the environments of that function share it through a Hyperplane VPC-to-VPC NAT. Lambda builds it lazily and out-of-band, not on the hot path of each cold start. The result: VPC-attached cold starts went from “up to +10 s” to sub-100 ms of added latency. ENI count now scales with unique network configs, not with concurrency.

Retries, idempotency, and where events go to die

Retry behavior is not uniform across Lambda. That assumption has cost more than one team a double-charged customer. The invocation type is the whole story:

Because retries exist, every Lambda that mutates state must be idempotent. “At-least-once” is the honest delivery guarantee across almost all of these paths. Exactly-once is a fiction you have to build yourself. The standard pattern is an idempotency key: derive a deterministic key from the event, record it conditionally, and short-circuit duplicates.

import boto3
from botocore.exceptions import ClientError

_ddb = boto3.resource("dynamodb").Table("idempotency")

def handler(event, context):
    key = event["order_id"]           # deterministic, per-logical-operation
    try:
        # Conditional put: only succeeds if we've NOT seen this key.
        _ddb.put_item(
            Item={"pk": key, "ttl": ttl_24h()},
            ConditionExpression="attribute_not_exists(pk)",
        )
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return {"status": "duplicate, ignored"}   # already processed
        raise

    charge_customer(event)            # the non-idempotent side effect
    return {"status": "charged"}

DLQs vs. failure destinations

Both catch async invocations that exhausted their retries, but they are not the same tool:

• Dead-letter queue (the older mechanism): on final failure, Lambda sends the original event payload to an SQS queue or SNS topic. You get the input, but no context about why it failed.
• Failure destinations (newer, richer): route on both success and failure to SQS, SNS, EventBridge, or another Lambda. The record includes the invocation context: request ID, error message, stack trace, and the response. Strictly more useful for debugging.

Cost and performance tuning: the memory dial is a CPU dial

The most counter-intuitive and most valuable Lambda fact: the memory setting is really a compute setting. Lambda allocates CPU (and network, and I/O) proportionally to memory. At 1,769 MB you get roughly one full vCPU. Below that you get a fraction. Above it you get more than one vCPU, up to ~6 vCPUs at the 10,240 MB ceiling. You do not configure CPU directly. You buy it by the MB.

This creates a genuinely non-obvious result: more memory is often cheaper. Lambda bills you for allocated_memory × duration. Suppose you double the memory, and the duration more than halves. Then the bill goes down while latency also goes down. The duration drops because you went from a fraction of a CPU to a full one, or because your workload is CPU-bound. It is the rare knob that improves both axes at once.

The shape is almost always the same. Cost falls or stays flat as you add memory, up to the point where the function is no longer CPU-starved. Then it rises, once you pay for cores the workload cannot use. The sweet spot is the elbow. Don’t guess at it — use AWS Lambda Power Tuning, Alex Casalboni’s open-source Step Functions state machine. It runs your function across a sweep of memory settings and plots cost against speed. It tells you the optimum for whichever axis you optimize. I have never run it on a function tuned by gut and found the gut was right.

When NOT to use Lambda

Lambda is very good, and it is also frequently the wrong tool. The serverless-everything crowd rarely says so out loud. The honest trade-offs:

The takeaway

If you remember nothing else, remember that almost every Lambda surprise traces back to one physical fact and its consequences:

1. One microVM, one invocation at a time. Concurrency is a count of busy Firecracker microVMs. That is why 800 concurrent executions means 800 DB connections. Slow functions are expensive for the same reason, by Little’s Law: rate × duration. And you cannot share a pool across environments. Put expensive setup in global scope, so every invocation in the freeze/thaw lifecycle shares the cost. Put a connection pooler (RDS Proxy) between Lambda and any relational database.
2. Cold starts are your startup work, made visible. The platform’s share is tens of milliseconds. The rest is your imports, your framework, your init code. Keep it lean. Reach for SnapStart on the JVM, and re-seed randomness in afterRestore. Use provisioned concurrency only where a hard latency SLA justifies paying for warm capacity.
3. Know your invocation type cold. Sync throws 429, and retries are your problem. Async retries twice, then needs a destination or the event is gone. Streams block the shard on a poison record. Every mutating handler must be idempotent, because at-least-once is the real guarantee.
4. The memory slider is a CPU slider — power-tune it, and don’t be surprised when more memory is cheaper and faster.
5. And know when to walk away. Steady saturation, long jobs, GPUs, and hard-real-time latency belong on containers. Lambda is a scalpel for spiky, event-driven, stateless work — not a hammer for every server you’d rather not operate.

The reason to understand the internals isn’t trivia. Once you can see the Firecracker microVM, the placement hop, and the freeze/thaw cycle behind your handler, the weird production behavior stops being weird. The bimodal latency, the connection storms, the mysterious throttling at 40% of your limit all become obvious, predictable, and fixable. The black box was never magic. It was just a very well-engineered set of trade-offs you now get to make on purpose.