Slack Sends Messages Over HTTP, Not Its WebSocket — and the Durability Bug That Forced the Switch

Tags: WebSockets & Real-time · Real Problems · Distributed Systems · HTTP · Reading time: ~20 min · Category: Real Problems → WebSockets & Real-time

There is a fact about Slack’s architecture that surprises almost everyone who learns it for the first time:

When you press Enter on a Slack message, your client does not send that message over the WebSocket connection it holds open. Your client sends it in a separate HTTPS POST request to chat.postMessage.

Slack has a WebSocket. It uses it heavily. But think about the one operation you would most expect a WebSocket to handle. That operation is the user sending a message. Slack ignores the WebSocket and uses HTTP for it.

This isn’t a quirk. It’s a deliberate design decision that came from a class of durability bug. That bug hurt early real-time products badly. This post explains exactly what that bug looks like. It explains why WebSocket sends are fundamentally not the right tool for the job. You will see what HTTP gives you that the WebSocket protocol does not. The post also covers the receive-side migration (RTM API → Events API). That migration made the same architectural rule public for every app developer.

Part 1 — What people think Slack does vs. what it actually does

The intuitive mental model of a real-time chat app is something like this:

[Client] ──── WebSocket (everything) ────► [Server]
        ◄─── WebSocket (everything) ─────

One pipe. Bidirectional. Sends go up the pipe, events come back down the pipe. Simple, low-latency, “real-time.”

Slack’s actual model splits the two directions:

                       SEND PATH
[Client] ─── HTTPS POST /api/chat.postMessage ───► [Slack API]
         ◄── HTTP 200 { ok:true, ts, channel } ──

                      RECEIVE PATH
[Client] ◄── WebSocket frames (events) ──── [Slack edge/gateway]
          (typing, new messages from others, reactions, presence…)

The send and receive paths use different transports because they have different requirements. The send path needs durability and acknowledgement. The receive path needs low-latency push. One transport is good at exactly one of those things.

This split shows up in two other places at Slack too:

• Bots and apps. Slack’s original Real Time Messaging (RTM) API delivered events to bots over a WebSocket. In 2017, Slack introduced the Events API, which delivers events to bot servers over HTTPS POST webhooks. Slack now strongly discourages RTM for new apps. The official guidance: receive events via HTTP webhooks, take actions via HTTP API calls.
• Slash commands & interactivity. A user runs a slash command or clicks a button in a Slack message. Slack then calls your app with an HTTPS POST. Slack does not use any open WebSocket your app might have. The pattern is consistent: actions that must not be lost go over HTTP.

Part 2 — Why the obvious answer (“send over the WebSocket”) is wrong

If you already have a WebSocket open, sending the message over it looks like the cheapest, fastest option. Zero new handshake, zero extra round trips, the connection is right there. So why doesn’t Slack do that?

Because WebSocket.send() does not mean what you think it means.

What WebSocket.send() actually returns

In the browser, WebSocket.send(data) returns undefined. It can throw if the connection is already closed. But if it returns normally, here is what it actually guarantees:

1. The browser queues the data into the WebSocket’s internal buffer.

That’s it. That’s the whole guarantee.

It does not guarantee that the bytes left your machine. The browser hands them to the kernel’s TCP send buffer. That buffer hands them to the NIC. The NIC puts them on the wire, eventually. You can inspect WebSocket.bufferedAmount to count the bytes that still wait in the WebSocket’s queue for TCP. But even bufferedAmount === 0 only tells you that the browser finished with them. The kernel might still hold them.

And even when the bytes reach the server’s kernel, the server process may not have read them yet. And even after the server process reads them, it may not have parsed the frame. It may not have validated it, persisted it, or fanned it out to other clients.

The RFC 6455 gap

The WebSocket protocol (RFC 6455) defines six frame types: continuation, text, binary, close, ping, and pong. None of them is an application-layer acknowledgement.

TCP underneath provides byte-level reliability — bytes arrive in order, with retransmission. But TCP’s ACKs are between kernels, not between applications. When the client kernel’s TCP stack sees an ACK come back, all it learned is that the receiving kernel got the bytes. The receiving application process might not have called recv() yet. It might never call recv() because it just crashed.

HTTP’s genius is that the response is the application-layer ack. A 200 OK arrives only after the server’s handler ran. A 5xx means the server tried and failed. A timeout means you do not know. Each request is a small, scoped contract.

WebSocket has no equivalent. You can build one on top, and most production systems do. But every team builds it differently. To get it right costs the same work as just using HTTP.

Part 3 — The durability bug, walked through carefully

Let’s trace exactly what goes wrong when you send a chat message over a WebSocket without an application-layer ack.

The setup

A naive design:

// Client
ws.send(JSON.stringify({
  type: "message",
  channel: "C123",
  text: "Shipping the release in 5 min",
}));
// UI immediately shows the message as "sent"

// Server (Node-ish pseudocode)
ws.on("message", async (raw) => {
  const msg = JSON.parse(raw);
  await db.insert("messages", msg);
  await fanout.broadcast(msg.channel, msg);
});

It works the majority of the time. The bug lives in the unhappy paths.

Failure mode 1: the connection dies between send and persist

[Client]
  ws.send(msg)                              ✓ queued in browser buffer
  TCP hands bytes to kernel                 ✓
  bytes leave NIC                           ✓
  ── packet in flight ──
                                  💥 server process OOM-killed
                                     between recv() and db.insert
  TCP eventually RSTs the socket
  Client gets a "close" event
  msg was never written to the message store

From the user’s perspective, the message looks sent — the UI showed it instantly. From the server’s perspective, the message never existed. Refresh the channel: it’s gone. Talk to the recipient: they never got it.

This is the bug. The client thought it shipped a durable message. The server made no such promise. There was no ack to wait for, so the client never knew the difference.

Failure mode 2: persist succeeded, the “you’re sent” signal was lost

[Client]
  ws.send(msg)
                                  server recv() ✓
                                  db.insert ✓
                                  fanout.broadcast ✓
                                  server sends ack frame
  ── ack packet drops ──
  Network blips, socket closes
  Client never sees the ack
  Client reconnects, retries the send
                                  server recv() ✓
                                  db.insert  → duplicate row (or duplicate fanout)

Same physics as the “two generals” problem. The client can attach an idempotency key to the second attempt. Without that key, the server cannot tell two cases apart. The first case is a retry of the message it already saved. The second case is a brand new message that only looks similar. Two copies in the channel. Two notifications on the recipient’s phone. Probably a support ticket.

Failure mode 3: half-open socket, send disappears into the void

WebSocket connections sit on top of TCP, and TCP’s ESTABLISHED state lies. A NAT or load balancer can drop the flow mapping after an idle period (AWS ALB defaults to 60s of idle). Neither end learns this until something tries to send.

[Client]                              [LB]                       [Server]
  WebSocket ESTABLISHED  ─────────  flow mapping  ─────────  ESTABLISHED
                                    💥 evicted after 60s idle
  ws.send("important msg")
  Bytes go into TCP send buffer
  Retransmit, retransmit, retransmit…
  Eventually RST, ~15 min later by default
  Client app may not surface that for minutes

The message looks sent. There is no error event. The user moves on. Fifteen minutes later the client surfaces a connection error. But the “sent” message is gone. The client buffered it into a socket whose other end no longer existed.

You can layer application heartbeats on top to catch this faster (and you should — see TCP Keep-Alive vs Application Heartbeat). But none of that gives you per-message durability. It only tells you faster that the pipe is broken.

Failure mode 4: server fanned out before persisting

To minimise latency, an early implementation might broadcast the message to other clients first, then write it to the database. If the database write fails after the fanout, other people in the channel saw a message that doesn’t exist. If they reply, their reply references a parent message ID that’s not in the store. If they leave and re-open the channel, the original message is gone but the reply is there.

HTTP’s request/response shape pushes you toward the right order naturally. You cannot return 200 ok until persistence succeeded. Durability therefore gates the “message visible” signal. WebSocket has no such structural pressure. The temptation to fanout first is real, and several real-time apps fell into it.

Part 4 — What HTTP gives you that raw WebSocket does not

Now flip the design. The send path goes through HTTPS:

// Client
const res = await fetch("/api/chat.postMessage", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": `Bearer ${token}`,
  },
  body: JSON.stringify({
    channel: "C123",
    text: "Shipping the release in 5 min",
    client_msg_id: "5b3c…-uuid",  // idempotency key
  }),
});
const { ok, ts, channel } = await res.json();

Look at what you get for free with this shape:

The idempotency key, in detail

The Slack message payload includes a client_msg_id — a UUID generated by the client. The server stores it alongside the message row. A first POST may time out, and then the client does not know whether it succeeded. If the client retries, the second POST carries the same client_msg_id. The server sees a uniqueness conflict and returns the existing message’s metadata (the ts assigned the first time) instead of inserting a duplicate.

CREATE TABLE messages (
  ts             TEXT PRIMARY KEY,           -- server-assigned
  channel        TEXT NOT NULL,
  user_id        TEXT NOT NULL,
  text           TEXT NOT NULL,
  client_msg_id  TEXT NOT NULL,
  UNIQUE (channel, client_msg_id)            -- dedup key
);

The retry is safe. Here is the UI behaviour: “Sending…” → either “Sent” (200, possibly after one transparent retry) or “Failed, tap to retry” (4xx/5xx after backoff). The user never sees a phantom message.

You could implement the same scheme over WebSocket. The framing differences don’t matter — JSON is JSON. But once you’ve added per-message IDs, an ack frame, retry-with-backoff, server-side dedup, and a state machine for in-flight messages, you have rebuilt a worse HTTP. With more bugs.

Load balancing is the other half

WebSockets are a load-balancing problem. The connection is sticky to one backend for its entire lifetime — minutes, hours, sometimes days. Deploying a new version means either gracefully draining millions of connections (slow), or just dropping them (chaos). A surge of new users hitting the same edge means rebalancing connections you can’t easily move.

HTTP requests are short-lived and stateless. The load balancer can route each chat.postMessage POST to whichever message-service instance is least loaded. A deploy is a rolling restart with zero user-visible impact. Capacity scales linearly with request rate, not connection count.

Putting the send path on HTTP means the most important operation in the product runs on the easiest-to-operate transport. Slack reserves the hard transport (WebSocket) for the use case that genuinely needs it. That use case is low-latency push of other people’s events.

Part 5 — The receive side: RTM → Events API, the same lesson in public

For Slack’s own clients, the receive side is still a WebSocket. That is fine, because the receive side is a hint, not a source of truth. (We’ll come back to why this is fine in Part 6.)

But for bots and third-party apps, Slack made the same architectural decision visible to the outside world. The history is worth tracing.

RTM API (legacy) — WebSocket events to your bot

The original way to write a Slack bot: call rtm.connect and take the wss:// URL it returns. Then hold the WebSocket open and read a stream of event JSON. Every message in every channel your bot was a member of came down that pipe.

The problems were the same ones that motivated the send-side decision:

• If your bot lost the connection, Slack dropped the events. Slack maintained a small replay window after reconnect, but it was best-effort. A long disconnect = missing events.
• No per-event acknowledgement. If your bot received an event but crashed before processing it, Slack had no way to know — no retry.
• Hard to operate. A bot serving many workspaces had to hold many WebSocket connections, manage reconnect storms, handle backpressure manually.
• One bug took down everything. A bug in your event loop could starve every event for every workspace, with no isolation per request.

Events API (2017) — HTTP webhooks to your bot

The replacement: Slack POSTs each event to an HTTPS endpoint you register. Your endpoint returns 200. If you return anything else, or time out (3 seconds), Slack retries — up to 3 retries spread over an hour, with backoff. Every retry carries an X-Slack-Retry-Num header so you can tell. Every event has a unique event_id so you can deduplicate.

This is the same trade Slack made on the send side, made public for app developers: important deliveries go over HTTP. The transport with the built-in ack semantics, the per-request scope, the standard middleware. Slack reserves WebSocket for the cases where push latency matters more than per-event durability.

Socket Mode (2020) — the nuance

Socket Mode returned because some apps cannot expose a public HTTPS endpoint. Think of a CLI tool running on a laptop, or a corporate bot behind a firewall. So Slack opens a WebSocket back to your app and tunnels Events API messages through it. The wire is a WebSocket, but the contract is still HTTP-shaped. Each event has an envelope_id. Your app must send back an explicit ack, and Slack retries unacked events.

That distinction is the whole point of this post. The transport is not the architecture. Slack didn’t go back to fire-and-forget WebSocket. They kept the per-message acknowledgement model and only changed how the bytes move. The bug they guarded against (silent message loss) is a bug at the application protocol layer, not the transport layer. They fixed it at the right layer.

Part 6 — “WebSocket as hint, REST as truth”

If sends are over HTTP, what is the WebSocket actually carrying in a Slack client?

• Other people’s messages. The fanout push.
• Typing indicators, presence changes, reactions, edits. Low-value, high-frequency hints where loss is acceptable.
• Push wake-ups. "Something changed in channel C123 — go re-fetch if you care."

The key design pattern: the WebSocket message is a hint, the REST API is the truth. The client treats every event as “go check.” On reconnect, the client asks for state since its last known cursor:

GET /api/conversations.history?channel=C123&oldest=1717070400.000123

If the WebSocket missed three events while the laptop slept, the client doesn’t care. A catch-up REST call fills the gap. The UI shows whatever the REST call returns. The WebSocket just shortens the window between “something happened” and “the UI knows.”

This is the pattern that survives a flaky network, a server restart, a deploy, a Wi-Fi-to-cellular handoff, and a closed laptop lid. It’s the pattern Slack converged on. It’s the pattern Discord, Linear, Notion, Figma, and Google Docs all use. This is also the pattern that scales. The expensive operations (durable writes) run on the cheap-to-operate transport (HTTP). The cheap operations (hints) run on the expensive-to-operate transport (WebSocket).

Part 7 — When you should (and shouldn’t) do this

The takeaway is not “WebSockets are bad” or “always use HTTP.” It’s that sends and receives have different reliability requirements, and one transport rarely satisfies both.

If you find yourself reaching for a WebSocket for the send path because “the connection is already there,” stop and ask:

• What happens if the connection drops between send() and the server’s persist?
• How does the client know the message was actually saved?
• How do you retry a send safely without duplicates?
• What does the UI show in the ambiguous case where you sent and don’t know if it landed?

If your answers are vague, you have the same bug Slack’s early architecture had. The fix is not to make the WebSocket smarter. The fix is to send the message over a transport that already has an ack built into it.

The honest one-liner

Slack sends over HTTP not because HTTP is faster (it isn’t), and not because WebSockets are unreliable at the transport level (TCP is fine). It sends over HTTP because the HTTP request/response shape forces three things: an application-layer acknowledgement, an idempotency model, and a stateless load-balancing story. A raw WebSocket leaves you to invent all of that yourself. One operation in the product cannot afford to lose a message. That operation is the user pressing Enter. For it, the cheapest correct answer is to use the protocol that was already designed for it.

The WebSocket is still there. It is doing the job it’s good at: pushing other people’s activity to your client with low latency, so the UI feels alive. The REST API is doing the job it’s good at: being the source of truth that the client trusts when the WebSocket misses something.

Two transports. Two jobs. One product that doesn’t lose your messages.