Subscription Resilience and Delivery Guarantees

At-Most-Once and Why That Scares Finance

GraphQL subscriptions are fire-and-forget. If the socket drops between the moment the server writes a frame and the moment the client reads it, that event exists nowhere. The protocol has no acknowledgement, no replay cursor, and no redelivery — the guarantee is at-most-once.

That is acceptable for some consumers and unacceptable for others:

The rule that follows: a subscription is a fast path, never the only path to a fact that matters. Anything important must also be readable through a query the client can run after reconnect. Deleting the poll "because we have subscriptions" removes the safety net the subscription was built on.

Closing the loop when loss is visible

Streams where loss shows up to a human need explicit acknowledgement: the server tracks events it has written but not confirmed, and the client confirms ids as it persists them.

server:  next id:41   (pending)
server:  next id:42   (pending)
client:  mutation acknowledge(ids: [41, 42])   -> pending cleared
client disconnects before acking 43
reconnect: client asks for events after its last stored id
server:   redelivers 43 from the replay buffer

Acknowledgement converts at-most-once into at-least-once, which moves the work onto the client: every consumer must deduplicate by id before applying. That cost is worth paying for the payment feed row in the table and never for chat, where a duplicate message is more annoying than a missing one.

There is also a category no transport fixes: events that were never published because the producer failed between committing the database write and calling publish. A transactional outbox — write the event in the same transaction as the state change, then publish from a relay — is the fix, and it is orthogonal to which GraphQL transport you chose.

Resume Points and Bounded Buffers

Resume is two mechanisms cooperating: an id on every event and a buffer that can still produce anything a client asks for.

class ReplayBuffer {
  constructor(limit = 500) {
    this.limit = limit;
    this.items = [];
    this.seq = 0;
  }

  push(payload) {
    const id = ++this.seq;
    this.items.push({ id, payload });
    if (this.items.length > this.limit) this.items.shift(); // drop oldest
    return id;
  }

  since(lastEventId) {
    if (lastEventId == null) return []; // fresh subscriber: live from now
    const gap = this.seq - lastEventId;
    if (gap > this.limit) return null; // too old: client must refetch
    return this.items.filter((e) => e.id > lastEventId);
  }
}

since() returning null is the important branch. Silently streaming from the oldest buffered event pretends a continuity that does not exist; telling the client "your gap is too large, refetch state" is the honest contract, and the client answers with a normal query that rebuilds its view.

Note that ids are per topic, not global. A client resuming a chat room only needs that room's sequence, which keeps buffers small and lets different topics keep different retention windows.

Slow consumers

Every subscriber that reads slower than the publish rate grows a queue somewhere. Unbounded, that queue is an out-of-memory incident with extra steps. Pick a policy per topic and write it down:

• Drop oldest — right for tick and metric streams where the newest value is the only one that matters.
• Drop newest, counted — right when the client should detect loss and resync rather than silently skip.
• Disconnect the consumer — right when a silent gap would corrupt the client's model; closing the socket forces a deliberate reconnect and refetch.

if (queue.length >= MAX_QUEUE) {
  metrics.increment('subscription.slow_consumer_dropped');
  queue.shift();
  if (queue.length >= MAX_QUEUE * 2) {
    socket.close(1013, 'slow consumer'); // 1013 = try again later
  }
}

Bound the queue, count the drops, alert on the count. A rising slow_consumer_dropped is the earliest signal that one consumer — or one region — is falling behind, and it is far better than the alternative: an unbounded buffer that fails for everyone at once.

Keeping the Broker Path Honest

In a multi-instance deployment the delivery chain is:

publish -> broker -> each instance's local subscriber table -> socket write

Each hop fails differently, and they do not fail together.

• Publish without a broker. An in-process emitter works with one process. With two instances it silently drops half the events, because the publish landed on the node that does not own the subscriber.
• Broker without a local table. Forwarding raw events through the load balancer breaks the moment a subscriber's socket lives on a different instance than the publish arrives on.
• Local table without cleanup. The failure that hides: sockets that closed without triggering teardown leave entries delivering into the void until memory alerts fire. Audit entry count against live sockets on a timer.

Draining a deploy

SIGTERM -> reject new subscribe messages
        -> send complete for each active operation
        -> wait for in-flight writes (bounded, about 5s)
        -> close broker subscriptions, then sockets

Clients that receive complete know the server ended the stream on purpose and reconnect with backoff. Clients that only see a TCP reset reconnect immediately, all at once, against a fleet that is still starting — the classic post-deploy thundering herd. Backoff with jitter on the client and a bounded drain on the server solve it from both ends.

Delivery guarantees per hop

The broker itself may be at-least-once (Redis pub/sub is actually at-most-once: it drops messages for consumers that disconnected). Know which one you bought: Kafka with consumer offsets gives you replay, Redis pub/sub gives you none, and a managed stream gives you whatever its retention window says. GraphQL's guarantee is the weakest link in that chain, never stronger.

Metrics worth alerting on

active_streams, events_published versus events_delivered (the gap is filtering, loss, or a dead instance), replay_gap_expired, slow_consumer_dropped, and reconnect rate after deploys. Publish-to-socket p99 latency tells you whether the broker or the consumers are the constraint.

Interview framing: state the guarantee (at-most-once), name the mechanism that upgrades it (client ack plus replay buffer), and identify the bounded queue as what protects the process from a slow consumer.