The hardest part of my RAG bot was handing control to a human

I added a RAG assistant to the support chat of a creator platform I worked on. Users already had a chat with the support team. Staff answered from an admin panel, and conversations that reached a person were mirrored into Discord threads. The bot's job was to answer first from the help articles and step aside when it could not help.

I expected the work to be about retrieval and prompts. The part that took the most thought was a smaller question: at any given moment, who owns the next reply?

An answer could finish after ownership changed

A bot turn was not instant. It searched the knowledge base, called a model and validated the structured response. While that ran, the conversation kept moving. The user could send another message or ask for a person. A staff member could open the session and start typing.

If the bot checked ownership only when a turn started, it could finish after a human had taken over and post into a conversation it no longer owned. The user would see a bot answer under the line saying they were being connected to the team. As I found later, staff might never see that answer at all.

There was also a zero-budget constraint. Provider quota could run out mid-conversation. Retrieval could miss. The model could return output that failed validation. Each of those needed somewhere to go.

Ownership lived on the session

Each support session had a mode field: bot or human. I kept it in the database rather than in memory or in the client, because it described the conversation, not a particular request.

Every path to a human went through one function. The bot escalating on its own, the user asking for a person, a staff member pressing take over and a staff member simply replying all called the same escalation. A staff reply counted as a silent takeover: it skipped the "connecting you" message, because the reply itself was the human response.

The transition was a compare-and-set. This is simplified from the implementation:

async function escalateSession(sessionId, reason, { silent = false } = {}) {
  // Only the first caller matches. Everyone after it finds mode !== "bot".
  const session = await SupportSession.findOneAndUpdate(
    { _id: sessionId, mode: "bot" },
    { $set: { mode: "human", escalatedAt: new Date(), escalationReason: reason } },
    { new: true }
  );
  if (!session) return; // already human

  if (!silent) await postHandoffMessage(session);

  // Mark the whole transcript as unsent, then relay it to a new staff thread.
  await SupportMessage.updateMany(
    { _session_id: session._id },
    { $set: { discordRelayedAt: null } }
  );
  relayPendingToDiscord(session._id).catch(() => {});
}

If the bot and a staff member escalated at the same moment, one update matched and the other changed nothing. The handoff message could not appear twice, and the first escalation reason was the one kept. Nothing switched a session back to bot mode, so once a person owned a conversation the model never ran for it again.

Every failure had a destination

Inside a turn, each way of failing ended in the same place:

try {
  const { articles, kbMiss } = await retrieve(question);
  if (kbMiss) return escalateSession(sessionId, "low_confidence");

  const result = await generate({ articles, history, question });
  if (!result.canAnswer || result.escalate || result.sensitive || !result.reply) {
    return escalateSession(sessionId, reasonFor(result));
  }
  await postBotMessage(session, result.reply);
} catch (err) {
  const exhausted = err?.code === "ALL_EXHAUSTED" || err?.code === "EMBED_QUOTA";
  return escalateSession(sessionId, exhausted ? "provider_exhausted" : "error");
}

A spent quota, a retrieval miss and an unparseable response all led to the experience users had before the bot existed: a person picked up the conversation. The reason was stored on the session, so "the knowledge base had nothing" stayed distinguishable from "a provider ran out".

The alternative I rejected was opening a staff thread for every conversation from the start. Staff would have had context earlier, but their queue would have filled with conversations the bot was handling. Instead, the thread was created at escalation. While in bot mode, bot replies were marked as already relayed so they never reached Discord. Escalation cleared that mark across the session and replayed the full transcript in order.

The cost was that handoff did more work. Creating a thread and relaying a transcript were now part of the escalation path, and each could fail on its own.

Turns were serialized inside the process

Two things could overlap. A user sending two messages quickly could start two turns. A takeover could arrive while a turn was waiting on the model.

Both went through a per-session promise chain:

const botChains = new Map();

function runSerialized(sessionId, fn) {
  const key = String(sessionId);
  const tail = (botChains.get(key) || Promise.resolve())
    .then(() => fn())
    .catch((err) => console.error("[support-bot] turn failed:", err?.message || err));
  botChains.set(key, tail);
  tail.finally(() => {
    if (botChains.get(key) === tail) botChains.delete(key);
  });
  return tail;
}

A takeover that arrived mid-turn waited its place in line. The in-flight answer posted first, then the session flipped to human, and staff received the transcript in the same order the user had seen it.

Inside one process, a takeover waited behind the in-flight bot turn.

I chose to let the in-flight answer finish rather than cancel it. It was written for the user's latest question and was often useful. The cost was that a takeover could wait for the current turn. Every provider request had a 20-second timeout, so a hung call delayed the queue rather than blocking it indefinitely.

Rapid messages queued several turns. Each turn checked whether a bot or staff message already followed the latest user message, and skipped itself if so. One answer covered the burst instead of three answers arriving in a row.

Where the guarantee stopped

Before posting, the bot claimed the session with a conditional update and only then inserted its message:

const claim = await SupportSession.updateOne(
  { _id: session._id, mode: "bot", status: "open" },
  { $set: { lastMessageAt: new Date() }, $inc: { unreadForUser: 1 } }
);
if (!claim.matchedCount) return; // escalated or closed mid-turn: drop the reply

await SupportMessage.create({
  _session_id: session._id,
  sender: "bot",
  body,
  discordRelayedAt: new Date(), // bot-mode replies never go to staff
});

Inside one process, the queue already kept a takeover from landing between those two writes. The guard was for what the queue could not see. The chain was a Map in one Node process. A second server instance would have its own map, and nothing coordinated the two.

That left a narrow gap. If the guard passed on one instance and an escalation on another instance cleared the relay marks before the insert, the bot reply would land after the handoff, already marked as relayed. The user would see it. Staff would not.

Across two instances, nothing ordered the claim and the insert.

Two separate writes are not one operation, and an in-memory queue is not a distributed lock. The design held for a single instance and had a known gap beyond it.

The tradeoffs

What I'd do differently

I would decide the multi-instance guarantee before running more than one instance. The fix is to make "still owned by the bot" and "message published" a single durable step. That could be a transaction across the session and message writes, a version number on the session checked at insert, or routing every turn for a session to one worker. Then I would test it with two workers racing on the same session.

I would also track the whole handoff, not just the escalation event: why it happened, whether the transcript reached staff and how long it took a person to reply. I recorded that an escalation happened, which is not the same as knowing someone helped.

Building this changed how I think about RAG support. Retrieval and generation decide how good the answers are. The ownership rules decide what happens when there is no good answer, and I would want that path clear before making the model more ambitious.