LLM Fallback Chain: Groq, Gemini, DeepSeek on OpenRouter

Why a single LLM vendor is a liability

Every hosted LLM is someone else's server you do not control. The failure modes are not theoretical:

• Rate limits appear the moment you ship to real traffic. The vendor you tuned against at 10 req/day throttles you at 10 req/min.
• Network blocks are silent and total. We run an AI auto-reply pipeline where Groq's egress got blocked by a WAF layer — every call returned 403 and the entire reply path died until we re-pointed it at Gemini. Nothing in our code changed; the vendor simply became unreachable.
• Moderation refusals cascade: one model's safety filter rejects a prompt your users consider benign, so the whole feature looks broken.
• Pricing or outage: a provider raises prices or drops a model you pinned, and your pinned model string 404s.

A fallback chain turns each of these from "product down" into "one slow request."

The three vendors, and what each is actually for

The order is deliberate: you want the fast/cheap one first to keep latency and cost down, and the boring-reliable one as the backstop. Do not lead with the most capable model — capability is useless if the request times out.

How OpenRouter fallback works (server-side)

OpenRouter is an OpenAI-compatible gateway in front of 500+ models. The key feature for resilience is the models array: pass an ordered list, and OpenRouter tries each in turn on any error (rate limit, downtime, moderation refusal, even context-length overflow). Your application sees one response.

// openrouter fallback — server-side, no retry code in your app
const res = await fetch('https://openrouter.ai/api/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}`,
    'Content-Type': 'application/json',
    'HTTP-Referer': 'https://wenboom.com',
    'X-Title': 'wenboom-pipeline',
  },
  body: JSON.stringify({
    // OpenRouter tries these in order; first success wins.
    models: [
      'groq/llama-3.3-70b-versatile',   // primary: fastest
      'google/gemini-2.5-flash',        // secondary: reliable
      'deepseek/deepseek-chat',        // tertiary: cheapest
    ],
    route: 'fallback',
    messages: [{ role: 'user', content: prompt }],
    max_tokens: 1024,
    temperature: 0.2,
  }),
});
const data = await res.json();
// OpenRouter reports the model that ACTUALLY served the request.
console.log('served by:', data.model);

Two things to internalize:

1. response.model is the truth. OpenRouter prices and returns the model that ultimately answered. Log it. Your dashboards should show "served by Gemini 12% of the time" — that is your failover health signal.
2. ~ pins the family, not a frozen slug. Prefix a model with ~ (e.g. ~groq/llama-3.3-70b-versatile) to let OpenRouter pick the latest minor revision. Slugs rotate; the catalog is the source of truth, so confirm the live slugs before shipping.

Model slugs change. The pattern above is what matters; verify groq/..., google/..., and deepseek/... identifiers against the current OpenRouter catalog before you deploy.

Client-side fallback chain (when you need control)

The models array is perfect when OpenRouter is your gateway. You need a client-side chain when: you call vendors directly (raw Groq / Gemini / DeepSeek APIs), you run BYOK keys per provider, you must match capabilities per attempt, or you want a hard wall-clock budget so one slow vendor never blocks a request forever.

// client-side fallback chain with per-attempt timeouts
const CHAIN = [
  { model: 'groq/llama-3.3-70b-versatile', timeout: 8000,  label: 'groq' },
  { model: 'google/gemini-2.5-flash',       timeout: 12000, label: 'gemini' },
  { model: 'deepseek/deepseek-chat',       timeout: 12000, label: 'deepseek' },
];

async function resilientChat(messages, max_tokens = 1024) {
  let lastErr;
  for (const hop of CHAIN) {
    try {
      const ctrl = new AbortController();
      const t = setTimeout(() => ctrl.abort(), hop.timeout);
      const r = await fetch('https://openrouter.ai/api/v1/chat/completions', {
        method: 'POST',
        signal: ctrl.signal,
        headers: {
          'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}`,
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ model: hop.model, messages, max_tokens, temperature: 0.2 }),
      });
      clearTimeout(t);
      if (!r.ok) { lastErr = new Error(`${hop.label} ${r.status}`); continue; }
      const data = await r.json();
      console.log(`served by ${hop.label} (${data.model})`);
      return data;
    } catch (e) {
      lastErr = e;
      console.warn(`${hop.label} failed: ${e.message}`);
      continue; // try the next vendor
    }
  }
  throw new Error(`All fallbacks exhausted: ${lastErr?.message}`);
}

The per-attempt AbortController timeout is the part people skip and then regret: without it, a vendor that accepts the connection but never streams a token hangs your request until the default 10-minute socket timeout. Set it to your real latency SLO (Groq under ~1s, Gemini/DeepSeek a few seconds).

Capability matching: not every link can do every job

A fallback chain only helps if the backup can actually do the task. Three mismatches bite:

• Tool calling / function calling. If your primary uses tools, every link must support them, or the fallback silently drops your tools. Either keep the whole chain tool-capable or degrade gracefully (retry without tools on the last resort).
• JSON / structured output. Same rule — if you parse response.choices[0].message.content as JSON, the backup must reliably emit JSON, or validate and fall back again.
• Context window. A request that overflows the primary's context will also overflow a smaller backup. Order by context size if long inputs are possible, not just by speed.

Failure modes a chain does NOT fix

A fallback is not a cure-all. Be honest about what it cannot save:

Vendor comparison at a glance