pi-model-auto-router: stalled stream (90s no events) is treated as fatal, should failover like transient errors #10

New issue

Closed

opened 2026-09-04 11:06:55 +08:00 by lengxf · 0 comments

lengxf commented

2026-09-04 11:06:55 +08:00

(Migrated from github.com)

Summary

When a target provider accepts the connection but stops sending SSE events for 90s, the stall watchdog treats it as a fatal error: it logs fatal, pushes the error to the caller, and aborts. It never attempts failover to the other targets in the route, even though a stalled upstream is transient in nature (same root cause as 503 BAD_UPSTREAM timeouts).

Environment

pi-model-auto-router 0.3.1

Reproduction

Route config with multiple targets (e.g. cache-first):

"kimi": {
  "strategy": "cache-first",
  "targets": [
    { "provider": "llm-gw", "model": "kimi-k3-local-joybuilder", "weight": 1 },
    { "provider": "llm-gw-gpt", "model": "kimi-k3-local-joybuilder", "weight": 1 },
    { "provider": "llm-gw", "model": "kimi-k3-joybuilder", "weight": 1 }
  ]
}

Upstream hangs after accepting the request (no SSE events at all).
After 90s the watchdog fires:

[model-auto-router] llm-gw/kimi-k3-local-joybuilder sent no events for 1m30s, treating as stalled

Router log shows event: fatal, the request fails — no failover to the other targets.

In the same session a minute later, the same target returned 503 {"cause":"Post \"http://kimi-k3.jd.com/v1/chat/completions\": net/http: timeout awaiting response headers"} — that one was classified transient and correctly failed over to the next target.

Observed code path (dist/index.js, v0.3.1)

In the watchdog callback (~line 800):

watchdog = setInterval(() => {
    if (deps.now() - lastActivityAt > stallTimeoutMs()) {
        const message = `[model-auto-router] ${key} sent no events for ${formatDuration(stallTimeoutMs())}, treating as stalled`;
        ended.done = true;          // stops the outer loop
        stopWatchdog();
        ...
        logEvent({ event: "fatal", route: routeId, target: key, error: message });
        pushError(outer, model, message);   // error surfaces to the user
        ...
        finishRunSummary(routeId, "failed", failovers);
        void iterator?.return?.().catch(() => { });
    }
}, stallCheckMs());

Note the error event path (if (event.type === "error" && !committed)) goes through classifyFailure() and the failover logic, but the stall path bypasses it entirely and is hardcoded to fatal.

Expected behavior

A stalled stream should be treated like a transient failure (upstream hang / timeout):

Classify it as transient (or make it configurable, e.g. stallClass: "transient" | "fatal").
Trigger the normal failover to the next target in the route, and the retry/backoff/cooldown machinery if all targets stall.
Apply the transient cooldown to the stalled target.

Why it matters

With a single hung upstream, the whole request fails even when healthy backups are configured — the failover list is effectively useless against the "connected but silent" failure mode, which is a very common one for self-hosted/LLM-gateway backends.

# Summary When a target provider accepts the connection but stops sending SSE events for 90s, the stall watchdog treats it as a **fatal** error: it logs `fatal`, pushes the error to the caller, and aborts. It never attempts failover to the other targets in the route, even though a stalled upstream is transient in nature (same root cause as 503 BAD_UPSTREAM timeouts). ## Environment - pi-model-auto-router 0.3.1 ## Reproduction Route config with multiple targets (e.g. `cache-first`): ```json "kimi": { "strategy": "cache-first", "targets": [ { "provider": "llm-gw", "model": "kimi-k3-local-joybuilder", "weight": 1 }, { "provider": "llm-gw-gpt", "model": "kimi-k3-local-joybuilder", "weight": 1 }, { "provider": "llm-gw", "model": "kimi-k3-joybuilder", "weight": 1 } ] } ``` 1. Upstream hangs after accepting the request (no SSE events at all). 2. After 90s the watchdog fires: ``` [model-auto-router] llm-gw/kimi-k3-local-joybuilder sent no events for 1m30s, treating as stalled ``` 3. Router log shows `event: fatal`, the request fails — **no failover to the other targets**. In the same session a minute later, the same target returned `503 {"cause":"Post \"http://kimi-k3.jd.com/v1/chat/completions\": net/http: timeout awaiting response headers"}` — that one was classified transient and correctly failed over to the next target. ## Observed code path (dist/index.js, v0.3.1) In the watchdog callback (~line 800): ```js watchdog = setInterval(() => { if (deps.now() - lastActivityAt > stallTimeoutMs()) { const message = `[model-auto-router] ${key} sent no events for ${formatDuration(stallTimeoutMs())}, treating as stalled`; ended.done = true; // stops the outer loop stopWatchdog(); ... logEvent({ event: "fatal", route: routeId, target: key, error: message }); pushError(outer, model, message); // error surfaces to the user ... finishRunSummary(routeId, "failed", failovers); void iterator?.return?.().catch(() => { }); } }, stallCheckMs()); ``` Note the error event path (`if (event.type === "error" && !committed)`) goes through `classifyFailure()` and the failover logic, but the stall path bypasses it entirely and is hardcoded to fatal. ## Expected behavior A stalled stream should be treated like a transient failure (upstream hang / timeout): 1. Classify it as **transient** (or make it configurable, e.g. `stallClass: "transient" | "fatal"`). 2. Trigger the normal failover to the next target in the route, and the retry/backoff/cooldown machinery if all targets stall. 3. Apply the transient cooldown to the stalled target. ## Why it matters With a single hung upstream, the whole request fails even when healthy backups are configured — the failover list is effectively useless against the "connected but silent" failure mode, which is a very common one for self-hosted/LLM-gateway backends.

👀 1