fix(gateway): 熔断计入持续 4xx + 去 assert + Protocol/transient 去重

P1-2 非可重试错误(持续 401/403)也 record_failure,坏 key 可触发熔断。
P1-9 _complete_structured 的 assert 改显式 raise ValueError(-O 安全)。
P2 GatewayRun 抽到 orchestrator/_protocols.py 单点(去 4 处重复);
  _is_transient 抽到 adapters/base.py is_transient_by_name(去 3 处重复);
  Gemini Protocol 改 async def;gateway._retrying 去无用 async。
This commit is contained in:
Yaojia Wang
2026-06-21 19:32:49 +02:00
parent f7004e8d74
commit 016509c5c6
15 changed files with 178 additions and 65 deletions

View File

@@ -46,6 +46,7 @@
- [2026-06-18] @llm **三审并行图测试按 `req.output_schema` 路由 parsed**`SchemaRoutingRunGateway`)——单 `FakeRunGateway` 对所有审返同一 parsed 会让三审拿错 schema。验失败隔离某 schema 不登记→网关抛 KeyError→`run_review` 隔离为 `incomplete`,无需改 gateway。 - [2026-06-18] @llm **三审并行图测试按 `req.output_schema` 路由 parsed**`SchemaRoutingRunGateway`)——单 `FakeRunGateway` 对所有审返同一 parsed 会让三审拿错 schema。验失败隔离某 schema 不登记→网关抛 KeyError→`run_review` 隔离为 `incomplete`,无需改 gateway。
- [2026-06-18] @backend **伏笔 `record_progress` append JSONB 必须新建 list 重赋值**`row.progress = [*old, entry]`),不可原地 `.append()`——SQLAlchemy 默认不侦测可变 JSONB 原地突变原地改不脏标记→flush 丢失。`scan_overdue` 仅在有变更时 flush空扫描零写。状态机`transition` 同态(current==to)幂等放行、CLOSED 为终态(离开 CLOSED 全非法抛 `InvalidTransition`)`is_overdue` 严格大于(current==expected_close_to 仍在窗口内不逾期)、无 expected_close_to 永不逾期。 - [2026-06-18] @backend **伏笔 `record_progress` append JSONB 必须新建 list 重赋值**`row.progress = [*old, entry]`),不可原地 `.append()`——SQLAlchemy 默认不侦测可变 JSONB 原地突变原地改不脏标记→flush 丢失。`scan_overdue` 仅在有变更时 flush空扫描零写。状态机`transition` 同态(current==to)幂等放行、CLOSED 为终态(离开 CLOSED 全非法抛 `InvalidTransition`)`is_overdue` 严格大于(current==expected_close_to 仍在窗口内不逾期)、无 expected_close_to 永不逾期。
- [2026-06-18] @llm **orchestrator 内每模块各自声明 `GatewayRun` Protocol**`review_node.py``outline_node.py` 各一份,按模块最小依赖)——**不跨模块复用、不在 orchestrator `__init__` 重复导出**`__init__` 只导出 review_node 那个,避免 re-export 名冲突outline 的为模块内部用。 - [2026-06-18] @llm **orchestrator 内每模块各自声明 `GatewayRun` Protocol**`review_node.py``outline_node.py` 各一份,按模块最小依赖)——**不跨模块复用、不在 orchestrator `__init__` 重复导出**`__init__` 只导出 review_node 那个,避免 re-export 名冲突outline 的为模块内部用。
- [2026-06-21] @llm **【撤销上条】`GatewayRun` 已收敛为单点定义**`orchestrator/_protocols.py`,仅依赖 `ww_llm_gateway.types` 无环)→ review/generation/outline/style_extract 4 节点 + graph/`__init__` 统一 `from ._protocols import GatewayRun`,删除 4 处重复声明CODE_REVIEW P2 DRY。上条「每模块各自声明、不复用」的理由怕环/re-export 冲突)不成立——单点 Protocol 无环、`__init__` 单点导出无冲突。
- [2026-06-18] @qa **M2 E2E 多档位假适配器**`config.tier_defaults` writer/analyst/light 默认同 provider(deepseek)→单个假适配器(`provider="deepseek"`)即覆盖三档位;据 `req.output_schema is ContinuityReview`(续审)/否则 digest facts schema 分支返回 `parsed`;三档位用不同 `input_tokens` 区分以断言各自落 `usage_ledger`。三端点记账闭环 = review 端点流末 commit + accept 验收事务末 commit 都把网关 ledger flush 真正提交M1 ledger bug 在 M2 无复发)。 - [2026-06-18] @qa **M2 E2E 多档位假适配器**`config.tier_defaults` writer/analyst/light 默认同 provider(deepseek)→单个假适配器(`provider="deepseek"`)即覆盖三档位;据 `req.output_schema is ContinuityReview`(续审)/否则 digest facts schema 分支返回 `parsed`;三档位用不同 `input_tokens` 区分以断言各自落 `usage_ledger`。三端点记账闭环 = review 端点流末 commit + accept 验收事务末 commit 都把网关 ledger flush 真正提交M1 ledger bug 在 M2 无复发)。
- [2026-06-18] @qa **E2E 验证「digest 从终稿非草稿」(#4) 手法**final_text 注入草稿没有的标记串,假 light 适配器把它放进 digest facts 的 `summary`,断言 `chapter_digests.facts["summary"]==标记``标记 not in draft_text`。accept 409 gate 经 ASGITransport 正常返回(`AppError` 不上抛),断言 `resp.json()["error"]["details"]["missing_conflict_indices"]``ErrorCode` StrEnum → `"CONFLICT_UNRESOLVED"`)。 - [2026-06-18] @qa **E2E 验证「digest 从终稿非草稿」(#4) 手法**final_text 注入草稿没有的标记串,假 light 适配器把它放进 digest facts 的 `summary`,断言 `chapter_digests.facts["summary"]==标记``标记 not in draft_text`。accept 409 gate 经 ASGITransport 正常返回(`AppError` 不上抛),断言 `resp.json()["error"]["details"]["missing_conflict_indices"]``ErrorCode` StrEnum → `"CONFLICT_UNRESOLVED"`)。
- [2026-06-18] @frontend **审稿历史 `conflicts` 在 OpenAPI 被标松散 `{[k]:unknown}[]`**(后端用 dict/JSONB 列)→ 前端 `lib/review/history.ts` 安全收窄成 `ReviewConflict{type,where,refs,suggestion}`,缺字段给默认、**保序**(顺序=冲突 gate 的 `conflict_index` 身份,不可重排,否则裁决错位)。 - [2026-06-18] @frontend **审稿历史 `conflicts` 在 OpenAPI 被标松散 `{[k]:unknown}[]`**(后端用 dict/JSONB 列)→ 前端 `lib/review/history.ts` 安全收窄成 `ReviewConflict{type,where,refs,suggestion}`,缺字段给默认、**保序**(顺序=冲突 gate 的 `conflict_index` 身份,不可重排,否则裁决错位)。

View File

@@ -8,6 +8,7 @@
from __future__ import annotations from __future__ import annotations
from ._protocols import GatewayRun
from .collect import ( from .collect import (
CONTINUITY, CONTINUITY,
FORESHADOW, FORESHADOW,
@@ -43,7 +44,6 @@ from .review_node import (
REVIEW_INCOMPLETE, REVIEW_INCOMPLETE,
REVIEW_OK, REVIEW_OK,
BoundReviewNode, BoundReviewNode,
GatewayRun,
build_review_context, build_review_context,
build_review_request, build_review_request,
make_review_node, make_review_node,

View File

@@ -0,0 +1,17 @@
"""编排器节点对网关的最小依赖 Protocol单点定义DRY
仅依赖 `ww_llm_gateway.types`(无环)→ 各节点review/generation/outline/style_extract
共用此单一定义,不再每模块各自重复声明(见 memory/gotchas.md 更新)。
"""
from __future__ import annotations
from typing import Protocol
from ww_llm_gateway.types import LlmRequest, LlmResponse
class GatewayRun(Protocol):
"""编排器节点对网关的最小依赖——只需 `run`(注入真网关或 mock"""
async def run(self, req: LlmRequest) -> LlmResponse: ...

View File

@@ -25,7 +25,6 @@ from __future__ import annotations
import uuid import uuid
from collections.abc import Sequence from collections.abc import Sequence
from typing import Protocol
import structlog import structlog
from pydantic import BaseModel from pydantic import BaseModel
@@ -37,21 +36,13 @@ from ww_agents import (
ContinuityReview, ContinuityReview,
WorldGenResult, WorldGenResult,
) )
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope from ww_llm_gateway.types import Block, LlmRequest, Scope
from ._protocols import GatewayRun
log = structlog.get_logger(__name__) log = structlog.get_logger(__name__)
class GatewayRun(Protocol):
"""生成节点对网关的最小依赖——只需 `run`(注入真网关或 mock
模块内部用(每模块各自声明,不跨模块复用、不在 orchestrator `__init__` 重复导出,
见 gotchas 2026-06-18
"""
async def run(self, req: LlmRequest) -> LlmResponse: ...
def _build_request( def _build_request(
spec: AgentSpec, spec: AgentSpec,
*, *,

View File

@@ -23,8 +23,9 @@ from ww_agents import (
style_drift_spec, style_drift_spec,
) )
from ._protocols import GatewayRun
from .collect import ReviewRecorder, collect_reviews from .collect import ReviewRecorder, collect_reviews
from .review_node import GatewayRun, run_review from .review_node import run_review
from .state import ChapterState from .state import ChapterState
from .write_node import GatewayStream, write_node from .write_node import GatewayStream, write_node

View File

@@ -16,21 +16,16 @@
from __future__ import annotations from __future__ import annotations
import uuid import uuid
from typing import Protocol
import structlog import structlog
from ww_agents import AgentSpec, OutlineResult from ww_agents import AgentSpec, OutlineResult
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope from ww_llm_gateway.types import Block, LlmRequest, Scope
from ._protocols import GatewayRun
log = structlog.get_logger(__name__) log = structlog.get_logger(__name__)
class GatewayRun(Protocol):
"""大纲节点对网关的最小依赖——只需 `run`(注入真网关或 mock"""
async def run(self, req: LlmRequest) -> LlmResponse: ...
def build_outline_request( def build_outline_request(
spec: AgentSpec, spec: AgentSpec,
*, *,

View File

@@ -19,12 +19,13 @@ M2 先接 **continuity**C6 `continuity_spec`),设计成可扩——图按
from __future__ import annotations from __future__ import annotations
from collections.abc import Awaitable, Callable from collections.abc import Awaitable, Callable
from typing import Any, Protocol from typing import Any
import structlog import structlog
from ww_agents import AgentSpec from ww_agents import AgentSpec
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope from ww_llm_gateway.types import Block, LlmRequest, Scope
from ._protocols import GatewayRun
from .state import ChapterState from .state import ChapterState
log = structlog.get_logger(__name__) log = structlog.get_logger(__name__)
@@ -36,11 +37,16 @@ REVIEW_INCOMPLETE = "incomplete" # 该审网关失败 → 标未完成,不阻
# 已绑定 gateway 的节点形(图工厂 / T2.5 直接跑审稿时用)。 # 已绑定 gateway 的节点形(图工厂 / T2.5 直接跑审稿时用)。
BoundReviewNode = Callable[[ChapterState], Awaitable[dict[str, Any]]] BoundReviewNode = Callable[[ChapterState], Awaitable[dict[str, Any]]]
__all__ = [
class GatewayRun(Protocol): "REVIEW_INCOMPLETE",
"""审稿节点对网关的最小依赖——只需 `run`(注入真网关或 mock""" "REVIEW_OK",
"BoundReviewNode",
async def run(self, req: LlmRequest) -> LlmResponse: ... "GatewayRun",
"build_review_context",
"build_review_request",
"make_review_node",
"run_review",
]
def build_review_context(*, draft: str, stable_core: str, volatile: str) -> str: def build_review_context(*, draft: str, stable_core: str, volatile: str) -> str:

View File

@@ -17,25 +17,16 @@
from __future__ import annotations from __future__ import annotations
import uuid import uuid
from typing import Protocol
import structlog import structlog
from ww_agents import AgentSpec, StyleFingerprintResult from ww_agents import AgentSpec, StyleFingerprintResult
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope from ww_llm_gateway.types import Block, LlmRequest, Scope
from ._protocols import GatewayRun
log = structlog.get_logger(__name__) log = structlog.get_logger(__name__)
class GatewayRun(Protocol):
"""文风提取节点对网关的最小依赖——只需 `run`(注入真网关或 mock
模块内部用(每模块各自声明,不跨模块复用、不在 orchestrator `__init__` 重复导出,
见 gotchas 2026-06-18
"""
async def run(self, req: LlmRequest) -> LlmResponse: ...
def build_style_extract_request( def build_style_extract_request(
spec: AgentSpec, spec: AgentSpec,
*, *,

View File

@@ -108,6 +108,22 @@ def transient(msg: str = "boom") -> TransientProviderError:
return TransientProviderError(msg) return TransientProviderError(msg)
class AuthError(Exception):
"""模拟持续性鉴权失败(坏 key / 账号被禁):带 `status_code`**非**瞬时不可重试。
适配器对 401/403 不翻译为 `TransientProviderError`,原样上抛;网关须对其计入熔断
P1-2而非每次白打同一坏 provider。
"""
def __init__(self, msg: str = "unauthorized", *, status_code: int = 401) -> None:
super().__init__(msg)
self.status_code = status_code
def auth_error(status_code: int = 401) -> AuthError:
return AuthError(status_code=status_code)
def chain(*routes: tuple[str, str]) -> list[Route]: def chain(*routes: tuple[str, str]) -> list[Route]:
return [Route(provider=p, model=m) for p, m in routes] return [Route(provider=p, model=m) for p, m in routes]

View File

@@ -8,7 +8,15 @@
from __future__ import annotations from __future__ import annotations
import pytest import pytest
from fakes_resilience import FakeLedger, ScriptedAdapter, chain, chain_resolver, transient from fakes_resilience import (
AuthError,
FakeLedger,
ScriptedAdapter,
auth_error,
chain,
chain_resolver,
transient,
)
from ww_llm_gateway.gateway import CircuitBreaker, Gateway from ww_llm_gateway.gateway import CircuitBreaker, Gateway
from ww_llm_gateway.types import LlmRequest from ww_llm_gateway.types import LlmRequest
from ww_shared import AppError, ErrorCode from ww_shared import AppError, ErrorCode
@@ -154,6 +162,39 @@ def test_circuit_breaker_reopens_after_cooldown() -> None:
assert cb.is_open("deepseek") is False assert cb.is_open("deepseek") is False
async def test_persistent_auth_error_counts_toward_breaker(req: LlmRequest) -> None:
"""P1-2持续性 401坏 key虽不可重试但应计入熔断——连续 N 次后熔断打开。
每次 `run` 命中 401 立即上抛(不重试、不回退),但 raise 前 `record_failure`
达到阈值后熔断打开,后续请求直接跳过该 provider → 链耗尽抛 LLM_UNAVAILABLE。
"""
threshold = 3
primary = ScriptedAdapter("deepseek", failures=[auth_error() for _ in range(threshold + 2)])
cb = CircuitBreaker(threshold=threshold, reset_seconds=60.0)
gw = Gateway(
{"deepseek": primary},
FakeLedger(),
chain_resolver=chain_resolver(chain(("deepseek", "deepseek-chat"))),
breaker=cb,
)
# 前 threshold 次401 原样上抛(不可重试),但每次计一次熔断失败。
for _ in range(threshold):
assert cb.is_open("deepseek") is False
with pytest.raises(AuthError):
await gw.run(req)
# 第 threshold 次失败后熔断打开。
assert cb.is_open("deepseek") is True
# 后续请求provider 被熔断跳过 → 不再调 complete链耗尽 → LLM_UNAVAILABLE。
calls_before = primary.complete_calls
with pytest.raises(AppError) as exc:
await gw.run(req)
assert exc.value.code == ErrorCode.LLM_UNAVAILABLE
assert primary.complete_calls == calls_before # 熔断后未再触达坏 provider
async def test_open_circuit_skips_provider(req: LlmRequest) -> None: async def test_open_circuit_skips_provider(req: LlmRequest) -> None:
# 熔断已打开的主 provider 被直接跳过,连 complete 都不调,直接走回退。 # 熔断已打开的主 provider 被直接跳过,连 complete 都不调,直接走回退。
primary = ScriptedAdapter("deepseek", text="不该被调") primary = ScriptedAdapter("deepseek", text="不该被调")

View File

@@ -16,7 +16,13 @@ from pydantic import BaseModel
from ..errors import TransientProviderError from ..errors import TransientProviderError
from ..types import LlmRequest from ..types import LlmRequest
from .base import Capabilities, ProviderResult, ProviderUsage, StreamChunk from .base import (
Capabilities,
ProviderResult,
ProviderUsage,
StreamChunk,
is_transient_by_name,
)
if TYPE_CHECKING: if TYPE_CHECKING:
from anthropic import AsyncAnthropic from anthropic import AsyncAnthropic
@@ -44,11 +50,7 @@ class StructuredAnthropic(Protocol):
def _is_transient(exc: Exception) -> bool: def _is_transient(exc: Exception) -> bool:
name = type(exc).__name__ return is_transient_by_name(exc, _TRANSIENT_NAMES)
if name in _TRANSIENT_NAMES:
return True
status = getattr(exc, "status_code", None)
return isinstance(status, int) and (status == 429 or status >= 500)
def _system_blocks(req: LlmRequest) -> list[dict[str, Any]]: def _system_blocks(req: LlmRequest) -> list[dict[str, Any]]:
@@ -144,7 +146,8 @@ class AnthropicAdapter:
return ProviderResult(text=_text_from(resp), usage=usage) return ProviderResult(text=_text_from(resp), usage=usage)
async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult: async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult:
assert req.output_schema is not None if req.output_schema is None:
raise ValueError("_complete_structured called without output_schema")
kwargs: dict[str, Any] = { kwargs: dict[str, Any] = {
"model": model, "model": model,
"max_tokens": req.max_tokens or _DEFAULT_MAX_TOKENS, "max_tokens": req.max_tokens or _DEFAULT_MAX_TOKENS,

View File

@@ -12,6 +12,34 @@ from pydantic import BaseModel, ConfigDict
from ..types import LlmRequest from ..types import LlmRequest
# 瞬时(可退避重试)的 HTTP 状态码判定429 限流 或 5xx 服务端错误。
_RATE_LIMITED_STATUS = 429
_SERVER_ERROR_MIN_STATUS = 500
def _is_transient_status(status: object) -> bool:
return isinstance(status, int) and (
status == _RATE_LIMITED_STATUS or status >= _SERVER_ERROR_MIN_STATUS
)
def is_transient_by_name(
exc: Exception, names: frozenset[str], *, extra_codes: bool = False
) -> bool:
"""统一的瞬时错误判定DRY三个适配器共用
① 异常类名命中 `names`(按名匹配,避免硬依赖各厂商 SDK 异常类型);或
② `status_code` 属性是 429/5xx
③ `extra_codes=True` 时额外检查 `code` 属性是否为 429/5xxGemini 用 `exc.code`)。
"""
if type(exc).__name__ in names:
return True
if _is_transient_status(getattr(exc, "status_code", None)):
return True
if extra_codes and _is_transient_status(getattr(exc, "code", None)):
return True
return False
class Capabilities(BaseModel): class Capabilities(BaseModel):
structured_output: bool = False structured_output: bool = False

View File

@@ -14,7 +14,13 @@ from typing import Any, Protocol
from ..errors import TransientProviderError from ..errors import TransientProviderError
from ..types import LlmRequest from ..types import LlmRequest
from .base import Capabilities, ProviderResult, ProviderUsage, StreamChunk from .base import (
Capabilities,
ProviderResult,
ProviderUsage,
StreamChunk,
is_transient_by_name,
)
_TRANSIENT_NAMES = frozenset( _TRANSIENT_NAMES = frozenset(
{ {
@@ -31,7 +37,7 @@ _TRANSIENT_NAMES = frozenset(
class GeminiModels(Protocol): class GeminiModels(Protocol):
async def generate_content(self, **kwargs: Any) -> Any: ... async def generate_content(self, **kwargs: Any) -> Any: ...
def generate_content_stream(self, **kwargs: Any) -> Any: ... async def generate_content_stream(self, **kwargs: Any) -> Any: ...
class GeminiAio(Protocol): class GeminiAio(Protocol):
@@ -45,11 +51,8 @@ class GeminiClient(Protocol):
def _is_transient(exc: Exception) -> bool: def _is_transient(exc: Exception) -> bool:
name = type(exc).__name__ # Gemini SDK 部分错误用 `exc.code` 携带状态码 → extra_codes=True 一并检查。
if name in _TRANSIENT_NAMES: return is_transient_by_name(exc, _TRANSIENT_NAMES, extra_codes=True)
return True
status = getattr(exc, "code", None) or getattr(exc, "status_code", None)
return isinstance(status, int) and (status == 429 or status >= 500)
def _contents(req: LlmRequest) -> str: def _contents(req: LlmRequest) -> str:

View File

@@ -15,7 +15,13 @@ from pydantic import BaseModel
from ..errors import TransientProviderError from ..errors import TransientProviderError
from ..types import LlmRequest from ..types import LlmRequest
from .base import Capabilities, ProviderResult, ProviderUsage, StreamChunk from .base import (
Capabilities,
ProviderResult,
ProviderUsage,
StreamChunk,
is_transient_by_name,
)
# OpenAI 兼容 SDK 的瞬时错误类名(按名匹配,覆盖 DeepSeek/Kimi/Qwen/GLM 等共用 SDK # OpenAI 兼容 SDK 的瞬时错误类名(按名匹配,覆盖 DeepSeek/Kimi/Qwen/GLM 等共用 SDK
_TRANSIENT_NAMES = frozenset( _TRANSIENT_NAMES = frozenset(
@@ -30,11 +36,7 @@ _TRANSIENT_NAMES = frozenset(
def _is_transient(exc: Exception) -> bool: def _is_transient(exc: Exception) -> bool:
name = type(exc).__name__ return is_transient_by_name(exc, _TRANSIENT_NAMES)
if name in _TRANSIENT_NAMES:
return True
status = getattr(exc, "status_code", None)
return isinstance(status, int) and (status == 429 or status >= 500)
class StructuredClient(Protocol): class StructuredClient(Protocol):
@@ -126,7 +128,8 @@ class OpenAICompatAdapter:
return ProviderResult(text=text, usage=_usage_from(resp.usage)) return ProviderResult(text=text, usage=_usage_from(resp.usage))
async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult: async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult:
assert req.output_schema is not None if req.output_schema is None:
raise ValueError("_complete_structured called without output_schema")
parsed, raw = await self._structured().create_with_completion( parsed, raw = await self._structured().create_with_completion(
messages=_messages(req), messages=_messages(req),
response_model=req.output_schema, response_model=req.output_schema,

View File

@@ -52,6 +52,19 @@ def _is_retryable(exc: BaseException) -> bool:
return isinstance(exc, AppError) and exc.code == ErrorCode.RATE_LIMITED return isinstance(exc, AppError) and exc.code == ErrorCode.RATE_LIMITED
# 持续性鉴权错误状态码:坏 key / 被禁用 → 每次重打都失败应计入熔断P1-2
_PERSISTENT_AUTH_STATUSES = frozenset({401, 403})
def _is_persistent_auth_error(exc: BaseException) -> bool:
"""持续性鉴权失败401/403错误 key / 账号被禁,重打无意义 → 计入熔断。
按 `status_code` 属性识别(不硬依赖任何厂商 SDK 异常类型)。
"""
status = getattr(exc, "status_code", None)
return isinstance(status, int) and status in _PERSISTENT_AUTH_STATUSES
def _input_len(req: LlmRequest) -> int: def _input_len(req: LlmRequest) -> int:
if isinstance(req.input, str): if isinstance(req.input, str):
return len(req.input) return len(req.input)
@@ -187,7 +200,7 @@ class Gateway:
project_id=str(req.scope.project_id) if req.scope.project_id else None, project_id=str(req.scope.project_id) if req.scope.project_id else None,
) )
async def _retrying(self) -> AsyncRetrying: def _retrying(self) -> AsyncRetrying:
return AsyncRetrying( return AsyncRetrying(
stop=stop_after_attempt(self._max_retries + 1), stop=stop_after_attempt(self._max_retries + 1),
wait=wait_exponential(min=_RETRY_MIN_SECONDS, max=_RETRY_MAX_SECONDS), wait=wait_exponential(min=_RETRY_MIN_SECONDS, max=_RETRY_MAX_SECONDS),
@@ -212,6 +225,10 @@ class Gateway:
result = await self._complete_with_retry(adapter, req, route.model) result = await self._complete_with_retry(adapter, req, route.model)
except Exception as exc: # noqa: BLE001 — 链内逐 provider 兜底,最终统一上抛 except Exception as exc: # noqa: BLE001 — 链内逐 provider 兜底,最终统一上抛
if not _is_retryable(exc): if not _is_retryable(exc):
# 持续性鉴权失败401/403虽不可重试但坏 key 应触发熔断P1-2
# 否则每次都白打同一坏 provider。其它不可重试错误直接上抛。
if _is_persistent_auth_error(exc):
self._breaker.record_failure(route.provider)
raise raise
self._breaker.record_failure(route.provider) self._breaker.record_failure(route.provider)
last_error = exc last_error = exc
@@ -240,7 +257,7 @@ class Gateway:
async def _complete_with_retry( async def _complete_with_retry(
self, adapter: ProviderAdapter, req: LlmRequest, model: str self, adapter: ProviderAdapter, req: LlmRequest, model: str
) -> ProviderResult: ) -> ProviderResult:
retrying = await self._retrying() retrying = self._retrying()
async for attempt in retrying: async for attempt in retrying:
with attempt: with attempt:
return await adapter.complete(req, model) return await adapter.complete(req, model)