fix(gateway): 熔断计入持续 4xx + 去 assert + Protocol/transient 去重
P1-2 非可重试错误(持续 401/403)也 record_failure,坏 key 可触发熔断。 P1-9 _complete_structured 的 assert 改显式 raise ValueError(-O 安全)。 P2 GatewayRun 抽到 orchestrator/_protocols.py 单点(去 4 处重复); _is_transient 抽到 adapters/base.py is_transient_by_name(去 3 处重复); Gemini Protocol 改 async def;gateway._retrying 去无用 async。
This commit is contained in:
@@ -46,6 +46,7 @@
|
|||||||
- [2026-06-18] @llm **三审并行图测试按 `req.output_schema` 路由 parsed**(`SchemaRoutingRunGateway`)——单 `FakeRunGateway` 对所有审返同一 parsed 会让三审拿错 schema。验失败隔离:某 schema 不登记→网关抛 KeyError→`run_review` 隔离为 `incomplete`,无需改 gateway。
|
- [2026-06-18] @llm **三审并行图测试按 `req.output_schema` 路由 parsed**(`SchemaRoutingRunGateway`)——单 `FakeRunGateway` 对所有审返同一 parsed 会让三审拿错 schema。验失败隔离:某 schema 不登记→网关抛 KeyError→`run_review` 隔离为 `incomplete`,无需改 gateway。
|
||||||
- [2026-06-18] @backend **伏笔 `record_progress` append JSONB 必须新建 list 重赋值**(`row.progress = [*old, entry]`),不可原地 `.append()`——SQLAlchemy 默认不侦测可变 JSONB 原地突变,原地改不脏标记→flush 丢失。`scan_overdue` 仅在有变更时 flush(空扫描零写)。状态机:`transition` 同态(current==to)幂等放行、CLOSED 为终态(离开 CLOSED 全非法抛 `InvalidTransition`);`is_overdue` 严格大于(current==expected_close_to 仍在窗口内不逾期)、无 expected_close_to 永不逾期。
|
- [2026-06-18] @backend **伏笔 `record_progress` append JSONB 必须新建 list 重赋值**(`row.progress = [*old, entry]`),不可原地 `.append()`——SQLAlchemy 默认不侦测可变 JSONB 原地突变,原地改不脏标记→flush 丢失。`scan_overdue` 仅在有变更时 flush(空扫描零写)。状态机:`transition` 同态(current==to)幂等放行、CLOSED 为终态(离开 CLOSED 全非法抛 `InvalidTransition`);`is_overdue` 严格大于(current==expected_close_to 仍在窗口内不逾期)、无 expected_close_to 永不逾期。
|
||||||
- [2026-06-18] @llm **orchestrator 内每模块各自声明 `GatewayRun` Protocol**(`review_node.py` 与 `outline_node.py` 各一份,按模块最小依赖)——**不跨模块复用、不在 orchestrator `__init__` 重复导出**(`__init__` 只导出 review_node 那个,避免 re-export 名冲突);outline 的为模块内部用。
|
- [2026-06-18] @llm **orchestrator 内每模块各自声明 `GatewayRun` Protocol**(`review_node.py` 与 `outline_node.py` 各一份,按模块最小依赖)——**不跨模块复用、不在 orchestrator `__init__` 重复导出**(`__init__` 只导出 review_node 那个,避免 re-export 名冲突);outline 的为模块内部用。
|
||||||
|
- [2026-06-21] @llm **【撤销上条】`GatewayRun` 已收敛为单点定义**(`orchestrator/_protocols.py`,仅依赖 `ww_llm_gateway.types` 无环)→ review/generation/outline/style_extract 4 节点 + graph/`__init__` 统一 `from ._protocols import GatewayRun`,删除 4 处重复声明(CODE_REVIEW P2 DRY)。上条「每模块各自声明、不复用」的理由(怕环/re-export 冲突)不成立——单点 Protocol 无环、`__init__` 单点导出无冲突。
|
||||||
- [2026-06-18] @qa **M2 E2E 多档位假适配器**:`config.tier_defaults` writer/analyst/light 默认同 provider(deepseek)→单个假适配器(`provider="deepseek"`)即覆盖三档位;据 `req.output_schema is ContinuityReview`(续审)/否则 digest facts schema 分支返回 `parsed`;三档位用不同 `input_tokens` 区分以断言各自落 `usage_ledger`。三端点记账闭环 = review 端点流末 commit + accept 验收事务末 commit 都把网关 ledger flush 真正提交(M1 ledger bug 在 M2 无复发)。
|
- [2026-06-18] @qa **M2 E2E 多档位假适配器**:`config.tier_defaults` writer/analyst/light 默认同 provider(deepseek)→单个假适配器(`provider="deepseek"`)即覆盖三档位;据 `req.output_schema is ContinuityReview`(续审)/否则 digest facts schema 分支返回 `parsed`;三档位用不同 `input_tokens` 区分以断言各自落 `usage_ledger`。三端点记账闭环 = review 端点流末 commit + accept 验收事务末 commit 都把网关 ledger flush 真正提交(M1 ledger bug 在 M2 无复发)。
|
||||||
- [2026-06-18] @qa **E2E 验证「digest 从终稿非草稿」(#4) 手法**:final_text 注入草稿没有的标记串,假 light 适配器把它放进 digest facts 的 `summary`,断言 `chapter_digests.facts["summary"]==标记` 且 `标记 not in draft_text`。accept 409 gate 经 ASGITransport 正常返回(`AppError` 不上抛),断言 `resp.json()["error"]["details"]["missing_conflict_indices"]`(`ErrorCode` StrEnum → `"CONFLICT_UNRESOLVED"`)。
|
- [2026-06-18] @qa **E2E 验证「digest 从终稿非草稿」(#4) 手法**:final_text 注入草稿没有的标记串,假 light 适配器把它放进 digest facts 的 `summary`,断言 `chapter_digests.facts["summary"]==标记` 且 `标记 not in draft_text`。accept 409 gate 经 ASGITransport 正常返回(`AppError` 不上抛),断言 `resp.json()["error"]["details"]["missing_conflict_indices"]`(`ErrorCode` StrEnum → `"CONFLICT_UNRESOLVED"`)。
|
||||||
- [2026-06-18] @frontend **审稿历史 `conflicts` 在 OpenAPI 被标松散 `{[k]:unknown}[]`**(后端用 dict/JSONB 列)→ 前端 `lib/review/history.ts` 安全收窄成 `ReviewConflict{type,where,refs,suggestion}`,缺字段给默认、**保序**(顺序=冲突 gate 的 `conflict_index` 身份,不可重排,否则裁决错位)。
|
- [2026-06-18] @frontend **审稿历史 `conflicts` 在 OpenAPI 被标松散 `{[k]:unknown}[]`**(后端用 dict/JSONB 列)→ 前端 `lib/review/history.ts` 安全收窄成 `ReviewConflict{type,where,refs,suggestion}`,缺字段给默认、**保序**(顺序=冲突 gate 的 `conflict_index` 身份,不可重排,否则裁决错位)。
|
||||||
|
|||||||
@@ -8,6 +8,7 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from ._protocols import GatewayRun
|
||||||
from .collect import (
|
from .collect import (
|
||||||
CONTINUITY,
|
CONTINUITY,
|
||||||
FORESHADOW,
|
FORESHADOW,
|
||||||
@@ -43,7 +44,6 @@ from .review_node import (
|
|||||||
REVIEW_INCOMPLETE,
|
REVIEW_INCOMPLETE,
|
||||||
REVIEW_OK,
|
REVIEW_OK,
|
||||||
BoundReviewNode,
|
BoundReviewNode,
|
||||||
GatewayRun,
|
|
||||||
build_review_context,
|
build_review_context,
|
||||||
build_review_request,
|
build_review_request,
|
||||||
make_review_node,
|
make_review_node,
|
||||||
|
|||||||
17
packages/core/ww_core/orchestrator/_protocols.py
Normal file
17
packages/core/ww_core/orchestrator/_protocols.py
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
"""编排器节点对网关的最小依赖 Protocol(单点定义,DRY)。
|
||||||
|
|
||||||
|
仅依赖 `ww_llm_gateway.types`(无环)→ 各节点(review/generation/outline/style_extract)
|
||||||
|
共用此单一定义,不再每模块各自重复声明(见 memory/gotchas.md 更新)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from ww_llm_gateway.types import LlmRequest, LlmResponse
|
||||||
|
|
||||||
|
|
||||||
|
class GatewayRun(Protocol):
|
||||||
|
"""编排器节点对网关的最小依赖——只需 `run`(注入真网关或 mock)。"""
|
||||||
|
|
||||||
|
async def run(self, req: LlmRequest) -> LlmResponse: ...
|
||||||
@@ -25,7 +25,6 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from typing import Protocol
|
|
||||||
|
|
||||||
import structlog
|
import structlog
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
@@ -37,21 +36,13 @@ from ww_agents import (
|
|||||||
ContinuityReview,
|
ContinuityReview,
|
||||||
WorldGenResult,
|
WorldGenResult,
|
||||||
)
|
)
|
||||||
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope
|
from ww_llm_gateway.types import Block, LlmRequest, Scope
|
||||||
|
|
||||||
|
from ._protocols import GatewayRun
|
||||||
|
|
||||||
log = structlog.get_logger(__name__)
|
log = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class GatewayRun(Protocol):
|
|
||||||
"""生成节点对网关的最小依赖——只需 `run`(注入真网关或 mock)。
|
|
||||||
|
|
||||||
模块内部用(每模块各自声明,不跨模块复用、不在 orchestrator `__init__` 重复导出,
|
|
||||||
见 gotchas 2026-06-18)。
|
|
||||||
"""
|
|
||||||
|
|
||||||
async def run(self, req: LlmRequest) -> LlmResponse: ...
|
|
||||||
|
|
||||||
|
|
||||||
def _build_request(
|
def _build_request(
|
||||||
spec: AgentSpec,
|
spec: AgentSpec,
|
||||||
*,
|
*,
|
||||||
|
|||||||
@@ -23,8 +23,9 @@ from ww_agents import (
|
|||||||
style_drift_spec,
|
style_drift_spec,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
from ._protocols import GatewayRun
|
||||||
from .collect import ReviewRecorder, collect_reviews
|
from .collect import ReviewRecorder, collect_reviews
|
||||||
from .review_node import GatewayRun, run_review
|
from .review_node import run_review
|
||||||
from .state import ChapterState
|
from .state import ChapterState
|
||||||
from .write_node import GatewayStream, write_node
|
from .write_node import GatewayStream, write_node
|
||||||
|
|
||||||
|
|||||||
@@ -16,21 +16,16 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import uuid
|
import uuid
|
||||||
from typing import Protocol
|
|
||||||
|
|
||||||
import structlog
|
import structlog
|
||||||
from ww_agents import AgentSpec, OutlineResult
|
from ww_agents import AgentSpec, OutlineResult
|
||||||
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope
|
from ww_llm_gateway.types import Block, LlmRequest, Scope
|
||||||
|
|
||||||
|
from ._protocols import GatewayRun
|
||||||
|
|
||||||
log = structlog.get_logger(__name__)
|
log = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class GatewayRun(Protocol):
|
|
||||||
"""大纲节点对网关的最小依赖——只需 `run`(注入真网关或 mock)。"""
|
|
||||||
|
|
||||||
async def run(self, req: LlmRequest) -> LlmResponse: ...
|
|
||||||
|
|
||||||
|
|
||||||
def build_outline_request(
|
def build_outline_request(
|
||||||
spec: AgentSpec,
|
spec: AgentSpec,
|
||||||
*,
|
*,
|
||||||
|
|||||||
@@ -19,12 +19,13 @@ M2 先接 **continuity**(C6 `continuity_spec`),设计成可扩——图按
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from collections.abc import Awaitable, Callable
|
from collections.abc import Awaitable, Callable
|
||||||
from typing import Any, Protocol
|
from typing import Any
|
||||||
|
|
||||||
import structlog
|
import structlog
|
||||||
from ww_agents import AgentSpec
|
from ww_agents import AgentSpec
|
||||||
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope
|
from ww_llm_gateway.types import Block, LlmRequest, Scope
|
||||||
|
|
||||||
|
from ._protocols import GatewayRun
|
||||||
from .state import ChapterState
|
from .state import ChapterState
|
||||||
|
|
||||||
log = structlog.get_logger(__name__)
|
log = structlog.get_logger(__name__)
|
||||||
@@ -36,11 +37,16 @@ REVIEW_INCOMPLETE = "incomplete" # 该审网关失败 → 标未完成,不阻
|
|||||||
# 已绑定 gateway 的节点形(图工厂 / T2.5 直接跑审稿时用)。
|
# 已绑定 gateway 的节点形(图工厂 / T2.5 直接跑审稿时用)。
|
||||||
BoundReviewNode = Callable[[ChapterState], Awaitable[dict[str, Any]]]
|
BoundReviewNode = Callable[[ChapterState], Awaitable[dict[str, Any]]]
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
class GatewayRun(Protocol):
|
"REVIEW_INCOMPLETE",
|
||||||
"""审稿节点对网关的最小依赖——只需 `run`(注入真网关或 mock)。"""
|
"REVIEW_OK",
|
||||||
|
"BoundReviewNode",
|
||||||
async def run(self, req: LlmRequest) -> LlmResponse: ...
|
"GatewayRun",
|
||||||
|
"build_review_context",
|
||||||
|
"build_review_request",
|
||||||
|
"make_review_node",
|
||||||
|
"run_review",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def build_review_context(*, draft: str, stable_core: str, volatile: str) -> str:
|
def build_review_context(*, draft: str, stable_core: str, volatile: str) -> str:
|
||||||
|
|||||||
@@ -17,25 +17,16 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import uuid
|
import uuid
|
||||||
from typing import Protocol
|
|
||||||
|
|
||||||
import structlog
|
import structlog
|
||||||
from ww_agents import AgentSpec, StyleFingerprintResult
|
from ww_agents import AgentSpec, StyleFingerprintResult
|
||||||
from ww_llm_gateway.types import Block, LlmRequest, LlmResponse, Scope
|
from ww_llm_gateway.types import Block, LlmRequest, Scope
|
||||||
|
|
||||||
|
from ._protocols import GatewayRun
|
||||||
|
|
||||||
log = structlog.get_logger(__name__)
|
log = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class GatewayRun(Protocol):
|
|
||||||
"""文风提取节点对网关的最小依赖——只需 `run`(注入真网关或 mock)。
|
|
||||||
|
|
||||||
模块内部用(每模块各自声明,不跨模块复用、不在 orchestrator `__init__` 重复导出,
|
|
||||||
见 gotchas 2026-06-18)。
|
|
||||||
"""
|
|
||||||
|
|
||||||
async def run(self, req: LlmRequest) -> LlmResponse: ...
|
|
||||||
|
|
||||||
|
|
||||||
def build_style_extract_request(
|
def build_style_extract_request(
|
||||||
spec: AgentSpec,
|
spec: AgentSpec,
|
||||||
*,
|
*,
|
||||||
|
|||||||
@@ -108,6 +108,22 @@ def transient(msg: str = "boom") -> TransientProviderError:
|
|||||||
return TransientProviderError(msg)
|
return TransientProviderError(msg)
|
||||||
|
|
||||||
|
|
||||||
|
class AuthError(Exception):
|
||||||
|
"""模拟持续性鉴权失败(坏 key / 账号被禁):带 `status_code`,**非**瞬时不可重试。
|
||||||
|
|
||||||
|
适配器对 401/403 不翻译为 `TransientProviderError`,原样上抛;网关须对其计入熔断
|
||||||
|
(P1-2)而非每次白打同一坏 provider。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, msg: str = "unauthorized", *, status_code: int = 401) -> None:
|
||||||
|
super().__init__(msg)
|
||||||
|
self.status_code = status_code
|
||||||
|
|
||||||
|
|
||||||
|
def auth_error(status_code: int = 401) -> AuthError:
|
||||||
|
return AuthError(status_code=status_code)
|
||||||
|
|
||||||
|
|
||||||
def chain(*routes: tuple[str, str]) -> list[Route]:
|
def chain(*routes: tuple[str, str]) -> list[Route]:
|
||||||
return [Route(provider=p, model=m) for p, m in routes]
|
return [Route(provider=p, model=m) for p, m in routes]
|
||||||
|
|
||||||
|
|||||||
@@ -8,7 +8,15 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from fakes_resilience import FakeLedger, ScriptedAdapter, chain, chain_resolver, transient
|
from fakes_resilience import (
|
||||||
|
AuthError,
|
||||||
|
FakeLedger,
|
||||||
|
ScriptedAdapter,
|
||||||
|
auth_error,
|
||||||
|
chain,
|
||||||
|
chain_resolver,
|
||||||
|
transient,
|
||||||
|
)
|
||||||
from ww_llm_gateway.gateway import CircuitBreaker, Gateway
|
from ww_llm_gateway.gateway import CircuitBreaker, Gateway
|
||||||
from ww_llm_gateway.types import LlmRequest
|
from ww_llm_gateway.types import LlmRequest
|
||||||
from ww_shared import AppError, ErrorCode
|
from ww_shared import AppError, ErrorCode
|
||||||
@@ -154,6 +162,39 @@ def test_circuit_breaker_reopens_after_cooldown() -> None:
|
|||||||
assert cb.is_open("deepseek") is False
|
assert cb.is_open("deepseek") is False
|
||||||
|
|
||||||
|
|
||||||
|
async def test_persistent_auth_error_counts_toward_breaker(req: LlmRequest) -> None:
|
||||||
|
"""P1-2:持续性 401(坏 key)虽不可重试,但应计入熔断——连续 N 次后熔断打开。
|
||||||
|
|
||||||
|
每次 `run` 命中 401 立即上抛(不重试、不回退),但 raise 前 `record_failure`;
|
||||||
|
达到阈值后熔断打开,后续请求直接跳过该 provider → 链耗尽抛 LLM_UNAVAILABLE。
|
||||||
|
"""
|
||||||
|
threshold = 3
|
||||||
|
primary = ScriptedAdapter("deepseek", failures=[auth_error() for _ in range(threshold + 2)])
|
||||||
|
cb = CircuitBreaker(threshold=threshold, reset_seconds=60.0)
|
||||||
|
gw = Gateway(
|
||||||
|
{"deepseek": primary},
|
||||||
|
FakeLedger(),
|
||||||
|
chain_resolver=chain_resolver(chain(("deepseek", "deepseek-chat"))),
|
||||||
|
breaker=cb,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 前 threshold 次:401 原样上抛(不可重试),但每次计一次熔断失败。
|
||||||
|
for _ in range(threshold):
|
||||||
|
assert cb.is_open("deepseek") is False
|
||||||
|
with pytest.raises(AuthError):
|
||||||
|
await gw.run(req)
|
||||||
|
|
||||||
|
# 第 threshold 次失败后熔断打开。
|
||||||
|
assert cb.is_open("deepseek") is True
|
||||||
|
|
||||||
|
# 后续请求:provider 被熔断跳过 → 不再调 complete,链耗尽 → LLM_UNAVAILABLE。
|
||||||
|
calls_before = primary.complete_calls
|
||||||
|
with pytest.raises(AppError) as exc:
|
||||||
|
await gw.run(req)
|
||||||
|
assert exc.value.code == ErrorCode.LLM_UNAVAILABLE
|
||||||
|
assert primary.complete_calls == calls_before # 熔断后未再触达坏 provider
|
||||||
|
|
||||||
|
|
||||||
async def test_open_circuit_skips_provider(req: LlmRequest) -> None:
|
async def test_open_circuit_skips_provider(req: LlmRequest) -> None:
|
||||||
# 熔断已打开的主 provider 被直接跳过,连 complete 都不调,直接走回退。
|
# 熔断已打开的主 provider 被直接跳过,连 complete 都不调,直接走回退。
|
||||||
primary = ScriptedAdapter("deepseek", text="不该被调")
|
primary = ScriptedAdapter("deepseek", text="不该被调")
|
||||||
|
|||||||
@@ -16,7 +16,13 @@ from pydantic import BaseModel
|
|||||||
|
|
||||||
from ..errors import TransientProviderError
|
from ..errors import TransientProviderError
|
||||||
from ..types import LlmRequest
|
from ..types import LlmRequest
|
||||||
from .base import Capabilities, ProviderResult, ProviderUsage, StreamChunk
|
from .base import (
|
||||||
|
Capabilities,
|
||||||
|
ProviderResult,
|
||||||
|
ProviderUsage,
|
||||||
|
StreamChunk,
|
||||||
|
is_transient_by_name,
|
||||||
|
)
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from anthropic import AsyncAnthropic
|
from anthropic import AsyncAnthropic
|
||||||
@@ -44,11 +50,7 @@ class StructuredAnthropic(Protocol):
|
|||||||
|
|
||||||
|
|
||||||
def _is_transient(exc: Exception) -> bool:
|
def _is_transient(exc: Exception) -> bool:
|
||||||
name = type(exc).__name__
|
return is_transient_by_name(exc, _TRANSIENT_NAMES)
|
||||||
if name in _TRANSIENT_NAMES:
|
|
||||||
return True
|
|
||||||
status = getattr(exc, "status_code", None)
|
|
||||||
return isinstance(status, int) and (status == 429 or status >= 500)
|
|
||||||
|
|
||||||
|
|
||||||
def _system_blocks(req: LlmRequest) -> list[dict[str, Any]]:
|
def _system_blocks(req: LlmRequest) -> list[dict[str, Any]]:
|
||||||
@@ -144,7 +146,8 @@ class AnthropicAdapter:
|
|||||||
return ProviderResult(text=_text_from(resp), usage=usage)
|
return ProviderResult(text=_text_from(resp), usage=usage)
|
||||||
|
|
||||||
async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult:
|
async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult:
|
||||||
assert req.output_schema is not None
|
if req.output_schema is None:
|
||||||
|
raise ValueError("_complete_structured called without output_schema")
|
||||||
kwargs: dict[str, Any] = {
|
kwargs: dict[str, Any] = {
|
||||||
"model": model,
|
"model": model,
|
||||||
"max_tokens": req.max_tokens or _DEFAULT_MAX_TOKENS,
|
"max_tokens": req.max_tokens or _DEFAULT_MAX_TOKENS,
|
||||||
|
|||||||
@@ -12,6 +12,34 @@ from pydantic import BaseModel, ConfigDict
|
|||||||
|
|
||||||
from ..types import LlmRequest
|
from ..types import LlmRequest
|
||||||
|
|
||||||
|
# 瞬时(可退避重试)的 HTTP 状态码判定:429 限流 或 5xx 服务端错误。
|
||||||
|
_RATE_LIMITED_STATUS = 429
|
||||||
|
_SERVER_ERROR_MIN_STATUS = 500
|
||||||
|
|
||||||
|
|
||||||
|
def _is_transient_status(status: object) -> bool:
|
||||||
|
return isinstance(status, int) and (
|
||||||
|
status == _RATE_LIMITED_STATUS or status >= _SERVER_ERROR_MIN_STATUS
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def is_transient_by_name(
|
||||||
|
exc: Exception, names: frozenset[str], *, extra_codes: bool = False
|
||||||
|
) -> bool:
|
||||||
|
"""统一的瞬时错误判定(DRY:三个适配器共用)。
|
||||||
|
|
||||||
|
① 异常类名命中 `names`(按名匹配,避免硬依赖各厂商 SDK 异常类型);或
|
||||||
|
② `status_code` 属性是 429/5xx;或
|
||||||
|
③ `extra_codes=True` 时额外检查 `code` 属性是否为 429/5xx(Gemini 用 `exc.code`)。
|
||||||
|
"""
|
||||||
|
if type(exc).__name__ in names:
|
||||||
|
return True
|
||||||
|
if _is_transient_status(getattr(exc, "status_code", None)):
|
||||||
|
return True
|
||||||
|
if extra_codes and _is_transient_status(getattr(exc, "code", None)):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
class Capabilities(BaseModel):
|
class Capabilities(BaseModel):
|
||||||
structured_output: bool = False
|
structured_output: bool = False
|
||||||
|
|||||||
@@ -14,7 +14,13 @@ from typing import Any, Protocol
|
|||||||
|
|
||||||
from ..errors import TransientProviderError
|
from ..errors import TransientProviderError
|
||||||
from ..types import LlmRequest
|
from ..types import LlmRequest
|
||||||
from .base import Capabilities, ProviderResult, ProviderUsage, StreamChunk
|
from .base import (
|
||||||
|
Capabilities,
|
||||||
|
ProviderResult,
|
||||||
|
ProviderUsage,
|
||||||
|
StreamChunk,
|
||||||
|
is_transient_by_name,
|
||||||
|
)
|
||||||
|
|
||||||
_TRANSIENT_NAMES = frozenset(
|
_TRANSIENT_NAMES = frozenset(
|
||||||
{
|
{
|
||||||
@@ -31,7 +37,7 @@ _TRANSIENT_NAMES = frozenset(
|
|||||||
class GeminiModels(Protocol):
|
class GeminiModels(Protocol):
|
||||||
async def generate_content(self, **kwargs: Any) -> Any: ...
|
async def generate_content(self, **kwargs: Any) -> Any: ...
|
||||||
|
|
||||||
def generate_content_stream(self, **kwargs: Any) -> Any: ...
|
async def generate_content_stream(self, **kwargs: Any) -> Any: ...
|
||||||
|
|
||||||
|
|
||||||
class GeminiAio(Protocol):
|
class GeminiAio(Protocol):
|
||||||
@@ -45,11 +51,8 @@ class GeminiClient(Protocol):
|
|||||||
|
|
||||||
|
|
||||||
def _is_transient(exc: Exception) -> bool:
|
def _is_transient(exc: Exception) -> bool:
|
||||||
name = type(exc).__name__
|
# Gemini SDK 部分错误用 `exc.code` 携带状态码 → extra_codes=True 一并检查。
|
||||||
if name in _TRANSIENT_NAMES:
|
return is_transient_by_name(exc, _TRANSIENT_NAMES, extra_codes=True)
|
||||||
return True
|
|
||||||
status = getattr(exc, "code", None) or getattr(exc, "status_code", None)
|
|
||||||
return isinstance(status, int) and (status == 429 or status >= 500)
|
|
||||||
|
|
||||||
|
|
||||||
def _contents(req: LlmRequest) -> str:
|
def _contents(req: LlmRequest) -> str:
|
||||||
|
|||||||
@@ -15,7 +15,13 @@ from pydantic import BaseModel
|
|||||||
|
|
||||||
from ..errors import TransientProviderError
|
from ..errors import TransientProviderError
|
||||||
from ..types import LlmRequest
|
from ..types import LlmRequest
|
||||||
from .base import Capabilities, ProviderResult, ProviderUsage, StreamChunk
|
from .base import (
|
||||||
|
Capabilities,
|
||||||
|
ProviderResult,
|
||||||
|
ProviderUsage,
|
||||||
|
StreamChunk,
|
||||||
|
is_transient_by_name,
|
||||||
|
)
|
||||||
|
|
||||||
# OpenAI 兼容 SDK 的瞬时错误类名(按名匹配,覆盖 DeepSeek/Kimi/Qwen/GLM 等共用 SDK)。
|
# OpenAI 兼容 SDK 的瞬时错误类名(按名匹配,覆盖 DeepSeek/Kimi/Qwen/GLM 等共用 SDK)。
|
||||||
_TRANSIENT_NAMES = frozenset(
|
_TRANSIENT_NAMES = frozenset(
|
||||||
@@ -30,11 +36,7 @@ _TRANSIENT_NAMES = frozenset(
|
|||||||
|
|
||||||
|
|
||||||
def _is_transient(exc: Exception) -> bool:
|
def _is_transient(exc: Exception) -> bool:
|
||||||
name = type(exc).__name__
|
return is_transient_by_name(exc, _TRANSIENT_NAMES)
|
||||||
if name in _TRANSIENT_NAMES:
|
|
||||||
return True
|
|
||||||
status = getattr(exc, "status_code", None)
|
|
||||||
return isinstance(status, int) and (status == 429 or status >= 500)
|
|
||||||
|
|
||||||
|
|
||||||
class StructuredClient(Protocol):
|
class StructuredClient(Protocol):
|
||||||
@@ -126,7 +128,8 @@ class OpenAICompatAdapter:
|
|||||||
return ProviderResult(text=text, usage=_usage_from(resp.usage))
|
return ProviderResult(text=text, usage=_usage_from(resp.usage))
|
||||||
|
|
||||||
async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult:
|
async def _complete_structured(self, req: LlmRequest, model: str) -> ProviderResult:
|
||||||
assert req.output_schema is not None
|
if req.output_schema is None:
|
||||||
|
raise ValueError("_complete_structured called without output_schema")
|
||||||
parsed, raw = await self._structured().create_with_completion(
|
parsed, raw = await self._structured().create_with_completion(
|
||||||
messages=_messages(req),
|
messages=_messages(req),
|
||||||
response_model=req.output_schema,
|
response_model=req.output_schema,
|
||||||
|
|||||||
@@ -52,6 +52,19 @@ def _is_retryable(exc: BaseException) -> bool:
|
|||||||
return isinstance(exc, AppError) and exc.code == ErrorCode.RATE_LIMITED
|
return isinstance(exc, AppError) and exc.code == ErrorCode.RATE_LIMITED
|
||||||
|
|
||||||
|
|
||||||
|
# 持续性鉴权错误状态码:坏 key / 被禁用 → 每次重打都失败,应计入熔断(P1-2)。
|
||||||
|
_PERSISTENT_AUTH_STATUSES = frozenset({401, 403})
|
||||||
|
|
||||||
|
|
||||||
|
def _is_persistent_auth_error(exc: BaseException) -> bool:
|
||||||
|
"""持续性鉴权失败(401/403):错误 key / 账号被禁,重打无意义 → 计入熔断。
|
||||||
|
|
||||||
|
按 `status_code` 属性识别(不硬依赖任何厂商 SDK 异常类型)。
|
||||||
|
"""
|
||||||
|
status = getattr(exc, "status_code", None)
|
||||||
|
return isinstance(status, int) and status in _PERSISTENT_AUTH_STATUSES
|
||||||
|
|
||||||
|
|
||||||
def _input_len(req: LlmRequest) -> int:
|
def _input_len(req: LlmRequest) -> int:
|
||||||
if isinstance(req.input, str):
|
if isinstance(req.input, str):
|
||||||
return len(req.input)
|
return len(req.input)
|
||||||
@@ -187,7 +200,7 @@ class Gateway:
|
|||||||
project_id=str(req.scope.project_id) if req.scope.project_id else None,
|
project_id=str(req.scope.project_id) if req.scope.project_id else None,
|
||||||
)
|
)
|
||||||
|
|
||||||
async def _retrying(self) -> AsyncRetrying:
|
def _retrying(self) -> AsyncRetrying:
|
||||||
return AsyncRetrying(
|
return AsyncRetrying(
|
||||||
stop=stop_after_attempt(self._max_retries + 1),
|
stop=stop_after_attempt(self._max_retries + 1),
|
||||||
wait=wait_exponential(min=_RETRY_MIN_SECONDS, max=_RETRY_MAX_SECONDS),
|
wait=wait_exponential(min=_RETRY_MIN_SECONDS, max=_RETRY_MAX_SECONDS),
|
||||||
@@ -212,6 +225,10 @@ class Gateway:
|
|||||||
result = await self._complete_with_retry(adapter, req, route.model)
|
result = await self._complete_with_retry(adapter, req, route.model)
|
||||||
except Exception as exc: # noqa: BLE001 — 链内逐 provider 兜底,最终统一上抛
|
except Exception as exc: # noqa: BLE001 — 链内逐 provider 兜底,最终统一上抛
|
||||||
if not _is_retryable(exc):
|
if not _is_retryable(exc):
|
||||||
|
# 持续性鉴权失败(401/403)虽不可重试,但坏 key 应触发熔断(P1-2),
|
||||||
|
# 否则每次都白打同一坏 provider。其它不可重试错误直接上抛。
|
||||||
|
if _is_persistent_auth_error(exc):
|
||||||
|
self._breaker.record_failure(route.provider)
|
||||||
raise
|
raise
|
||||||
self._breaker.record_failure(route.provider)
|
self._breaker.record_failure(route.provider)
|
||||||
last_error = exc
|
last_error = exc
|
||||||
@@ -240,7 +257,7 @@ class Gateway:
|
|||||||
async def _complete_with_retry(
|
async def _complete_with_retry(
|
||||||
self, adapter: ProviderAdapter, req: LlmRequest, model: str
|
self, adapter: ProviderAdapter, req: LlmRequest, model: str
|
||||||
) -> ProviderResult:
|
) -> ProviderResult:
|
||||||
retrying = await self._retrying()
|
retrying = self._retrying()
|
||||||
async for attempt in retrying:
|
async for attempt in retrying:
|
||||||
with attempt:
|
with attempt:
|
||||||
return await adapter.complete(req, model)
|
return await adapter.complete(req, model)
|
||||||
|
|||||||
Reference in New Issue
Block a user