test(qa): craft 成稿质量前后对照评估回路（灵感 F2）
固定 3 题材金标准输入（仙侠/都市/悬疑，各含最小 spec+world+outline 及开篇章）， 经真实 assemble()+build_write_request() 组装 craft-on / craft-off 两版写章请求： - dry 模式（默认，零成本、免真实 LLM，可进 CI）：断言 craft 教条块、genre 片段、 开篇黄金三章 marker 确实按 fixture 注入正确内容，且 craft-off 基线不含这三者； 并把两版将发送给模型的 prompt 并排 dump 供人眼看。这些断言即 tests/test_craft_eval.py， 是 craft/审查注入路径的回归门禁。 - --live 模式（真实付费 LLM，绝不进 CI）：经 gateway 各生成一章 + analyst 五维裁判， 凭据读 DB、路由走 tier_routing，与生产写章同一网关路径。 入口 `uv run python -m tests.craft_eval [--live]`；用法/成本/门禁说明见 README.md。
This commit is contained in:
Yaojia Wang
59
tests/craft_eval/README.md
Normal file
59
tests/craft_eval/README.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# craft A/B 成稿质量前后对照评估回路（灵感 F2）
|
||||
|
||||
把「加 craft 杠杆」与「验证 craft 有效」绑成一件事的评估回路。每引入或改动一个 craft
|
||||
杠杆（① 写章教条 / F1 genre 片段 / 开篇黄金三章 / 后续 ② 反哺、③ 人物塑造审查等），
|
||||
就跑一次这里，做 craft-on vs craft-off 的前后对照——**没有它，craft 改动都是盲飞，
|
||||
很可能把文风做得更 AI 公式化而不自知**（见计划 §5 风险 #13）。
|
||||
|
||||
## 它是什么
|
||||
|
||||
- 固定一组**金标准题材输入**（`fixtures.py`：仙侠 / 都市 / 悬疑，各含最小 spec+world+outline，
|
||||
且都是开篇章以覆盖黄金三章 marker）。
|
||||
- 对每份 fixture 走**真实生产组装路径**（`assemble()` + `build_write_request()`）产出两版
|
||||
写章请求：
|
||||
- **craft-on** = 生产路径：craft 教条置顶 system 块 + genre 片段进 stable_core 缓存前缀 +
|
||||
开篇 marker 进 volatile。
|
||||
- **craft-off** = 同一 fixture 抹掉 craft 层的基线（剥掉 genre/开篇两节、不加教条块）。
|
||||
两版**唯一差异即 craft 贡献**，便于并排 diff 与 LLM 裁判对比。
|
||||
|
||||
## 两种模式
|
||||
|
||||
### dry（默认，零成本、免真实 LLM）
|
||||
|
||||
```bash
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
uv run python -m tests.craft_eval # 全部题材
|
||||
uv run python -m tests.craft_eval --genre 仙侠
|
||||
```
|
||||
|
||||
只做**确定性校验 + prompt 并排 dump**：断言 craft 教条块、genre 片段、开篇 marker 确实按
|
||||
fixture 注入了正确内容（且 craft-off 基线不含这三者），并把两版**将发送给模型的 prompt**
|
||||
并排写进报告供人眼看。**不碰 LLM**，任一断言失败退出码非零——这些断言即
|
||||
`tests/test_craft_eval.py`，随常规 `pytest` / CI 跑，是 craft 注入路径的回归门禁。
|
||||
|
||||
### live（真实付费生成 + LLM 五维裁判）
|
||||
|
||||
```bash
|
||||
uv run python -m tests.craft_eval --live
|
||||
```
|
||||
|
||||
**成本警告**：`--live` **真实调用付费 LLM**——每个题材跑 writer 生成 craft-on/craft-off 两章
|
||||
＋ analyst 五维裁判 1 次（3 题材 ≈ 6 次写章 + 3 次裁判）。凭据从 DB 读、路由走 `tier_routing`，
|
||||
与生产写章同一条网关路径（需 pg 在跑且已配好 provider 凭据；`--live` 前会先跑 dry 校验，
|
||||
未全绿则拒绝进入付费生成）。**绝不进 CI**：live 路径在 `live.py`，pytest 不导入它。
|
||||
|
||||
报告默认写系统临时目录（`--out PATH` 自定义）；含每题材的 dry 校验清单、prompt 并排、
|
||||
以及 live 模式下的两版成稿与五维裁判结论。
|
||||
|
||||
## 加了新 craft 杠杆后怎么用它把关
|
||||
|
||||
1. 改 craft（教条 `write_craft.md` / `GENRE_CRAFT` / 新审查维度……）。
|
||||
2. 跑 `pytest tests/test_craft_eval.py` 确认注入路径没被改坏（dry 门禁）。
|
||||
3. 跑 `--live` 生成前后对照，人眼 + LLM 五维裁判判断**是更对味还是更公式化**——
|
||||
据此决定该杠杆保留 / 回退（计划 §5 通用回滚：每杠杆可独立回退）。
|
||||
|
||||
## 加题材 / 调 fixture
|
||||
|
||||
在 `fixtures.py` 追加一个 `FixtureCase`（`genre` 必须命中 `ww_config.GENRE_CRAFT` 才有
|
||||
片段可注入）。`harness.py` 的 craft-off 靠 `assemble` 的节标题（`本作题材写法` / `开篇黄金三章`）
|
||||
剥离——上游若改节名，剥离失配会被 dry 断言当场炸出（craft-off 里仍含片段 → 断言失败）。
|
||||
38
tests/craft_eval/__init__.py
Normal file
38
tests/craft_eval/__init__.py
Normal file
@@ -0,0 +1,38 @@
|
||||
"""craft A/B 成稿质量前后对照评估回路（灵感 F2）。
|
||||
|
||||
固定金标准题材输入 → 组装 craft-on / craft-off 两版写章请求 → dry 确定性校验（零成本、
|
||||
可进 CI）或 `--live` 真实生成 + LLM 五维裁判。用法见 `README.md` / `python -m tests.craft_eval -h`。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .fixtures import FIXTURES, FixtureCase
|
||||
from .harness import (
|
||||
OPENING_CHAPTER_LIMIT,
|
||||
CaseReport,
|
||||
Check,
|
||||
PromptPair,
|
||||
all_passed,
|
||||
build_prompt_pair,
|
||||
input_text,
|
||||
render_report,
|
||||
system_text,
|
||||
verify_case,
|
||||
write_report,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"FIXTURES",
|
||||
"OPENING_CHAPTER_LIMIT",
|
||||
"CaseReport",
|
||||
"Check",
|
||||
"FixtureCase",
|
||||
"PromptPair",
|
||||
"all_passed",
|
||||
"build_prompt_pair",
|
||||
"input_text",
|
||||
"render_report",
|
||||
"system_text",
|
||||
"verify_case",
|
||||
"write_report",
|
||||
]
|
||||
104
tests/craft_eval/__main__.py
Normal file
104
tests/craft_eval/__main__.py
Normal file
@@ -0,0 +1,104 @@
|
||||
"""craft A/B 评估脚本入口（灵感 F2）。
|
||||
|
||||
# dry（默认，零成本、免真实 LLM）——组装+校验+并排 dump prompt
|
||||
uv run python -m tests.craft_eval
|
||||
|
||||
# live（真实付费生成 + LLM 五维裁判）
|
||||
uv run python -m tests.craft_eval --live
|
||||
|
||||
选项：`--genre 仙侠` 只跑某题材；`--out PATH` 指定报告路径；`--quiet` 少打印。
|
||||
退出码：dry 任一断言失败 → 非零（可当回归门禁）。详见 README.md。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from .fixtures import FIXTURES, FixtureCase
|
||||
from .harness import (
|
||||
CaseReport,
|
||||
all_passed,
|
||||
build_prompt_pair,
|
||||
render_report,
|
||||
verify_case,
|
||||
write_report,
|
||||
)
|
||||
|
||||
|
||||
def _select(genre: str | None) -> list[FixtureCase]:
|
||||
if genre is None:
|
||||
return list(FIXTURES)
|
||||
picked = [c for c in FIXTURES if c.genre == genre]
|
||||
if not picked:
|
||||
raise SystemExit(f"没有题材为 {genre!r} 的 fixture；可选：{[c.genre for c in FIXTURES]}")
|
||||
return picked
|
||||
|
||||
|
||||
def _default_out(live: bool) -> Path:
|
||||
name = "craft_ab_live.md" if live else "craft_ab_dry.md"
|
||||
return Path(tempfile.gettempdir()) / name
|
||||
|
||||
|
||||
async def _run(args: argparse.Namespace) -> int:
|
||||
cases = _select(args.genre)
|
||||
|
||||
reports: list[CaseReport] = []
|
||||
for case in cases:
|
||||
pair = await build_prompt_pair(case)
|
||||
reports.append(CaseReport(case=case, pair=pair, checks=verify_case(case, pair)))
|
||||
|
||||
failed = [(r.case.slug, c) for r in reports for c in r.checks if not c.passed]
|
||||
if not args.quiet:
|
||||
for r in reports:
|
||||
status = "OK" if all_passed(r.checks) else "FAIL"
|
||||
print(f"[dry] {r.case.slug:10s} {r.case.genre} 第{r.pair.chapter_no}章 {status}")
|
||||
for slug, chk in failed:
|
||||
print(f"[dry][FAIL] {slug} · {chk.name}: {chk.detail}", file=sys.stderr)
|
||||
|
||||
if args.live:
|
||||
if failed:
|
||||
print("[live] dry 校验未全绿，拒绝进入付费生成——先修正注入。", file=sys.stderr)
|
||||
return 1
|
||||
from .live import run_live, summarize
|
||||
|
||||
print("[live] 开始真实生成（付费）……", flush=True)
|
||||
reports = await run_live(reports)
|
||||
print("\n[live] 摘要：\n" + summarize(reports))
|
||||
|
||||
out = Path(args.out) if args.out else _default_out(args.live)
|
||||
write_report(render_report(reports, live=args.live), out)
|
||||
print(f"\n[out] 报告写至 {out}")
|
||||
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="python -m tests.craft_eval",
|
||||
description="craft A/B 成稿质量前后对照评估（灵感 F2）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--live",
|
||||
action="store_true",
|
||||
help="真实调用付费 LLM 生成两版正文 + 五维裁判（默认 dry，免真实 LLM）",
|
||||
)
|
||||
parser.add_argument("--genre", default=None, help="只跑指定题材（如 仙侠/都市/悬疑）")
|
||||
parser.add_argument("--out", default=None, help="报告输出路径（默认写系统临时目录）")
|
||||
parser.add_argument("--quiet", action="store_true", help="减少 stdout 打印")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.live:
|
||||
# get_settings() 按 cwd 读 .env；确保 cwd = 仓库根（本文件在 tests/craft_eval/ 下三层）。
|
||||
import os
|
||||
|
||||
os.chdir(Path(__file__).resolve().parents[2])
|
||||
|
||||
raise SystemExit(asyncio.run(_run(args)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
158
tests/craft_eval/fixtures.py
Normal file
158
tests/craft_eval/fixtures.py
Normal file
@@ -0,0 +1,158 @@
|
||||
"""金标准输入 fixtures（灵感 F2）——3 个不同题材的最小 spec+world+outline。
|
||||
|
||||
每个 fixture 是一份**自造、不依赖 DB** 的最小设定：书级蓝本（含 genre）+ 世界观硬规则 +
|
||||
定型主角 + 本章大纲节拍。评估脚本据此组装 craft-on / craft-off 两版写章请求。
|
||||
新增题材只需追加一个 `FixtureCase`（genre 必须命中 `ww_config.GENRE_CRAFT` 才有片段可注）。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from ww_core.domain.repositories import CharacterView, ProjectSpecView, WorldEntityView
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FixtureCase:
|
||||
"""一份金标准写章输入（题材 / 章号 / 书级蓝本 / 世界观 / 主角 / 本章节拍）。"""
|
||||
|
||||
slug: str
|
||||
genre: str
|
||||
chapter_no: int
|
||||
spec: ProjectSpecView
|
||||
world: tuple[WorldEntityView, ...]
|
||||
characters: tuple[CharacterView, ...]
|
||||
beats: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
_XIANXIA = FixtureCase(
|
||||
slug="xianxia",
|
||||
genre="仙侠",
|
||||
chapter_no=3, # 开篇章（≤3）——覆盖黄金三章 marker 注入
|
||||
spec=ProjectSpecView(
|
||||
title="逆脉证道",
|
||||
genre="仙侠",
|
||||
logline="废脉少年叶尘为救绝症妹妹寻回春灵药，滴血觉醒上古逆天功法，逆修九境搅动仙门风云。",
|
||||
premise="天地灵气稀薄，修行看灵脉品阶；废脉者纳灵极慢，被视作修行绝路。",
|
||||
theme="卑微者的不屈与救赎——修的是道，护的是人。",
|
||||
tone="热血隐忍",
|
||||
narrative_pov="第三人称限知",
|
||||
ending_type="正剧",
|
||||
),
|
||||
world=(
|
||||
WorldEntityView(
|
||||
type="设定",
|
||||
name="九境修为",
|
||||
rules={
|
||||
"境界": ["淬体", "开脉", "灵动", "纳灵", "凝魂", "化虚", "通玄", "渡劫", "证道"],
|
||||
"每境": "分前中后三重",
|
||||
},
|
||||
),
|
||||
WorldEntityView(
|
||||
type="设定",
|
||||
name="灵脉",
|
||||
rules={"作用": "决定纳灵速度", "废脉": "纳灵极慢，被视作修行绝路"},
|
||||
),
|
||||
),
|
||||
characters=(
|
||||
CharacterView(
|
||||
name="叶尘",
|
||||
role="主角",
|
||||
motive="为身患寒髓之症、命不久矣的妹妹叶暖寻回春灵药",
|
||||
backstory="青云宗外门弟子，天生废脉、常年受人轻慢，性子沉静隐忍。",
|
||||
speech_tics={"风格": "话少、点到为止，逼到绝境时语气反而更冷更稳"},
|
||||
),
|
||||
),
|
||||
beats={
|
||||
"summary": "叶尘被同门构陷逐出青云宗，独行荒山避雨于古洞，"
|
||||
"滴血觉醒《逆脉真解》，一招震退追杀者。",
|
||||
"entities": ["叶尘"],
|
||||
"text": "叶尘 古洞 滴血 觉醒 逆脉真解 震退 追杀者",
|
||||
},
|
||||
)
|
||||
|
||||
_URBAN = FixtureCase(
|
||||
slug="urban",
|
||||
genre="都市",
|
||||
chapter_no=1, # 开篇第一章
|
||||
spec=ProjectSpecView(
|
||||
title="重启我的黄金时代",
|
||||
genre="都市",
|
||||
logline="落魄程序员林川一夜重生回到二十年前，凭前世记忆在时代风口逆天改命。",
|
||||
premise="重生回到世纪初互联网草莽年代，机会遍地、格局未定。",
|
||||
theme="不甘平庸者抓住的第二次机会。",
|
||||
tone="爽快励志",
|
||||
narrative_pov="第一人称",
|
||||
ending_type="HE",
|
||||
),
|
||||
world=(
|
||||
WorldEntityView(
|
||||
type="设定",
|
||||
name="重生规则",
|
||||
rules={
|
||||
"回溯": "重生回到 2003 年的大学时代",
|
||||
"记忆": "完整保留前世记忆与二十年行业趋势",
|
||||
},
|
||||
),
|
||||
),
|
||||
characters=(
|
||||
CharacterView(
|
||||
name="林川",
|
||||
role="主角",
|
||||
motive="改写前世潦倒猝死的结局，护住家人、抓住时代机遇翻身",
|
||||
backstory="前世三十五岁猝死于出租屋的落魄外包程序员。",
|
||||
speech_tics={"风格": "表面随和、心里门儿清，说话带点自嘲"},
|
||||
),
|
||||
),
|
||||
beats={
|
||||
"summary": "林川在出租屋猝死后睁眼回到 2003 年的大学宿舍，"
|
||||
"认出这是重来一次的机会，立下第一个目标。",
|
||||
"entities": ["林川"],
|
||||
"text": "林川 重生 2003 大学宿舍 认清机会",
|
||||
},
|
||||
)
|
||||
|
||||
_SUSPENSE = FixtureCase(
|
||||
slug="suspense",
|
||||
genre="悬疑",
|
||||
chapter_no=2, # 开篇第二章
|
||||
spec=ProjectSpecView(
|
||||
title="雾隐镇失踪案",
|
||||
genre="悬疑",
|
||||
logline="刑警周砚回到多雾的故乡小镇调查连环失踪案，线索却一步步指向自己尘封的童年。",
|
||||
premise="常年浓雾笼罩的封闭小镇，二十年前也发生过同样的失踪，无人被起诉。",
|
||||
theme="逼近真相与自我和解的代价。",
|
||||
tone="冷峻压抑",
|
||||
narrative_pov="第三人称限知",
|
||||
ending_type="开放",
|
||||
),
|
||||
world=(
|
||||
WorldEntityView(
|
||||
type="设定",
|
||||
name="雾隐镇",
|
||||
rules={
|
||||
"环境": "常年浓雾、信号时断时续",
|
||||
"旧案": "二十年前发生过同类失踪，卷宗离奇缺页",
|
||||
},
|
||||
),
|
||||
),
|
||||
characters=(
|
||||
CharacterView(
|
||||
name="周砚",
|
||||
role="主角",
|
||||
motive="查清连环失踪真相，也为解开自己童年缺失的那段记忆",
|
||||
backstory="市局刑警，幼年从雾隐镇被送走，对故乡的记忆断片。",
|
||||
speech_tics={"风格": "话短、爱用反问，审讯时不动声色"},
|
||||
),
|
||||
),
|
||||
beats={
|
||||
"summary": "周砚勘查第二名失踪者的住所，"
|
||||
"在墙缝里发现一张与二十年前旧案笔迹相同的纸条，心头一凛。",
|
||||
"entities": ["周砚"],
|
||||
"text": "周砚 现场勘查 墙缝 纸条 二十年前 旧案",
|
||||
},
|
||||
)
|
||||
|
||||
# 金标准题材集：仙侠 / 都市 / 悬疑（各含开篇章，覆盖黄金三章 marker）。
|
||||
FIXTURES: tuple[FixtureCase, ...] = (_XIANXIA, _URBAN, _SUSPENSE)
|
||||
363
tests/craft_eval/harness.py
Normal file
363
tests/craft_eval/harness.py
Normal file
@@ -0,0 +1,363 @@
|
||||
"""craft A/B 评估回路核心（灵感 F2）——组装请求 + dry 确定性校验 + 报告渲染。
|
||||
|
||||
**零 LLM 依赖**：本模块只走确定性组装（真 `assemble()` + 真 `build_write_request()`）与
|
||||
纯字符串校验，可安全进 CI。真实生成/裁判在 `live.py`（惰性导入，不进本模块）。
|
||||
|
||||
craft-on = 生产路径（`assemble` 注入 genre 片段进 stable_core + 开篇 marker 进 volatile，
|
||||
`build_write_request` 把 craft 教条置为最前 system 块）。
|
||||
craft-off = 同一 fixture 抹掉 craft 层的基线：剥掉 genre/开篇两节 + 不加教条块——
|
||||
两版**唯一差异即 craft 贡献**，便于人眼并排 diff 与 LLM 裁判对比。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from ww_core.domain.repositories import (
|
||||
CharacterView,
|
||||
DigestView,
|
||||
ForeshadowView,
|
||||
MemoryRepos,
|
||||
OutlineView,
|
||||
ProjectSpecView,
|
||||
RuleView,
|
||||
StyleView,
|
||||
WorldEntityView,
|
||||
)
|
||||
from ww_core.memory.assemble import OPENING_CHAPTER_LIMIT, assemble
|
||||
from ww_core.orchestrator.write_node import build_write_request
|
||||
from ww_llm_gateway.types import Block, LlmRequest, Scope
|
||||
|
||||
from .fixtures import FixtureCase
|
||||
|
||||
# 单用户原型 stub（dry 路径不落库，取值无关紧要，仅为满足 Scope）。
|
||||
PROJECT_ID = uuid.UUID("00000000-0000-0000-0000-0000000c7a17")
|
||||
USER_ID = uuid.UUID(int=1)
|
||||
|
||||
# assemble 内联的 craft 节标题（assemble.py:97 `本作题材写法` / :141 `开篇黄金三章`）。
|
||||
# craft-off 据此剥离；若上游改名，本表失配 → craft-off 仍含片段 → 校验断言会当场炸出漂移。
|
||||
_GENRE_SECTION_TITLE = "本作题材写法"
|
||||
_OPENING_SECTION_TITLE = "开篇黄金三章"
|
||||
|
||||
__all__ = [
|
||||
"OPENING_CHAPTER_LIMIT",
|
||||
"PROJECT_ID",
|
||||
"USER_ID",
|
||||
"CaseReport",
|
||||
"Check",
|
||||
"PromptPair",
|
||||
"all_passed",
|
||||
"build_prompt_pair",
|
||||
"input_text",
|
||||
"render_report",
|
||||
"system_text",
|
||||
"verify_case",
|
||||
"write_report",
|
||||
]
|
||||
|
||||
|
||||
# ---- 内存 fake repositories（只喂 assemble 所需，其余返回空）----
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeOutlineRepo:
|
||||
"""按查询章号合成 OutlineView（复用同一份 beats），支持任意 chapter_no 的边界测试。"""
|
||||
|
||||
volume: int
|
||||
beats: dict[str, Any]
|
||||
|
||||
async def get(self, project_id: uuid.UUID, chapter_no: int) -> OutlineView | None:
|
||||
return OutlineView(volume=self.volume, chapter_no=chapter_no, beats=dict(self.beats))
|
||||
|
||||
async def list_for_project(
|
||||
self, project_id: uuid.UUID
|
||||
) -> list[OutlineView]: # pragma: no cover
|
||||
return []
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeCharacterRepo:
|
||||
rows: tuple[CharacterView, ...]
|
||||
|
||||
async def list_for_project(self, project_id: uuid.UUID) -> list[CharacterView]:
|
||||
return list(self.rows)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeWorldEntityRepo:
|
||||
rows: tuple[WorldEntityView, ...]
|
||||
|
||||
async def list_for_project(self, project_id: uuid.UUID) -> list[WorldEntityView]:
|
||||
return list(self.rows)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeDigestRepo:
|
||||
async def recent(self, project_id: uuid.UUID, k: int) -> list[DigestView]:
|
||||
return []
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeForeshadowRepo:
|
||||
async def list_for_codes(self, project_id: uuid.UUID, codes: list[str]) -> list[ForeshadowView]:
|
||||
return []
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeStyleRepo:
|
||||
async def latest(self, project_id: uuid.UUID) -> StyleView | None:
|
||||
return None
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeRulesRepo:
|
||||
async def all_for_project(self, project_id: uuid.UUID) -> list[RuleView]:
|
||||
return []
|
||||
|
||||
|
||||
@dataclass
|
||||
class _FakeProjectSpecRepo:
|
||||
row: ProjectSpecView
|
||||
|
||||
async def spec(self, project_id: uuid.UUID) -> ProjectSpecView | None:
|
||||
return self.row
|
||||
|
||||
|
||||
def _build_repos(case: FixtureCase) -> MemoryRepos:
|
||||
volume = 1
|
||||
return MemoryRepos(
|
||||
outline=_FakeOutlineRepo(volume=volume, beats=case.beats),
|
||||
character=_FakeCharacterRepo(case.characters),
|
||||
world_entity=_FakeWorldEntityRepo(case.world),
|
||||
digest=_FakeDigestRepo(),
|
||||
foreshadow=_FakeForeshadowRepo(),
|
||||
style=_FakeStyleRepo(),
|
||||
rules=_FakeRulesRepo(),
|
||||
project=_FakeProjectSpecRepo(case.spec),
|
||||
review=None, # 冲突裁决反哺不在 craft A/B 范围；优雅降级不反哺
|
||||
)
|
||||
|
||||
|
||||
# ---- craft-on / craft-off 请求组装 ----
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PromptPair:
|
||||
craft_on: LlmRequest
|
||||
craft_off: LlmRequest
|
||||
chapter_no: int
|
||||
|
||||
|
||||
def _strip_section(text: str, title: str) -> str:
|
||||
"""按 `## {title}` 剥掉一节（节以 "\\n\\n" 分隔，节体无空行——见 assemble._section）。"""
|
||||
header = f"## {title}"
|
||||
kept = [chunk for chunk in text.split("\n\n") if not chunk.startswith(header)]
|
||||
return "\n\n".join(kept)
|
||||
|
||||
|
||||
async def build_prompt_pair(case: FixtureCase, *, chapter_no: int | None = None) -> PromptPair:
|
||||
"""据 fixture 组装 craft-on（生产路径）与 craft-off（抹除 craft 层的基线）两版请求。"""
|
||||
ch = chapter_no if chapter_no is not None else case.chapter_no
|
||||
ctx = await assemble(_build_repos(case), PROJECT_ID, ch)
|
||||
|
||||
craft_on = build_write_request(
|
||||
stable_core=ctx.stable_core,
|
||||
volatile=ctx.volatile,
|
||||
user_id=USER_ID,
|
||||
project_id=PROJECT_ID,
|
||||
)
|
||||
|
||||
off_stable = _strip_section(ctx.stable_core, _GENRE_SECTION_TITLE)
|
||||
off_volatile = _strip_section(ctx.volatile, _OPENING_SECTION_TITLE)
|
||||
craft_off = LlmRequest(
|
||||
tier="writer",
|
||||
system=[Block(text=off_stable, cache=True)],
|
||||
input=off_volatile,
|
||||
stream=True,
|
||||
scope=Scope(user_id=USER_ID, project_id=PROJECT_ID),
|
||||
)
|
||||
return PromptPair(craft_on=craft_on, craft_off=craft_off, chapter_no=ch)
|
||||
|
||||
|
||||
# ---- prompt 文本抽取（供校验与报告）----
|
||||
|
||||
|
||||
def system_text(req: LlmRequest) -> str:
|
||||
return "\n\n".join(b.text for b in req.system)
|
||||
|
||||
|
||||
def input_text(req: LlmRequest) -> str:
|
||||
if isinstance(req.input, str):
|
||||
return req.input
|
||||
return "\n\n".join(b.text for b in req.input)
|
||||
|
||||
|
||||
# ---- dry 确定性校验（无 LLM，可进 CI）----
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Check:
|
||||
name: str
|
||||
passed: bool
|
||||
detail: str
|
||||
|
||||
|
||||
def _load_doctrine() -> str:
|
||||
# 惰性导入避免 fixtures 模块与本模块的导入顺序耦合。
|
||||
from ww_agents import load_craft_doctrine
|
||||
|
||||
return load_craft_doctrine()
|
||||
|
||||
|
||||
def _genre_fragment(genre: str) -> str:
|
||||
from ww_config import genre_craft_fragment
|
||||
|
||||
fragment = genre_craft_fragment(genre)
|
||||
if fragment is None: # pragma: no cover - fixtures 已保证命中
|
||||
raise ValueError(f"题材 {genre!r} 无 GENRE_CRAFT 片段")
|
||||
return fragment
|
||||
|
||||
|
||||
def verify_case(case: FixtureCase, pair: PromptPair) -> list[Check]:
|
||||
"""对一份 fixture 的 craft-on/off 请求做确定性断言（craft 改动的回归门禁）。"""
|
||||
on, off = pair.craft_on, pair.craft_off
|
||||
doctrine = _load_doctrine()
|
||||
fragment = _genre_fragment(case.genre)
|
||||
on_sys, on_in = system_text(on), input_text(on)
|
||||
off_sys, off_in = system_text(off), input_text(off)
|
||||
is_opening = pair.chapter_no <= OPENING_CHAPTER_LIMIT
|
||||
|
||||
checks: list[Check] = [
|
||||
Check(
|
||||
"craft_on_doctrine_first_block",
|
||||
len(on.system) >= 2 and on.system[0].text == doctrine and on.system[0].cache,
|
||||
"教条须为首个 system 块、字节等于 load_craft_doctrine()、cache=True（#9）",
|
||||
),
|
||||
Check(
|
||||
"craft_on_genre_in_stable_not_doctrine_not_input",
|
||||
fragment in on.system[1].text
|
||||
and fragment not in on.system[0].text
|
||||
and fragment not in on_in,
|
||||
f"genre 片段须进 stable_core 缓存块、不进教条块/volatile（{case.genre}）",
|
||||
),
|
||||
Check(
|
||||
"craft_on_opening_marker",
|
||||
(_OPENING_SECTION_TITLE in on_in and _OPENING_SECTION_TITLE not in on_sys)
|
||||
if is_opening
|
||||
else (_OPENING_SECTION_TITLE not in on_in),
|
||||
f"开篇 marker：第 {pair.chapter_no} 章"
|
||||
+ ("须进 volatile、不进缓存 system" if is_opening else "非开篇章须不注入"),
|
||||
),
|
||||
Check(
|
||||
"craft_on_tier_writer",
|
||||
on.tier == "writer",
|
||||
"不变量 #2：只声明 tier=writer，不写具体 model",
|
||||
),
|
||||
Check(
|
||||
"craft_off_no_doctrine",
|
||||
doctrine not in off_sys and doctrine not in off_in,
|
||||
"craft-off 基线不含教条块",
|
||||
),
|
||||
Check(
|
||||
"craft_off_no_genre_fragment",
|
||||
fragment not in off_sys and fragment not in off_in,
|
||||
"craft-off 基线不含 genre 片段",
|
||||
),
|
||||
Check(
|
||||
"craft_off_no_opening_marker",
|
||||
_OPENING_SECTION_TITLE not in off_in,
|
||||
"craft-off 基线不含开篇 marker",
|
||||
),
|
||||
Check(
|
||||
"craft_off_retains_body",
|
||||
case.spec.title in off_sys and "请创作第" in off_in,
|
||||
"剥离未误伤本体：书名与写作指令仍在",
|
||||
),
|
||||
]
|
||||
return checks
|
||||
|
||||
|
||||
def all_passed(checks: list[Check]) -> bool:
|
||||
return all(c.passed for c in checks)
|
||||
|
||||
|
||||
# ---- 报告渲染 ----
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CaseReport:
|
||||
case: FixtureCase
|
||||
pair: PromptPair
|
||||
checks: list[Check]
|
||||
live: Any = None # live.LiveResult | None（惰性，避免 dry 路径导入 live）
|
||||
|
||||
|
||||
def _fence(label: str, body: str) -> str:
|
||||
return f"**{label}**\n\n```\n{body}\n```"
|
||||
|
||||
|
||||
def _render_pair(pair: PromptPair) -> str:
|
||||
on, off = pair.craft_on, pair.craft_off
|
||||
return "\n\n".join(
|
||||
[
|
||||
"#### craft-on（发送给模型的 prompt）",
|
||||
_fence("system（缓存前缀）", system_text(on)),
|
||||
_fence("input（volatile）", input_text(on)),
|
||||
"#### craft-off（基线 prompt）",
|
||||
_fence("system（缓存前缀）", system_text(off)),
|
||||
_fence("input（volatile）", input_text(off)),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _render_live(live: Any) -> str:
|
||||
return "\n\n".join(
|
||||
[
|
||||
"#### 生成结果（--live）",
|
||||
f"served_by: `{live.served_by}`",
|
||||
_fence(f"craft-on 成稿（{live.chars_on} 字）", live.chapter_on),
|
||||
_fence(f"craft-off 成稿（{live.chars_off} 字）", live.chapter_off),
|
||||
"#### LLM 五维裁判",
|
||||
live.judge_text,
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def render_report(reports: list[CaseReport], *, live: bool) -> str:
|
||||
total_checks = sum(len(r.checks) for r in reports)
|
||||
failed = sum(1 for r in reports for c in r.checks if not c.passed)
|
||||
lines: list[str] = [
|
||||
"# craft A/B 评估报告（灵感 F2）",
|
||||
"",
|
||||
f"模式：{'live（真实生成+裁判）' if live else 'dry（零成本确定性校验）'}",
|
||||
f"题材：{len(reports)} 个 · 断言：{total_checks - failed}/{total_checks} 通过",
|
||||
"",
|
||||
]
|
||||
for r in reports:
|
||||
c = r.case
|
||||
lines.append(f"## {c.slug} · {c.genre} · 《{c.spec.title}》 · 第 {r.pair.chapter_no} 章")
|
||||
lines.append("")
|
||||
lines.append("### dry 校验")
|
||||
for chk in r.checks:
|
||||
mark = "✅" if chk.passed else "❌"
|
||||
lines.append(f"- {mark} `{chk.name}` — {chk.detail}")
|
||||
lines.append("")
|
||||
lines.append("### prompt 并排")
|
||||
lines.append("")
|
||||
lines.append(_render_pair(r.pair))
|
||||
lines.append("")
|
||||
if r.live is not None:
|
||||
lines.append(_render_live(r.live))
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def write_report(text: str, path: Any) -> Any:
|
||||
from pathlib import Path
|
||||
|
||||
out = Path(path)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(text, encoding="utf-8")
|
||||
return out
|
||||
127
tests/craft_eval/live.py
Normal file
127
tests/craft_eval/live.py
Normal file
@@ -0,0 +1,127 @@
|
||||
"""`--live` 真实生成 + LLM 五维裁判（灵感 F2）。
|
||||
|
||||
**成本警告**：本模块**真实调用付费 LLM**（每题材：writer 生成 2 章 + analyst 裁判 1 次）。
|
||||
仅供人工手动跑，**绝不进 pytest / CI**（故与 dry 校验分文件，pytest 不导入本模块）。
|
||||
凭据从 DB 读（`SqlCredentialStore`），路由从 `tier_routing` 取——与生产写章同一条网关路径。
|
||||
|
||||
用法见 `README.md`：`uv run python -m tests.craft_eval --live`。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ww_api.services.credentials import STUB_OWNER_ID, SqlCredentialStore
|
||||
from ww_api.services.project_deps import build_gateway_for_tier
|
||||
from ww_db import get_sessionmaker
|
||||
from ww_llm_gateway.types import Block, LlmRequest, Scope
|
||||
|
||||
from .harness import CaseReport, PromptPair
|
||||
|
||||
# 五维对比裁判（对齐 craft_ab 实验的口径）——只就文本本身逐维对比，不复述原文。
|
||||
JUDGE_SYSTEM = """你是资深中文网络小说主编，阅稿无数，眼光毒辣、直言不讳。
|
||||
现在给你同一章大纲下、由两套不同写作指令产出的两版正文（A=craft-on，B=craft-off）。
|
||||
请仅就文本本身，逐维度对比谁更好，每个维度给出「A更好 / B更好 / 难分」并附一句依据：
|
||||
① show-don't-tell（情绪是否靠动作细节外化，而非旁白直说）
|
||||
② 对话是否有角色声音/口癖
|
||||
③ 章末钩子是否成立（是否让读者想追下一章）
|
||||
④ 是否注水/AI 腔（空泛排比、翻译腔、无信息量的辞藻堆砌）
|
||||
⑤ 开篇代入感（前几段是否立住冲突/身份钩子、抓人）
|
||||
最后给出总判：「A 更好 / B 更好 / 难分」，并用 1-2 句话说明理由。
|
||||
保持简洁、务实，不要复述原文。"""
|
||||
|
||||
# 裁判每稿截断字数（省 token）。
|
||||
_JUDGE_EXCERPT = 1500
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LiveResult:
|
||||
chapter_on: str
|
||||
chapter_off: str
|
||||
chars_on: int
|
||||
chars_off: int
|
||||
judge_text: str
|
||||
served_by: str
|
||||
|
||||
|
||||
def char_count(text: str) -> int:
|
||||
"""中文成稿近似字数：去空白后的字符数。"""
|
||||
return len("".join(text.split()))
|
||||
|
||||
|
||||
async def _stream_text(gateway: object, req: LlmRequest) -> str:
|
||||
parts: list[str] = []
|
||||
async for delta in gateway.stream(req): # type: ignore[attr-defined]
|
||||
parts.append(delta.text)
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _judge_request(pair: PromptPair, text_on: str, text_off: str) -> LlmRequest:
|
||||
body = (
|
||||
"以下是两版正文（各截取前若干字以省 token）。\n\n"
|
||||
f"===== A 稿 / craft-on（前 {_JUDGE_EXCERPT} 字）=====\n"
|
||||
+ text_on[:_JUDGE_EXCERPT]
|
||||
+ f"\n\n===== B 稿 / craft-off（前 {_JUDGE_EXCERPT} 字）=====\n"
|
||||
+ text_off[:_JUDGE_EXCERPT]
|
||||
)
|
||||
return LlmRequest(
|
||||
tier="analyst",
|
||||
system=[Block(text=JUDGE_SYSTEM, cache=True)],
|
||||
input=body,
|
||||
stream=False,
|
||||
scope=Scope(user_id=STUB_OWNER_ID, project_id=None),
|
||||
)
|
||||
|
||||
|
||||
async def run_live(reports: list[CaseReport]) -> list[CaseReport]:
|
||||
"""对每个 report 的 craft-on/off 请求真实生成一章 + 五维裁判，返回附 live 的新 report 列表。
|
||||
|
||||
不可变更新：用 `dataclasses.replace` 产出新 `CaseReport`，不原地改冻结实例。
|
||||
"""
|
||||
sm = get_sessionmaker()
|
||||
out: list[CaseReport] = []
|
||||
async with sm() as session:
|
||||
store = SqlCredentialStore(session)
|
||||
# writer / analyst 各建一次网关，跨题材复用（省装配开销）。
|
||||
writer_gw = await build_gateway_for_tier(session, store, "writer")
|
||||
analyst_gw = await build_gateway_for_tier(session, store, "analyst")
|
||||
|
||||
for report in reports:
|
||||
pair = report.pair
|
||||
print(f"[live] {report.case.slug}：生成 craft-on……", flush=True)
|
||||
text_on = await _stream_text(writer_gw, _as_scoped(pair.craft_on))
|
||||
print(f"[live] {report.case.slug}：生成 craft-off……", flush=True)
|
||||
text_off = await _stream_text(writer_gw, _as_scoped(pair.craft_off))
|
||||
print(f"[live] {report.case.slug}：analyst 五维裁判……", flush=True)
|
||||
resp = await analyst_gw.run(_judge_request(pair, text_on, text_off))
|
||||
|
||||
result = LiveResult(
|
||||
chapter_on=text_on,
|
||||
chapter_off=text_off,
|
||||
chars_on=char_count(text_on),
|
||||
chars_off=char_count(text_off),
|
||||
judge_text=resp.text,
|
||||
served_by=f"{resp.served_by.provider}:{resp.served_by.model}"
|
||||
+ (" (fell_back)" if resp.served_by.fell_back else ""),
|
||||
)
|
||||
out.append(dataclasses.replace(report, live=result))
|
||||
return out
|
||||
|
||||
|
||||
def _as_scoped(req: LlmRequest) -> LlmRequest:
|
||||
"""把请求 scope 换成 stub owner（usage_ledger.owner_id FK 须指向已 seed 的用户行）。"""
|
||||
scoped = Scope(user_id=STUB_OWNER_ID, project_id=None)
|
||||
return req.model_copy(update={"scope": scoped})
|
||||
|
||||
|
||||
def summarize(reports: list[CaseReport]) -> str:
|
||||
"""live 结果的一行式摘要（stdout 用）。"""
|
||||
lines: list[str] = []
|
||||
for r in reports:
|
||||
if r.live is None: # pragma: no cover - live 路径必有结果
|
||||
continue
|
||||
lines.append(
|
||||
f"{r.case.slug}: craft-on {r.live.chars_on}字 / craft-off {r.live.chars_off}字"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
121
tests/test_craft_eval.py
Normal file
121
tests/test_craft_eval.py
Normal file
@@ -0,0 +1,121 @@
|
||||
"""craft A/B 评估回路的 dry 模式确定性断言（灵感 F2）。
|
||||
|
||||
这些断言 = 评估脚本 dry 模式的回归门禁：验证 craft 教条块 / genre 片段 / 开篇 marker
|
||||
**确实按 fixture 注入了正确内容**，且 craft-off 基线**不含**这三者。dry 路径不碰 LLM，
|
||||
故可进常规 pytest / CI（`--live` 真实生成路径不在此、不进 CI，见 tests/craft_eval/README.md）。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from ww_agents import load_craft_doctrine
|
||||
from ww_config import genre_craft_fragment
|
||||
|
||||
from tests.craft_eval import (
|
||||
FIXTURES,
|
||||
FixtureCase,
|
||||
all_passed,
|
||||
build_prompt_pair,
|
||||
input_text,
|
||||
render_report,
|
||||
system_text,
|
||||
verify_case,
|
||||
write_report,
|
||||
)
|
||||
from tests.craft_eval.harness import OPENING_CHAPTER_LIMIT, CaseReport
|
||||
|
||||
_OPENING_HEADER = "开篇黄金三章"
|
||||
|
||||
|
||||
def test_fixtures_cover_three_distinct_genres() -> None:
|
||||
genres = {c.genre for c in FIXTURES}
|
||||
assert {"仙侠", "都市", "悬疑"} <= genres
|
||||
assert len(FIXTURES) == len(genres), "每个 fixture 应是不同题材"
|
||||
for genre in genres:
|
||||
assert genre_craft_fragment(genre), f"题材 {genre!r} 缺 GENRE_CRAFT 片段"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", FIXTURES, ids=lambda c: c.slug)
|
||||
async def test_dry_verification_all_pass(case: FixtureCase) -> None:
|
||||
"""每个金标准 fixture 的 dry 校验全绿——这是 craft 改动的回归门禁本体。"""
|
||||
pair = await build_prompt_pair(case)
|
||||
checks = verify_case(case, pair)
|
||||
failed = [c for c in checks if not c.passed]
|
||||
assert not failed, "dry 校验失败：\n" + "\n".join(f" {c.name}: {c.detail}" for c in failed)
|
||||
assert all_passed(checks)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", FIXTURES, ids=lambda c: c.slug)
|
||||
async def test_craft_on_injects_doctrine_genre_and_marker(case: FixtureCase) -> None:
|
||||
pair = await build_prompt_pair(case)
|
||||
on = pair.craft_on
|
||||
|
||||
# 教条块：第一个 system 块、字节等于 load_craft_doctrine()、cache=True（不变量 #9）。
|
||||
assert on.system[0].text == load_craft_doctrine()
|
||||
assert on.system[0].cache is True
|
||||
assert len(on.system) >= 2, "craft-on 应有 [教条, stable_core] 两块"
|
||||
|
||||
# genre 片段：进 stable_core 缓存块（book-stable → 缓存前缀），不进易变 input（#9）。
|
||||
fragment = genre_craft_fragment(case.genre)
|
||||
assert fragment is not None
|
||||
assert fragment in on.system[1].text
|
||||
assert fragment not in on.system[0].text, "genre 片段不应污染教条块"
|
||||
assert fragment not in input_text(on), "genre 片段是 book-stable，不应进 volatile input"
|
||||
|
||||
# 开篇 marker：开篇章进 volatile input、不进缓存 system（含章号→易变，#9）。
|
||||
if case.chapter_no <= OPENING_CHAPTER_LIMIT:
|
||||
assert _OPENING_HEADER in input_text(on)
|
||||
assert _OPENING_HEADER not in system_text(on)
|
||||
|
||||
# 不变量 #2：只声明 tier，不写 model。
|
||||
assert on.tier == "writer"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case", FIXTURES, ids=lambda c: c.slug)
|
||||
async def test_craft_off_baseline_excludes_craft(case: FixtureCase) -> None:
|
||||
pair = await build_prompt_pair(case)
|
||||
off = pair.craft_off
|
||||
|
||||
assert load_craft_doctrine() not in system_text(off)
|
||||
fragment = genre_craft_fragment(case.genre)
|
||||
assert fragment is not None
|
||||
assert fragment not in system_text(off)
|
||||
assert fragment not in input_text(off)
|
||||
assert _OPENING_HEADER not in input_text(off)
|
||||
# 基线仍是可用写章请求：保留书名与写作指令（stripping 未误伤本体）。
|
||||
assert case.spec.title in system_text(off)
|
||||
assert "请创作第" in input_text(off)
|
||||
|
||||
|
||||
async def test_opening_marker_absent_beyond_opening_limit() -> None:
|
||||
"""章号 > 开篇上限时 craft-on 不注入开篇 marker（黄金三章边界）。"""
|
||||
case = FIXTURES[0]
|
||||
pair = await build_prompt_pair(case, chapter_no=OPENING_CHAPTER_LIMIT + 1)
|
||||
assert _OPENING_HEADER not in input_text(pair.craft_on)
|
||||
assert _OPENING_HEADER not in system_text(pair.craft_on)
|
||||
|
||||
|
||||
async def test_render_and_write_report(tmp_path) -> None: # type: ignore[no-untyped-def]
|
||||
reports: list[CaseReport] = []
|
||||
for case in FIXTURES:
|
||||
pair = await build_prompt_pair(case)
|
||||
reports.append(CaseReport(case=case, pair=pair, checks=verify_case(case, pair)))
|
||||
text = render_report(reports, live=False)
|
||||
assert "craft-on" in text and "craft-off" in text
|
||||
for case in FIXTURES:
|
||||
assert case.slug in text
|
||||
out = write_report(text, tmp_path / "report.md")
|
||||
assert out.exists()
|
||||
assert out.read_text("utf-8") == text
|
||||
|
||||
|
||||
def test_build_prompt_pair_is_deterministic_across_runs() -> None:
|
||||
"""同一 fixture 两次组装字节一致（缓存前缀稳定 / 无时间戳-UUID，#9）。"""
|
||||
import asyncio
|
||||
|
||||
case = FIXTURES[0]
|
||||
a = asyncio.run(build_prompt_pair(case))
|
||||
b = asyncio.run(build_prompt_pair(case))
|
||||
assert system_text(a.craft_on) == system_text(b.craft_on)
|
||||
assert input_text(a.craft_on) == input_text(b.craft_on)
|
||||
assert system_text(a.craft_off) == system_text(b.craft_off)
|
||||
Reference in New Issue
Block a user