a54eb224e0
- SSRF protection: private IP blocking, DNS rebinding defense, redirect validation - OpenAPI fetcher with SSRF guard, JSON/YAML auto-detection, 10MB limit - Structural spec validator (3.0.x/3.1.x) - Endpoint parser with $ref resolution, auto-generated operation IDs - Heuristic + LLM endpoint classifier with Protocol interface - Review API at /api/openapi (import, job status, classification CRUD, approve) - @tool code generator + Agent YAML generator - Import orchestrator (fetch -> validate -> parse -> classify pipeline) - 125 new tests, 322 total passing, 93.23% coverage
85 lines
4.0 KiB
Markdown
85 lines
4.0 KiB
Markdown
# Phase 3: OpenAPI Auto-Discovery -- Development Log
|
|
|
|
> Status: COMPLETED
|
|
> Phase branch: `phase-3/openapi-discovery`
|
|
> Date started: 2026-03-30
|
|
> Date completed: 2026-03-30
|
|
> Related plan section: [Phase 3 in DEVELOPMENT-PLAN](../DEVELOPMENT-PLAN.md#phase-3-openapi-自动发现-第-4-6-周)
|
|
|
|
## What Was Built
|
|
|
|
- SSRF protection module with private IP blocking, DNS rebinding defense, redirect chain validation
|
|
- OpenAPI spec fetcher with SSRF protection, JSON/YAML auto-detection, 10MB size limit
|
|
- Structural OpenAPI spec validator (3.0.x and 3.1.x)
|
|
- Endpoint parser with $ref resolution, parameter extraction, auto-generated operation IDs
|
|
- Heuristic + LLM endpoint classifier with fallback (GET=read, POST/PUT/PATCH/DELETE=write)
|
|
- Review API (FastAPI router at /api/openapi) for import jobs, classification review, approval
|
|
- Tool code generator producing @tool-decorated async functions with httpx
|
|
- Agent YAML generator grouping endpoints by classification
|
|
- Import orchestrator coordinating the full pipeline (fetch -> validate -> parse -> classify)
|
|
- In-memory job store for import state tracking
|
|
|
|
## Code Structure
|
|
|
|
New files created:
|
|
|
|
| File | Purpose | Lines |
|
|
|------|---------|-------|
|
|
| `app/openapi/__init__.py` | Module entry point | 2 |
|
|
| `app/openapi/models.py` | Frozen dataclasses: EndpointInfo, ClassificationResult, ImportJob, etc. | 68 |
|
|
| `app/openapi/ssrf.py` | SSRF protection (validate_url, safe_fetch, DNS resolution) | 162 |
|
|
| `app/openapi/fetcher.py` | SSRF-safe spec fetching with format auto-detection | 94 |
|
|
| `app/openapi/validator.py` | Structural OpenAPI spec validation | 52 |
|
|
| `app/openapi/parser.py` | Endpoint extraction with $ref resolution | 153 |
|
|
| `app/openapi/classifier.py` | HeuristicClassifier + LLMClassifier with Protocol | 164 |
|
|
| `app/openapi/review_api.py` | FastAPI router for import/review workflow | 180 |
|
|
| `app/openapi/generator.py` | @tool code generation + YAML generation | 157 |
|
|
| `app/openapi/importer.py` | Async import pipeline orchestrator | 117 |
|
|
|
|
Modified files:
|
|
- `app/main.py` -- Wired openapi_router
|
|
- `pyproject.toml` -- Added openapi-spec-validator, pytest-httpx dependencies
|
|
|
|
Test files:
|
|
- `tests/unit/test_ssrf.py` (42 tests)
|
|
- `tests/unit/openapi/test_fetcher.py` (7 tests)
|
|
- `tests/unit/openapi/test_validator.py` (8 tests)
|
|
- `tests/unit/openapi/test_parser.py` (10 tests)
|
|
- `tests/unit/openapi/test_classifier.py` (18 tests)
|
|
- `tests/unit/openapi/test_review_api.py` (17 tests)
|
|
- `tests/unit/openapi/test_generator.py` (16 tests)
|
|
- `tests/integration/test_import_pipeline.py` (7 tests)
|
|
|
|
## Test Coverage
|
|
|
|
- Unit tests: 118 new tests across 8 test files
|
|
- Integration tests: 7 new tests for full import pipeline
|
|
- Total: 322 tests passing (125 new + 197 existing)
|
|
- Overall coverage: 93.23% (requirement: 80%)
|
|
|
|
Per-module coverage:
|
|
- classifier.py: 98%
|
|
- fetcher.py: 84%
|
|
- generator.py: 96%
|
|
- importer.py: 100%
|
|
- models.py: 100%
|
|
- parser.py: 89%
|
|
- review_api.py: 100%
|
|
- ssrf.py: 90%
|
|
- validator.py: 88%
|
|
|
|
## Deviations from Plan
|
|
|
|
1. **No custom tool base class (3.0.2 skipped):** Architecture doc explicitly says "do not build custom tool base class." Generated tools use @tool decorator directly.
|
|
2. **Structural validator instead of openapi-spec-validator:** Implemented a lightweight structural validator instead of wrapping the external library. The library is still in dependencies for potential future use.
|
|
3. **In-memory job store:** Used dict-based in-memory store instead of database. Can migrate to PostgreSQL in Phase 5 if needed.
|
|
4. **Frontend Review UI deferred:** ReviewPage.tsx not implemented in this phase; backend API is complete and testable via HTTP.
|
|
|
|
## Known Issues / Tech Debt
|
|
|
|
- Frontend Review UI (3.4.2) deferred -- API is ready, UI needs Phase 5
|
|
- Generated tool code uses string templates -- works for simple REST but may need AST-based generation for complex scenarios
|
|
- LLMClassifier prompt could be tuned with real-world examples
|
|
- No rate limiting on review API endpoints yet
|
|
- openapi-spec-validator library added but not actively used (structural validator is simpler)
|