ColaFlow/docs/mcp-sdk-phase3-runtime-test-report.md

MCP SDK Phase 3: Runtime Test Report

Date: 2025-11-09 Test Environment: Development SDK Version: ModelContextProtocol v0.4.0-preview.3 Application Port: http://localhost:5167 Test Duration: ~30 minutes

---

Executive Summary

The Phase 3 runtime testing revealed that the Microsoft MCP SDK is NOT being used in the current implementation. Instead, ColaFlow has a custom MCP Server implementation that uses HTTP JSON-RPC instead of the SDK's stdio transport.

Key Findings:

✅ What Works:

• Application starts successfully
• Custom MCP endpoint (/mcp) is operational
• MCP Resources are registered (6 resources discovered)
• MCP Protocol Handler is initialized with 6 method handlers
• API Key authentication is functional

❌ What Doesn't Work:

• Microsoft MCP SDK is registered but not actually used
• No stdio transport (required for Claude Desktop integration)
• SDK's .WithToolsFromAssembly() and .WithResourcesFromAssembly() don't integrate with custom implementation

---

1. Application Startup Analysis

Startup Logs (Successful)

[2025-11-09 22:37:10.571] [INF] Initializing MCP Resource Registry with auto-discovery...
[2025-11-09 22:37:10.591] [INF] Starting MCP Resource discovery via Assembly scanning...
[2025-11-09 22:37:10.598] [INF] Discovered 6 MCP Resource types
[2025-11-09 22:37:10.709] [INF] Instantiated 6 MCP Resources
[2025-11-09 22:37:10.710] [INF] Registered MCP Resource: colaflow://issues.get/{id} - Issue Details [Issues]
[2025-11-09 22:37:10.710] [INF] Registered MCP Resource: colaflow://issues.search - Issues Search [Issues]
[2025-11-09 22:37:10.711] [INF] Registered MCP Resource: colaflow://projects.get/{id} - Project Details [Projects]
[2025-11-09 22:37:10.711] [INF] Registered MCP Resource: colaflow://projects.list - Projects List [Projects]
[2025-11-09 22:37:10.711] [INF] Registered MCP Resource: colaflow://sprints.current - Current Sprint [Sprints]
[2025-11-09 22:37:10.711] [INF] Registered MCP Resource: colaflow://users.list - Team Members [Users]
[2025-11-09 22:37:10.711] [INF] MCP Resource Registry initialized: 6 resources in 4 categories

Analysis:

• Application starts successfully
• Custom MCP Resource Registry is initialized (not from SDK)
• 6 MCP Resources discovered via custom assembly scanning
• All database migrations applied successfully
• Application listening on http://localhost:5167

---

2. Endpoint Discovery Results

Test Results:

Endpoint Tested	Method	Result	Status Code	Notes
/health	GET	✅ Success	200	Health check endpoint works
/mcp	POST	⚠️ 401 Unauthorized	401	Requires API Key authentication
/api/mcp	POST	❌ Not Found	404	Endpoint does not exist
/.well-known/mcp	POST	❌ Not Found	404	Endpoint does not exist
/mcp/sse	POST	⚠️ 401 Unauthorized	401	Requires API Key authentication
/mcp/ws	POST	⚠️ 401 Unauthorized	401	Requires API Key authentication

Key Findings:

1. MCP Endpoint: /mcp (HTTP JSON-RPC, not stdio)
2. Authentication: API Key required (Authorization: Bearer <api_key>)
3. MCP Protocol Handler: Custom implementation, not from Microsoft SDK
4. Method Handlers: 6 handlers registered
   • initialize
   • resources/list
   • resources/read
   • resources/health
   • tools/list
   • tools/call

---

3. SDK Integration Analysis

Code Review Findings:

Program.cs (Lines 51-56)

// ============================================
// Register Microsoft MCP SDK (PoC - Phase 1)
// ============================================
builder.Services.AddMcpServer()
    .WithToolsFromAssembly()  // Auto-discover tools with [McpServerToolType] attribute
    .WithResourcesFromAssembly(); // Auto-discover resources with [McpServerResourceType] attribute

Log Evidence (Line 103)

[2025-11-09 22:41:00.442] [INF] MCP Protocol Handler initialized with 6 method handlers:
initialize, resources/list, resources/read, resources/health, tools/list, tools/call

Critical Issue:

The Microsoft MCP SDK is registered in DI but not actually used at runtime. The "MCP Protocol Handler" mentioned in logs is ColaFlow's custom implementation, not the SDK.

Evidence:

1. Custom middleware: McpApiKeyAuthenticationMiddleware (custom, not SDK)
2. Custom protocol handler mentioned in logs
3. HTTP transport (custom) vs SDK's stdio transport
4. Custom resource registry (not SDK's)

---

4. Transport Layer Analysis

Expected (Microsoft SDK): stdio

• What it is: Standard input/output communication
• Use case: Claude Desktop, command-line clients
• Protocol: MCP JSON-RPC over stdio

Actual (ColaFlow Custom): HTTP JSON-RPC

• What it is: HTTP REST API with JSON-RPC messages
• Use case: Web clients, HTTP-based integrations
• Protocol: MCP JSON-RPC over HTTP POST
• Authentication: Custom API Key via Bearer token

Conclusion: The Microsoft MCP SDK's stdio transport is not being used. ColaFlow has built a custom HTTP-based MCP implementation.

---

5. SDK PoC Tools Analysis

SdkPocTools.cs

[McpServerToolType]
public class SdkPocTools
{
    private readonly ILogger<SdkPocTools> _logger;

    public SdkPocTools(ILogger<SdkPocTools> logger)
    {
        _logger = logger;
    }

    [McpServerTool("Ping", "Simple ping tool to verify MCP SDK is working")]
    public string Ping()
    {
        return "Pong! MCP SDK is working.";
    }

    [McpServerTool("GetServerTime", "Get current server time in UTC")]
    public string GetServerTime()
    {
        _logger.LogInformation("GetServerTime called via MCP SDK");
        return $"Server time (UTC): {DateTime.UtcNow:yyyy-MM-dd HH:mm:ss}";
    }

    [McpServerTool("GetProjectInfo", "Get basic project information")]
    public string GetProjectInfo(string projectId)
    {
        return $"Project ID: {projectId} (This is a PoC - actual implementation would query database)";
    }
}

Testing Status:

Tool	Can be Called?	Reason
Ping	❌ No	SDK not integrated with custom MCP handler
GetServerTime	❌ No	SDK not integrated with custom MCP handler
GetProjectInfo	❌ No	SDK not integrated with custom MCP handler

Why it doesn't work:

• The SDK's .WithToolsFromAssembly() discovers tools with [McpServerToolType] attribute
• BUT the custom MCP protocol handler doesn't route to SDK tools
• The custom handler only knows about custom-registered tools

---

6. Dependency Injection Verification

Expected Behavior:

• SdkPocTools should be instantiated by DI
• ILogger<SdkPocTools> should be injected in constructor
• When GetServerTime is called, logger should output: "GetServerTime called via MCP SDK"

Actual Result:

• ❌ Cannot verify - SDK tools are not being called because SDK is not integrated with custom handler
• No log entries found for "GetServerTime called via MCP SDK"
• DI registration is correct, but tools are never invoked

---

7. Error Handling Analysis

Test: Missing API Key

POST /mcp HTTP/1.1
Content-Type: application/json

{"jsonrpc":"2.0","id":1,"method":"ping"}

Response:

{
  "jsonrpc": "2.0",
  "error": {
    "code": -32001,
    "message": "Unauthorized",
    "data": {
      "details": "Missing API Key. Please provide Authorization: Bearer <api_key> header."
    }
  }
}

Status: ✅ Error handling works correctly for authentication failures

---

8. Performance Analysis

Measurements:

Operation	Duration	Notes
Application Startup	~2 seconds	Includes DB migrations
MCP Resource Discovery	~120ms	Custom assembly scanning
Health Check Response	~50ms	First request (cold start)
MCP Endpoint (401)	<1ms	Authentication rejection

Conclusion: Performance is acceptable for custom implementation.

---

9. Critical Issues Discovered

Issue #1: SDK Not Actually Used

Severity: 🔴 Critical Description: Microsoft MCP SDK is registered but not integrated with custom MCP handler

Evidence:

• SDK registration in Program.cs (lines 54-56)
• Custom MCP handler used at runtime (logs show "MCP Protocol Handler")
• No stdio transport (SDK's default)
• SDK tools never called

Impact:

• Cannot test SDK features (stdio transport, tool discovery, resource discovery)
• Phase 1/2/3 testing goals not met
• Need to either: (a) use SDK properly, or (b) remove SDK dependency

Issue #2: HTTP vs stdio Transport Mismatch

Severity: 🟡 Medium Description: SDK expects stdio, but application uses HTTP

Impact:

• Cannot integrate with Claude Desktop (requires stdio)
• SDK's built-in features (stdio server, message framing) not utilized

Issue #3: Tool Discovery Not Working

Severity: 🟡 Medium Description: .WithToolsFromAssembly() discovers tools but they're not exposed via custom handler

Impact:

• SDK PoC tools (Ping, GetServerTime, GetProjectInfo) cannot be tested
• Cannot verify DI injection in tools

---

10. Recommendations

Short-term (Phase 3 Completion):

1. Decision Required: Choose one approach:

   • Option A: Remove Microsoft SDK, document custom implementation
   • Option B: Integrate SDK properly (use stdio transport)
   • Option C: Keep both (SDK for Claude Desktop, custom for web)

2. If keeping SDK (Option B/C):

   • Implement stdio transport using SDK's StdioServer
   • Bridge SDK tools to custom HTTP handler
   • Add configuration to switch between transports

3. Documentation:

   • Update architecture docs to reflect actual implementation
   • Document why custom implementation was chosen over SDK
   • Provide examples for both transports

Long-term (Production):

1. Transport Strategy:

   Client Type          → Transport
   -----------------------------------
   Claude Desktop       → stdio (SDK)
   Web Clients          → HTTP (custom)
   Mobile Apps          → HTTP (custom)
   CLI Tools            → stdio (SDK)
   

2. Unified Tool Registry:

   • Create bridge between SDK tools and custom tools
   • Single source of truth for tool definitions
   • Support both [McpServerTool] (SDK) and custom attributes

3. Testing:

   • Integration tests for SDK tools
   • E2E tests for HTTP MCP endpoint
   • Claude Desktop integration tests (stdio)

---

11. Test Artifacts

Files Created:

• ✅ scripts/test-mcp-runtime.ps1 - HTTP endpoint discovery script
• ✅ scripts/create-test-api-key.sql - SQL script for test API key
• ✅ scripts/create-test-api-key.ps1 - PowerShell script for API key creation

Test Results:

• ✅ scripts/mcp-runtime-test-results.json - Detailed test results

---

12. Answers to Key Questions

Q1: SDK uses what transport layer?

A: SDK is designed for stdio, but application uses custom HTTP transport

Q2: Endpoint URL is what?

A: http://localhost:5167/mcp (custom HTTP endpoint, not SDK)

Q3: How does SDK register endpoints?

A: SDK expects stdio, no HTTP endpoint registration. Custom handler provides HTTP endpoint.

Q4: Are tools and resources correctly discovered?

A:

• Resources: ✅ Yes (custom discovery, 6 resources)
• Tools: ⚠️ Discovered by SDK but not exposed via custom handler

Q5: Does DI work at runtime?

A: ❌ Cannot verify - SDK tools never called because SDK not integrated

Q6: Performance?

A: ✅ Good (startup: 2s, response: <50ms)

---

13. Next Steps

Immediate Action Required:

1. Clarify Architecture Decision:

   • Should we use Microsoft MCP SDK at all?
   • Or document this as "custom MCP implementation"?

2. If Using SDK:

   Phase 3a: Implement stdio transport
   Phase 3b: Bridge SDK tools to custom HTTP handler
   Phase 3c: Test with Claude Desktop
   

3. If Not Using SDK:

   Phase 3a: Remove SDK NuGet package
   Phase 3b: Remove SDK registration from Program.cs
   Phase 3c: Document custom MCP implementation
   

Testing Blocked:

❌ Cannot proceed with Phase 3 testing because:

• SDK tools cannot be called
• No stdio transport to test
• No integration between SDK and custom handler

Recommendation: Make architecture decision before proceeding.

---

Conclusion

The Microsoft MCP SDK (ModelContextProtocol v0.4.0-preview.3) is registered but not functional in the current ColaFlow implementation. The application uses a custom HTTP-based MCP protocol instead of the SDK's stdio-based approach.

Should we use the SDK?

Pros of SDK:

• ✅ Standard MCP protocol support
• ✅ stdio transport for Claude Desktop
• ✅ Built-in tool/resource discovery
• ✅ Microsoft-maintained

Cons of SDK:

• ❌ stdio only (no HTTP)
• ❌ Requires process management
• ❌ Not suitable for web APIs
• ❌ Preview version (unstable)

Recommendation: Keep custom HTTP implementation for web clients, optionally add SDK's stdio transport for Claude Desktop integration. Use a hybrid approach where:

• HTTP endpoint serves web/mobile clients (current implementation)
• stdio endpoint serves Claude Desktop (SDK integration)
• Shared tool/resource registry

---

Report Generated: 2025-11-09 Test Status: Phase 3 Incomplete (awaiting architecture decision) Next Phase: Phase 3a (stdio transport) OR Phase 4 (production implementation)