MCP vs OpenAPI Tool Calling: A Production Benchmark of Latency, Token Cost, and Reliability

Every team shipping a tool-using agent hits the same fork: hand the model a trimmed OpenAPI document and let it construct HTTP calls, or stand up an MCP server and let the agent negotiate capabilities at runtime. Vendor marketing says MCP is the future. Hardened production teams say static schemas are boring and boring is good.

Both camps are partially right. The decision is not really about protocol elegance — it is about token budget per turn, tail latency per tool call, and the blast radius of a schema change. This article walks through a reproducible benchmark harness, the three metrics that actually matter, and the hybrid architecture that beats both extremes.

The Architectural Divide: Static Contracts vs. Runtime Negotiation

Before benchmarking, be precise about what each approach is. They are not peers; they overlap only at the point where a tool definition reaches the model's context window.

How OpenAPI tool calling actually works

You take an OpenAPI 3.1 document, flatten paths into a fixed set of function definitions, and inject them into every LLM request. The model emits JSON arguments; your runtime validates them and issues an HTTP request.

• Stateless. No session, no handshake, no lifecycle. Auth is whatever your API already uses.
• Deterministic context. The tool list is a build-time artifact, byte-identical across turns — which means it participates in prompt caching.
• Fat context. Path templates, parameter locations (query/path/header), security schemes, and 4xx response schemas are all tokens the model never needs.

How MCP actually works

The Model Context Protocol is JSON-RPC 2.0 with a defined lifecycle: initialize (protocol version + capabilities), the notifications/initialized acknowledgement, then tools/list, tools/call, resources/*, prompts/*, and server-initiated notifications such as notifications/tools/list_changed. Transports are stdio (local subprocess) and Streamable HTTP (remote, session via Mcp-Session-Id).

The 2025-06-18 revision added structured tool output via outputSchema + structuredContent, resource links, and elicitation; it also removed JSON-RPC batching. The earlier 2025-03-26 revision introduced Streamable HTTP (replacing HTTP+SSE) and an OAuth 2.1 authorization model.

Critically: MCP does not replace your HTTP API. An MCP server is almost always a thin adapter over the same OpenAPI service. You are choosing an adapter layer, not a backend.

Benchmarking Methodology That Survives Review

A benchmark that only measures the LLM is useless — inference noise swamps protocol differences. Measure only the tool-execution leg, and measure the context injection cost separately.

Our fixture: 30 endpoints (orders, inventory, refunds), 200 iterations per path, single 8-vCPU runner, upstream service on the same VPC with ~180 ms p50 processing time. Three paths:

1. OpenAPI direct — pre-baked tool defs, fetch to the underlying REST service.
2. MCP over stdio — local subprocess, same upstream calls.
3. MCP over Streamable HTTP — remote gateway, session reuse, OAuth bearer token.

Two rules make the numbers trustworthy: amortize the handshake separately from steady-state calls, and report p50/p95/p99, never the mean — tail latency is what pages you at 3 a.m., as Dean and Barroso documented in The Tail at Scale.

The measurement harness

// bench/latency.ts
import { performance } from 'node:perf_hooks';

export interface Sample {
  path: 'openapi' | 'mcp-stdio' | 'mcp-http';
  phase: 'handshake' | 'warm';
  ms: number;
  ok: boolean;
  error?: string;
}

export function percentile(sorted: number[], p: number): number {
  if (sorted.length === 0) return NaN;
  const idx = Math.ceil((p / 100) * sorted.length) - 1;
  return sorted[Math.min(Math.max(idx, 0), sorted.length - 1)];
}

export async function measure<T>(
  path: Sample['path'],
  phase: Sample['phase'],
  fn: () => Promise<T>,
  sink: Sample[],
): Promise<T> {
  const start = performance.now();
  try {
    const result = await fn();
    sink.push({ path, phase, ms: performance.now() - start, ok: true });
    return result;
  } catch (err) {
    sink.push({
      path,
      phase,
      ms: performance.now() - start,
      ok: false,
      error: err instanceof Error ? err.message : String(err),
    });
    throw err;
  }
}

export function report(samples: Sample[]) {
  const groups = new Map<string, number[]>();
  for (const s of samples.filter((x) => x.ok)) {
    const key = `${s.path}/${s.phase}`;
    groups.set(key, [...(groups.get(key) ?? []), s.ms]);
  }
  for (const [key, values] of groups) {
    const sorted = [...values].sort((a, b) => a - b);
    console.table({
      [key]: {
        n: sorted.length,
        p50: percentile(sorted, 50).toFixed(1),
        p95: percentile(sorted, 95).toFixed(1),
        p99: percentile(sorted, 99).toFixed(1),
      },
    });
  }
}

The hard part is the OpenAPI → tool conversion, because a naive converter inflates your token bill by 3–5x. Discard security, servers, 4xx/5xx schemas, and human-facing prose; resolve only local $refs (reject external ones at build time so CI fails loudly instead of at runtime).

// tools/openapi-to-tools.ts
import type { OpenAPIV3_1 } from 'openapi-types';

export interface ToolDef {
  name: string;
  description: string;
  inputSchema: { type: 'object'; properties: Record<string, unknown>; required?: string[] };
}

const VERBS = ['get', 'post', 'put', 'patch', 'delete'] as const;

function resolveRef(doc: OpenAPIV3_1.Document, schema: any, depth = 0): any {
  if (depth > 6) throw new Error('Schema recursion too deep');
  if (schema?.$ref) {
    const m = /^#\/(.+)$/.exec(schema.$ref);
    if (!m) throw new Error(`External $ref unsupported: ${schema.$ref}`);
    const target = m[1].split('/').reduce<any>((acc, k) => acc?.[k], doc);
    if (!target) throw new Error(`Unresolvable $ref: ${schema.$ref}`);
    return resolveRef(doc, target, depth + 1);
  }
  return schema;
}

export function openApiToTools(
  doc: OpenAPIV3_1.Document,
  opts: { maxDescriptionChars?: number } = {},
): ToolDef[] {
  const max = opts.maxDescriptionChars ?? 240;
  const tools: ToolDef[] = [];

  for (const [path, item] of Object.entries(doc.paths ?? {})) {
    for (const verb of VERBS) {
      const op = (item as any)?.[verb] as OpenAPIV3_1.OperationObject | undefined;
      if (!op) continue;

      const properties: Record<string, unknown> = {};
      const required: string[] = [];

      for (const param of [...(item as any).parameters ?? [], ...(op.parameters ?? [])]) {
        const p = resolveRef(doc, param);
        properties[p.name] = { ...p.schema, description: p.description?.slice(0, max) };
        if (p.required) required.push(p.name);
      }

      const body = resolveRef(doc, op.requestBody?.content?.['application/json']?.schema);
      if (body?.properties) {
        Object.assign(properties, body.properties);
        required.push(...(body.required ?? []));
      }

      tools.push({
        name: `${verb}.${path.replace(/\//g, '.').replace(/[{}]/g, '').replace(/\.+/g, '.').replace(/\.$/, '')}`,
        description: (op.summary ?? op.operationId ?? `${verb} ${path}`).slice(0, max),
        inputSchema: { type: 'object', properties, required: [...new Set(required)] },
      });
    }
  }
  return tools;
}

Measuring Token Cost Honestly

Token cost is not a protocol property — it is a property of how much schema you put in the context window, times how many turns you re-send it.

import { getEncoding } from 'js-tiktoken';
import { openApiToTools } from '../tools/openapi-to-tools.js';

const enc = getEncoding('cl100k_base'); // use Anthropic count_tokens for Claude models
const tools = openApiToTools(doc, { maxDescriptionChars: 200 });
const payload = JSON.stringify({ tools });
const toolTokens = enc.encode(payload).length;

// Budget math for 12-turn sessions
const TURNS = 12;
const SESSIONS_PER_DAY = 10_000;
console.log({
  toolTokens,
  perSession: toolTokens * TURNS,
  dailyTokens: toolTokens * TURNS * SESSIONS_PER_DAY,
});

The three-tier cost model

1. Base schema cost. 30 terse tools land around 3k tokens; a naive OpenAPI flattening of the same surface lands near 5.4k — the delta is servers, security, and error schemas the model never reads.
2. Cache behavior. This is the decisive difference. Static OpenAPI tool defs are a byte-stable prefix, so they qualify for prompt caching (Anthropic discounts cached input heavily; OpenAI applies automatic cached-input pricing). MCP's notifications/tools/list_changed can silently invalidate that prefix mid-session — in our runs, a server push forced a full-price re-read of the entire prompt on the next turn. Treat dynamic tool lists as a cache-hostile design unless you version-pin them.
3. Progressive disclosure. Neither raw approach wins here. Anthropic's engineering team documented reducing a tool-definition payload from ~150,000 to ~2,000 tokens (98.7% savings) by exposing MCP tools as code APIs the agent searches rather than injecting every schema. If your tool count exceeds ~20, this dominates every other optimization in this article.

Latency: Where the Milliseconds Actually Go

Decompose the caller's wall clock into handshake, protocol overhead, and upstream work. Upstream dominates; protocol deltas are real but small — until they aren't.

// bench/mcp-client.ts
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';
import { measure, type Sample } from './latency.js';

export async function benchMcp(endpoint: string, token: string, sink: Sample[]) {
  const transport = new StreamableHTTPClientTransport(new URL(endpoint), {
    requestInit: { headers: { Authorization: `Bearer ${token}` } },
  });
  const client = new Client({ name: 'bench-agent', version: '1.0.0' }, { capabilities: {} });

  await measure('mcp-http', 'handshake', () => client.connect(transport), sink);
  const { tools } = await measure('mcp-http', 'handshake', () => client.listTools(), sink);

  for (let i = 0; i < 200; i++) {
    const res = await measure(
      'mcp-http',
      'warm',
      () =>
        client.callTool(
          { name: 'orders.get', arguments: { orderId: 'ord_123' } },
          undefined,
          { timeout: 5_000 }, // never omit: hung servers stall the whole agent loop
        ),
      sink,
    );
    if (res.isError) throw new Error(`Tool-level error: ${JSON.stringify(res.content)}`);
  }
  await client.close();
  return tools.length;
}

Representative results from our fixture (reproduce with the harness; absolute numbers are environment-specific):

Read it this way: stdio MCP is essentially free — a pipe write costs less than a TLS record. Remote MCP costs one extra network hop per call, which shows up disproportionately in p99 because you have added a second service that can cold-start, rate-limit, or drop a connection. The classic tail-amplification effect applies: at 30 tools per session, a 1% slow-path probability per call becomes a near-certain bad session.

Mitigations that actually work: reuse sessions and keep-alive connections, pre-warm remote MCP servers alongside your model endpoint, cache tools/list client-side with a TTL, and wrap every callTool in an AbortSignal.timeout().

Reliability: Error Semantics, Drift, and Blast Radius

The protocol you pick determines whether failures are typed or improvised.

• OpenAPI gives you HTTP status codes and a body your agent must interpret. Nothing tells the model whether 429 is retryable or 409 means "re-read state first." You must encode that in tool descriptions or in a wrapper that maps status → canonical error envelope.
• MCP gives you two distinct failure planes. JSON-RPC errors (-32602 invalid params, -32601 method not found, -32603 internal) are protocol failures. isError: true on a tool result is a domain failure the model can reason about and self-correct. That separation is genuinely better than HTTP semantics for agents, but only if your server honors it — wrapping a 500 as isError instead of a JSON-RPC error is a common and confusing bug.
• Drift. OpenAPI drifts silently: someone adds a required field, your baked schema is stale, and the model emits invalid arguments forever. MCP's tools/list_changed notification is the fix — and the cache-invalidation hazard described above.
• Blast radius. A remote MCP gateway is another multi-tenant service in your critical path. Budget for it like a database: SLOs, circuit breakers, per-tool concurrency limits.

The Hybrid Architecture Most Teams Ship

In practice, mature teams converge on MCP as the agent-facing adapter, OpenAPI as the system of record:

1. Keep the REST service and its OpenAPI spec as the source of truth in CI.
2. Generate the MCP server from that spec (deterministic codegen, not hand-written drift).
3. Expose ≤20 curated tools — not all 30 endpoints — with descriptions written for a model, not a human.
4. Pin the tool list per session to preserve prompt caching; publish changes via a versioned rollup and a tools/list_changed notification at session boundaries only.
5. Run MCP over stdio for co-located agents, Streamable HTTP only for genuinely multi-tenant or cross-network cases.

// gateway/tool-router.ts — one adapter, two front doors
import type { ToolDef } from '../tools/openapi-to-tools.js';

export class ToolRouter {
  constructor(
    private readonly handlers: Record<string, (args: unknown) => Promise<unknown>>,
    private readonly tools: ToolDef[],
    private readonly timeoutMs = 5_000,
  ) {}

  list(): ToolDef[] {
    return this.tools; // stable order => cacheable prefix
  }

  async call(name: string, args: unknown) {
    const handler = this.handlers[name];
    if (!handler) throw Object.assign(new Error(`Unknown tool: ${name}`), { code: -32601 });
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), this.timeoutMs);
    try {
      return { content: [{ type: 'text' as const, text: JSON.stringify(await handler(args)) }] };
    } catch (err) {
      if (controller.signal.aborted) {
        throw Object.assign(new Error(`Tool ${name} timed out`), { code: -32603 });
      }
      // Domain failure: let the model see and recover.
      return {
        isError: true,
        content: [{ type: 'text' as const, text: err instanceof Error ? err.message : 'unknown' }],
      };
    } finally {
      clearTimeout(timer);
    }
  }
}

Production Checklist

• Every callTool and fetch has an explicit timeout and abort signal.
• Tool list is version-pinned; list_changed fires only at session boundaries.
• Tool schemas are generated in CI and diffed; external $refs fail the build.
• Token budget per turn is asserted in tests (fail if tool defs exceed N tokens).
• Map HTTP status → canonical error envelope; map domain errors to isError: true, protocol errors to JSON-RPC codes.
• Track p50/p95/p99 per tool, not per agent.
• Circuit-break on repeated -32603 from a single server.
• Load-test with the tail in mind: 1% slow calls × 30 tools ≈ 26% bad sessions.

Pitfalls to Avoid

• Treating MCP as a backend replacement. It is an adapter. Skipping the OpenAPI source of truth guarantees drift.
• Shipping 50 tools because you can. Tool-count bloat degrades selection accuracy and inflates cost far more than any protocol choice.
• Ignoring cache invalidation. A chatty tools/list_changed will quietly erase your prompt-caching savings.
• Conflating error planes. A 500 is not isError; a bad argument is not -32603.
• Benchmarking the model. Always isolate the execution leg.

Sources & Technical References

• Model Context Protocol Specification, revision 2025-06-18 — https://modelcontextprotocol.io/specification/2025-06-18
• MCP Lifecycle (initialize / capability negotiation) — https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle
• MCP Transports (stdio, Streamable HTTP) — https://modelcontextprotocol.io/specification/2025-06-18/basic/transports
• MCP Server Tools (tools/list, tools/call, outputSchema, isError) — https://modelcontextprotocol.io/specification/2025-06-18/server/tools
• JSON-RPC 2.0 Specification — https://www.jsonrpc.org/specification
• OpenAPI Specification v3.1.0 — https://spec.openapis.org/oas/v3.1.0.html
• RFC 9110: HTTP Semantics (status code semantics) — https://www.rfc-editor.org/rfc/rfc9110.html
• OAuth 2.1 Authorization Framework (draft) — https://datatracker.ietf.org/doc/draft-ietf-oauth-v2-1/
• Anthropic Engineering, "Code execution with MCP: Building more efficient agents" — https://www.anthropic.com/engineering/code-execution-with-mcp
• Anthropic Docs, Prompt Caching — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
• OpenAI Platform Docs, Function Calling — https://platform.openai.com/docs/guides/function-calling
• MCP TypeScript SDK — https://github.com/modelcontextprotocol/typescript-sdk
• js-tiktoken (BPE tokenizer for JS) — https://github.com/dqbd/tiktoken
• MDN, AbortSignal.timeout() — https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal/timeout_static
• Dean & Barroso, "The Tail at Scale," Communications of the ACM, 2013 — https://research.google/pubs/pub40801/