fix(mcp): include image source for screenshot results by mushuiyu886 · Pull Request #90902 · openclaw/openclaw

mushuiyu886 · 2026-06-06T10:11:17Z

Summary

Fixes Browser screenshot fails MCP content validation after 2026.6.1 upgrade #90743.
The plugin tools MCP bridge now returns image content in the repository's pinned @modelcontextprotocol/sdk 1.29.0 result schema: { type: "image", data, mimeType }.
Source-shaped image blocks from tool implementations are normalized at the plugin-tools MCP serialization boundary before callTool() returns, while already-valid pinned SDK image blocks are preserved.
Fix classification: Root-cause protocol/data-format boundary fix for MCP tool content serialization.
Why it matters / User impact: Browser screenshot-style plugin tools could emit image content that failed the pinned MCP SDK result validation before the caller could consume the screenshot. Keeping the response in the pinned SDK schema restores screenshot usability on the affected MCP tool path.
What did NOT change: Screenshot capture, browser extension behavior, image data generation, image sanitization, provider adapters, non-image tool content handling, MCP SDK internals, and user configuration are unchanged.
Architecture / source-of-truth check: The source of truth for this repository's installed MCP result schema is the pinned SDK dependency. In @modelcontextprotocol/sdk 1.29.0, CallToolResultSchema rejects source-shaped image blocks and accepts image blocks with top-level data and mimeType; the bridge should not emit the incompatible source shape while the repo is pinned to that contract.
Scope boundary: This PR changes only src/mcp/plugin-tools-handlers.ts and the regression coverage in src/mcp/plugin-tools-serve.test.ts.
Out of scope: No MCP SDK upgrade, no schema migration to the source image shape, no live browser extension session, and no provider image-format changes.

Real behavior proof

Behavior or issue addressed: MCP plugin tool image content could be returned in a shape that the repository's pinned MCP SDK rejects for callTool() results. The failing issue path expects screenshot image content to satisfy the installed SDK schema, but source-shaped image content is invalid under the pinned 1.29.0 CallToolResultSchema.

Real environment tested: Linux local PR worktree at head 94beb3f25779e41a1d8e787da05221544cbe16bd, Node v22.22.0, pnpm 11.2.2, installed @modelcontextprotocol/sdk version 1.29.0. The proof runs the production createToolsMcpServer() MCP server with the SDK Client over InMemoryTransport, calls a screenshot-like plugin tool through the MCP client path, and validates the returned result with the pinned SDK CallToolResultSchema.

Exact steps or command run after this patch:

cd <repo-root>
node --input-type=module --import tsx <<'JS'
import { readFileSync } from "node:fs";
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { InMemoryTransport } from "@modelcontextprotocol/sdk/inMemory.js";
import { CallToolResultSchema } from "@modelcontextprotocol/sdk/types.js";
import { createToolsMcpServer } from "./src/mcp/tools-stdio-server.ts";

const sourceContent = [
  { type: "text", text: "browser screenshot" },
  {
    type: "image",
    source: { type: "base64", media_type: "image/png", data: "iVBORw0KGgo=" },
  },
];

const tool = {
  name: "browser_screenshot",
  description: "Capture a browser screenshot",
  parameters: { type: "object", properties: {} },
  execute: async () => ({ content: sourceContent }),
};

const server = createToolsMcpServer({ name: "openclaw-plugin-tools", tools: [tool] });
const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
const client = new Client(
  { name: "openclaw-mcp-client-proof", version: "1.0.0" },
  { capabilities: {} },
);

await Promise.all([server.connect(serverTransport), client.connect(clientTransport)]);
try {
  const rawSource = { content: sourceContent };
  const raw = CallToolResultSchema.safeParse(rawSource);
  const result = await client.callTool(
    { name: "browser_screenshot", arguments: {} },
    CallToolResultSchema,
  );
  const normalized = CallToolResultSchema.safeParse(result);
  const sdkPkg = JSON.parse(
    readFileSync("node_modules/@modelcontextprotocol/sdk/package.json", "utf8"),
  );
  const proof = {
    pinnedSdkVersion: sdkPkg.version,
    clientCallPath:
      "@modelcontextprotocol/sdk Client -> InMemoryTransport -> createToolsMcpServer -> tool.execute",
    rawSourceAcceptedByPinnedSdk: raw.success,
    clientResultAcceptedByPinnedSdk: normalized.success,
    clientResultContent: result.content,
    rawSourceErrorPaths: raw.success
      ? []
      : raw.error.issues.map((issue) => ({
          path: issue.path.join("."),
          message: issue.message,
        })),
  };
  console.log(JSON.stringify(proof, null, 2));
  if (proof.pinnedSdkVersion !== "1.29.0") {
    throw new Error(`unexpected MCP SDK version ${proof.pinnedSdkVersion}`);
  }
  if (raw.success) {
    throw new Error("raw source-shaped image content unexpectedly passed pinned SDK schema");
  }
  if (!normalized.success) {
    throw new Error("MCP client result did not pass pinned SDK schema");
  }
} finally {
  await client.close();
  await server.close();
}
JS

node scripts/run-vitest.mjs src/mcp/plugin-tools-serve.test.ts
pnpm format:check -- src/mcp/plugin-tools-handlers.ts src/mcp/plugin-tools-serve.test.ts
pnpm tsgo:core
git diff --check origin/main...HEAD

Evidence after fix:

{
  "pinnedSdkVersion": "1.29.0",
  "clientCallPath": "@modelcontextprotocol/sdk Client -> InMemoryTransport -> createToolsMcpServer -> tool.execute",
  "rawSourceAcceptedByPinnedSdk": false,
  "clientResultAcceptedByPinnedSdk": true,
  "clientResultContent": [
    {
      "type": "text",
      "text": "browser screenshot"
    },
    {
      "type": "image",
      "data": "iVBORw0KGgo=",
      "mimeType": "image/png"
    }
  ],
  "rawSourceErrorPaths": [
    {
      "path": "content.1",
      "message": "Invalid input"
    }
  ]
}

MCP regression: src/mcp/plugin-tools-serve.test.ts passed, 8 tests passed.
Format check: all matched files use the correct format.
tsgo:core and git diff --check passed.

Observed result after fix: The raw source image content is rejected by the pinned SDK, while the production MCP server path now returns { type: "image", data, mimeType } to an MCP SDK client and CallToolResultSchema accepts the client result.

What was not tested: I did not run a live Chrome/CDP browser screenshot session or a separate external MCP client process. The added proof exercises the local MCP SDK client/server transport and production plugin-tools MCP server boundary; final acceptance still depends on maintainers accepting the pinned SDK 1.29.0 result contract until an SDK upgrade changes that contract.

Review findings addressed

RF-001 — Status: maintainer decision disclosed with dependency-contract proof. This PR intentionally follows the repository's installed @modelcontextprotocol/sdk 1.29.0 CallToolResultSchema: source-shaped image content is rejected, and top-level { data, mimeType } image content is accepted. I did not attempt a broader MCP image-shape migration because that would require an SDK upgrade decision outside this focused bug fix.
RF-002 — Status: local MCP client proof supplied; live browser proof gap disclosed. The new proof calls the production MCP server through the SDK Client and validates the returned screenshot-like image content. I still did not run a live browser extension screenshot over Chrome/CDP, so that final live-browser acceptance remains a maintainer proof/risk decision rather than something this local source-level update claims to have completed.

Regression Test Plan

Target test file: src/mcp/plugin-tools-serve.test.ts
Scenario locked in: A plugin/browser screenshot-like tool returns mixed text plus a source-shaped image block; createPluginToolsMcpHandlers().callTool() must return text unchanged and normalize the image to { type: "image", data, mimeType }, then the pinned SDK CallToolResultSchema.parse(result) must accept the output.
Why this is the smallest reliable guardrail: The failure happens at the MCP bridge serialization boundary, so the regression test targets that boundary directly and validates against the installed SDK contract instead of asserting a hand-written shape only.
Patch quality notes: The production change is limited to image blocks and leaves non-image content and malformed image-like content behavior unchanged.

Merge risk

Risk labels considered: merge-risk: compatibility, data_format, schema/protocol boundary, size: S.
Risk explanation: The externally visible behavior for this pinned-SDK bridge changes from emitting invalid source image blocks to emitting image blocks accepted by the installed MCP SDK schema. Existing valid { type: "image", data, mimeType } blocks are preserved, non-image content is returned as before, and malformed image-like blocks still fall back to text coercion.
Why acceptable: This matches the repository's current pinned dependency contract and avoids a partial SDK-shape migration. Any future upgrade to a different MCP image shape should happen with the SDK version/schema change as the source of truth.

Root Cause

Root cause: The MCP plugin tools handler could pass through source-shaped image content while the repository is pinned to @modelcontextprotocol/sdk 1.29.0, whose CallToolResultSchema requires image blocks with top-level data and mimeType. That made screenshot-style image content contradict the installed SDK contract at the plugin-tools MCP boundary.
Why this is root-cause fix: The MCP boundary now returns the exact image shape accepted by the installed SDK. It preserves already-valid pinned SDK image blocks and normalizes source-shaped inputs only at the boundary, rather than relying on callers or external clients to accept a schema not provided by the current dependency.

clawsweeper · 2026-06-06T10:13:00Z

Codex review: needs maintainer review before merge. Reviewed June 26, 2026, 7:35 PM ET / 23:35 UTC.

Summary
The PR adds image content normalization to the shared tools MCP handler and tests that source-shaped image results are returned as pinned SDK data/mimeType image blocks.

PR surface: Source +33, Tests +79. Total +112 across 2 files.

Reproducibility: yes. for the MCP boundary: current main passes array content through unchanged, while the pinned SDK 1.29.0 server/client paths validate tool results as top-level data/mimeType image blocks. The exact live Chrome/CDP browser command was not replayed in this read-only review.

Review metrics: 1 noteworthy metric.

MCP result boundary: 1 content-shape normalization added. The PR changes an externally consumed MCP tool result shape, so maintainers should review the dependency-contract decision before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #90743
Summary: This PR is the open candidate fix for the linked browser screenshot MCP content validation issue; broader MCP binary relay work remains related but distinct.

Members:

canonical: Browser screenshot fails MCP content validation after 2026.6.1 upgrade #90743 - The PR has a live closing reference to the issue and targets the reported MCP screenshot content validation failure.
partial_overlap: [Feature]: First-class relay of MCP ImageContent (and other binary Content) to channel extensions (iMessage/BlueBubbles, Telegram, Signal, …) #87023 - That issue covers broader first-class relay of MCP binary content to channel extensions, while this PR targets the narrower tools MCP result schema.
partial_overlap: feat(mcp): stage binary tool content for relay #94663 - That open PR targets broader MCP binary media staging/relay and does not supersede this narrow result-schema fix.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] Merging intentionally changes source-shaped image blocks at the shared tools MCP result boundary to the pinned SDK 1.29.0 top-level data/mimeType shape.
[P1] The full live Chrome/CDP browser screenshot command was not replayed here; review evidence is strongest for the MCP server/client serialization boundary.

Maintainer options:

Land Against Pinned SDK Contract (recommended)
Accept the externally visible result-shape change because it matches the repository's pinned MCP SDK 1.29.0 schema.
Pause For SDK Migration Direction
Pause if maintainers want source-shaped MCP image blocks to wait for an explicit SDK upgrade or broader protocol-shape migration.

Next step before merge

[P2] The remaining action is maintainer acceptance of the pinned SDK result-shape direction, not an automated code repair.

Security
Cleared: The diff adds local content-shape normalization and regression tests without new dependencies, permissions, scripts, secrets handling, or supply-chain surface.

Review details

Best possible solution:

Land this boundary normalization if maintainers accept the pinned MCP SDK 1.29.0 result contract, and handle any future source-shaped image migration with an explicit SDK upgrade.

Do we have a high-confidence way to reproduce the issue?

Yes for the MCP boundary: current main passes array content through unchanged, while the pinned SDK 1.29.0 server/client paths validate tool results as top-level data/mimeType image blocks. The exact live Chrome/CDP browser command was not replayed in this read-only review.

Is this the best way to solve the issue?

Yes if maintainers accept the pinned SDK contract. Normalizing at the tools MCP serialization boundary is narrower than changing screenshot capture, provider adapters, or user configuration, and future source-shaped images should be handled with an explicit SDK upgrade or migration.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 91726e96244a.

Label changes

Label justifications:

P2: The PR targets a normal-priority MCP screenshot/content validation bug with a limited tools MCP bridge blast radius.
merge-risk: 🚨 compatibility: The PR changes the MCP result shape emitted for source-shaped image content at an externally consumed protocol boundary.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal proof using the production MCP server through the SDK Client/InMemoryTransport and validating the after-fix result shape; the later head commit adds test coverage for the same boundary.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal proof using the production MCP server through the SDK Client/InMemoryTransport and validating the after-fix result shape; the later head commit adds test coverage for the same boundary.

Evidence reviewed

PR surface:

Source +33, Tests +79. Total +112 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	34	1	+33
Tests	1	79	0	+79
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	113	1	+112

Acceptance criteria:

[P1] node scripts/run-vitest.mjs src/mcp/plugin-tools-serve.test.ts.
[P1] pnpm format:check -- src/mcp/plugin-tools-handlers.ts src/mcp/plugin-tools-serve.test.ts.
[P1] pnpm tsgo:core.
[P1] git diff --check origin/main...HEAD.

What I checked:

Repository policy read: Root AGENTS.md was read fully; no scoped AGENTS.md owns src/mcp, and the dependency-backed PR review policy was applied. (AGENTS.md:1, 91726e96244a)
Current main behavior: Current main returns array rawContent unchanged from createPluginToolsMcpHandlers().callTool(), so source-shaped image blocks are not normalized before the MCP SDK server validates them. (src/mcp/plugin-tools-handlers.ts:60, 91726e96244a)
MCP server caller: createToolsMcpServer() registers CallToolRequestSchema and delegates tools/call requests directly to handlers.callTool(). (src/mcp/tools-stdio-server.ts:17, 91726e96244a)
Shared MCP surface: openclaw-tools-serve.ts also calls createToolsMcpServer(), so the normalization applies to the shared tools MCP result boundary, not only standalone plugin-tools. (src/mcp/openclaw-tools-serve.ts:12, 91726e96244a)
PR head implementation: The PR maps source: { type: "base64", media_type, data } image blocks to top-level { type: "image", data, mimeType } before returning MCP result content. (src/mcp/plugin-tools-handlers.ts:32, 41a9c845c8cd)
PR regression coverage: The PR adds handler-level schema validation and SDK Client over InMemoryTransport coverage for source-shaped image content delivered through the real MCP server path. (src/mcp/plugin-tools-serve.test.ts:186, 41a9c845c8cd)

Likely related people:

steipete: GitHub path history includes MCP handler docs and earlier handler/test isolation work, and this PR is currently assigned to this user. (role: adjacent MCP bridge contributor; confidence: high; commits: d8326f13c330, e75cd46ba658; files: src/mcp/plugin-tools-handlers.ts, src/mcp/tools-stdio-server.ts, src/mcp/plugin-tools-serve.test.ts)
vincentkoc: GitHub path history shows prior raw plugin tool result serialization and plugin tool policy work in the same MCP bridge area. (role: MCP handler behavior contributor; confidence: high; commits: e4b09e1bf3b2, 5fa0d282a81a, 8f3b99c51281; files: src/mcp/plugin-tools-handlers.ts, src/mcp/plugin-tools-serve.test.ts)
joshavant: GitHub path history shows abort-signal forwarding in the shared MCP stdio server used by this bridge. (role: recent MCP stdio-server contributor; confidence: medium; commits: b7d61c8daffc; files: src/mcp/tools-stdio-server.ts, src/mcp/plugin-tools-handlers.ts)
scotthuang: Browser path history shows recent screenshot vision/media work on the producer side related to screenshot image results. (role: adjacent browser screenshot contributor; confidence: medium; commits: 7920af0c9ec4; files: extensions/browser/src/browser-tool.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

steipete · 2026-06-26T23:39:31Z

Land-ready verification complete.

Confirmed plugin image results normalize once at the plugin-to-MCP boundary to MCP SDK 1.29's verified {data,mimeType} image block; already-valid blocks remain unchanged.
Focused proof: node scripts/run-vitest.mjs src/mcp/plugin-tools-serve.test.ts — 9 tests passed, including an in-memory MCP client/server round trip.
Fresh autoreview: no accepted/actionable findings.
Repository exact-head hosted CI/Testbox gates passed for 41a9c845c8cdbe927f9a6d1f8f5c826df64a80e0.

No known proof gaps.

steipete · 2026-06-26T23:39:44Z

Merged via squash.

Prepared head SHA: 41a9c845c8cdbe927f9a6d1f8f5c826df64a80e0
Landed commit: a846b879ecd9bcb18c39f61e84bceea04b75bba3

* fix(mcp): emit image content with base64 source * fix(mcp): keep plugin tool images in SDK schema * test(mcp): exercise image bridge end to end --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>

openclaw-barnacle Bot added size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 6, 2026

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 6, 2026

mushuiyu886 force-pushed the feat/issue-90743 branch from a1ae503 to 94beb3f Compare June 6, 2026 14:32

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 6, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 6, 2026

clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jun 15, 2026

clawsweeper Bot mentioned this pull request Jun 15, 2026

Browser screenshot fails MCP content validation after 2026.6.1 upgrade #90743

Closed

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 18, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 19, 2026

clawsweeper Bot mentioned this pull request Jun 21, 2026

[Feature]: First-class relay of MCP ImageContent (and other binary Content) to channel extensions (iMessage/BlueBubbles, Telegram, Signal, …) #87023

Open

mushuiyu886 changed the title ~~fix #90743: emit MCP screenshot image source~~ fix(mcp): include image source for screenshot results Jun 23, 2026

steipete self-assigned this Jun 26, 2026

mushuiyu886 and others added 3 commits June 27, 2026 00:29

fix(mcp): emit image content with base64 source

726178a

fix(mcp): keep plugin tool images in SDK schema

a65a9d7

test(mcp): exercise image bridge end to end

41a9c84

steipete force-pushed the feat/issue-90743 branch from d152dd9 to 41a9c84 Compare June 26, 2026 23:29

steipete merged commit a846b87 into openclaw:main Jun 26, 2026
95 checks passed

github-actions Bot mentioned this pull request Jun 27, 2026

📡 Upstream Digest — 2026-06-27 02:29 UTC curtismercier/openclaw-mods#1157

Open

clawsweeper Bot mentioned this pull request Jul 1, 2026

feat(mcp): stage binary tool content for relay #94663

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(mcp): include image source for screenshot results#90902

fix(mcp): include image source for screenshot results#90902
steipete merged 3 commits into
openclaw:mainfrom
mushuiyu886:feat/issue-90743

mushuiyu886 commented Jun 6, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 6, 2026 •

edited

Loading

Uh oh!

steipete commented Jun 26, 2026

Uh oh!

Uh oh!

steipete commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mushuiyu886 commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Real behavior proof

Review findings addressed

Regression Test Plan

Merge risk

Root Cause

Uh oh!

clawsweeper Bot commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jun 26, 2026

Uh oh!

Uh oh!

steipete commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mushuiyu886 commented Jun 6, 2026 •

edited

Loading

clawsweeper Bot commented Jun 6, 2026 •

edited

Loading