Skip to content

fix(edit): keep mismatch hint truncation unicode-safe#99527

Merged
steipete merged 2 commits into
openclaw:mainfrom
mikasa0818:feat/issue-97069
Jul 4, 2026
Merged

fix(edit): keep mismatch hint truncation unicode-safe#99527
steipete merged 2 commits into
openclaw:mainfrom
mikasa0818:feat/issue-97069

Conversation

@mikasa0818

@mikasa0818 mikasa0818 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes the edit-tool mismatch diagnostic truncation path so the Current file contents: hint cannot split an emoji or other astral Unicode character at the 800-character cutoff.

  • Why it matters / User impact: When an edit replacement cannot find its exact oldText, the diagnostic should show trustworthy current-file context. A broken replacement character at the truncation boundary makes the hint harder to use when the file contains non-BMP Unicode near the cutoff.
  • Maintainer-ready confidence: High for this focused diagnostic fix. The patch touches only the edit-tool mismatch hint path and is covered by a live edit-tool invocation, a targeted regression test, and changed-file core checks.
  • What did NOT change: Edit matching, replacement semantics, file writes, newline normalization, post-write recovery, preview rendering, approval flows, gateway protocol, UI approval surfaces, generated Swift models, and plugin/native channel behavior are unchanged.

Real behavior proof

  • Behavior or issue addressed: Edit-tool exact-match failures append Current file contents: to the thrown mismatch error. Before this change, long current-file snippets were truncated with raw String.prototype.slice(0, 800), which can split a UTF-16 surrogate pair when the cutoff lands inside an emoji or other astral Unicode character.
  • Real environment tested: Windows 10 Enterprise LTSC 2019 via Git Bash, Node v22.17.0, pnpm 11.2.2.
  • Exact steps or command run after this patch: From the repository root, I ran a non-test live edit-tool invocation with node --import tsx. The script imported src/agents/sessions/tools/edit.ts, created a temporary file containing 799 ASCII characters, one 🙂 emoji, and trailing text, then called createEditTool(...).execute(...) with a missing oldText to trigger the real mismatch diagnostic path. I also ran the focused regression test and changed-file checks:
node --import tsx <live edit-tool invocation script>
node scripts/run-vitest.mjs src/agents/sessions/tools/edit.test.ts
git diff --check
OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 CI=1 PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN=false pnpm check:changed
  • Evidence after fix: The live edit-tool invocation produced this copied terminal output, with the temp path redacted:
temp file: <temp-dir>\unicode-boundary.txt
message includes exact mismatch header: true
message includes intact emoji at boundary: true
message includes replacement character: false
snippet tail:
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa🙂
... (truncated)

LIVE_PROOF_EXIT:0

The focused edit-tool test suite exited 0. git diff --check exited 0. The local changed-file check exited 0 and selected the core/core-test lanes for src/agents/sessions/tools/edit.ts and src/agents/sessions/tools/edit.test.ts.

  • Observed result after fix: The live mismatch diagnostic includes the exact mismatch header, keeps the full boundary emoji before ... (truncated), and does not contain the Unicode replacement character. The regression test verifies the same boundary case through createEditTool(...).execute(...), so the hint no longer relies on raw .slice() truncation at the UTF-16 boundary.
  • What was not tested: No live external channel, Telegram, Matrix, iMessage, Control UI, gateway, Mantis, or Desktop proof was run because the current patch does not touch those paths. A plain pnpm check:changed invocation attempted to delegate to Blacksmith Testbox through crabbox, but this host's crabbox binary failed its basic sanity check before running repo checks. The local child-mode command above was used for the actual changed-file verification.
  • Fix classification: Root cause fix.

What Problem This Solves

When an edit replacement cannot find its exact oldText, the edit tool appends Current file contents: to help the model/user see what actually exists in the file. For long files this diagnostic is capped at EDIT_MISMATCH_HINT_LIMIT characters.

Before this change, the cap used raw JavaScript string slicing. JavaScript slice() counts UTF-16 code units, not displayed Unicode characters/code points. If an emoji or other astral character straddled the 800-code-unit boundary, the hint could include only half of the surrogate pair. That produces a malformed diagnostic snippet at the exact point where the tool is trying to provide useful mismatch context.

Root Cause

  • Root cause: The edit mismatch diagnostic source of truth is appendMismatchHint(), and that formatter truncated currentContent with currentContent.slice(0, EDIT_MISMATCH_HINT_LIMIT). That operation counts UTF-16 code units instead of whole Unicode code points, so the diagnostic owner could create an invalid snippet by cutting between the high and low surrogate of a non-BMP character.
  • Why this is root-cause fix: The fix changes the truncation primitive at the diagnostic source of truth, before the mismatch hint string is assembled. Iterating with for...of makes the formatter count whole Unicode code points, so the invariant becomes: the edit mismatch hint may be shortened, but it never creates a malformed half-character at the truncation boundary. This fixes the source formatter rather than masking replacement characters after the fact.
  • Architecture / source-of-truth check: The source-of-truth boundary is the private edit-tool mismatch hint formatter in src/agents/sessions/tools/edit.ts. No public contract, schema, protocol, config, generated model, or API surface is changed; compatibility and default behavior for edit execution are unchanged. A related open PR / same-contract scan is not relevant because this patch does not define or expand a public contract surface. Out of scope: changing edit matching, replacement, persistence, preview rendering, gateway behavior, approval UI, or any protocol field.

Changes

  • Added a small edit-tool-local mismatch snippet helper that iterates currentContent with for...of, so the cutoff is counted by Unicode code point instead of raw UTF-16 code unit.
  • Preserved the existing mismatch hint limit and ... (truncated) suffix behavior.
  • Added a regression test where the 800th displayed character is an emoji, proving the hint keeps the emoji intact before appending the truncation marker.

Review findings addressed from the previous PR content

This PR previously carried a different approval-prompt/protocol/UI change. That old diff has been removed from the branch and replaced with the focused edit-tool fix above.

  • RF-001 / RF-002 / RF-004 / RF-006: old requests for real Telegram, Control UI, gateway, or approval prompt proof no longer apply to the current diff because the approval-flow changes were removed.
  • RF-003: old public-ish approval contract compatibility concern no longer applies; the current diff does not touch gateway protocol, generated Swift models, plugin SDK reply parameters, or approval metadata.
  • RF-005: the previous draft/readiness gate was for the old approval PR content. Current readiness should be judged from the new edit-tool diff and verification below.
  • RF-007: old UI i18n concern no longer applies; the current diff does not add or change approval UI warnings.

Evidence

The live proof and regression test both covered the real edit-tool mismatch error path through createEditTool(...).execute(...), not only a standalone helper. The boundary file contains 799 ASCII characters followed by an emoji and trailing text, then triggers an exact-match miss. The observed diagnostic includes the intact emoji and the existing truncation marker.

Changed-file verification also covered:

conflict markers
changelog attributions
guarded extension wildcard re-exports
plugin-sdk wildcard re-exports
duplicate scan target coverage
dependency pin guard
package patch guard
test temp creation report
typecheck core
typecheck core tests
lint core changed files
database-first legacy-store guard
media download helper guard
runtime sidecar loader guard
runtime import cycles
webhook body guard
pairing store guard
pairing account guard

All selected local changed-file lanes exited 0.

Regression Test Plan

  • Target test file: src/agents/sessions/tools/edit.test.ts.
  • Scenario locked in: A mismatch hint whose truncation boundary lands after 799 ASCII characters and one emoji.
  • Why this is the smallest reliable guardrail: The test exercises the public edit-tool execution path, so it verifies the actual error message users/models see without exporting a formatting helper or adding broader fixtures.
  • Expected behavior: the thrown mismatch error contains the full emoji followed by ... (truncated), not a sliced surrogate half or replacement character.
  • Existing mismatch behavior remains covered by the existing exact-match diagnostic test.

Merge risk

  • Risk labels considered: No merge-risk:* label should be required for the current diff. The old approval-protocol/UI compatibility risk no longer applies because that diff was removed.
  • Risk explanation: The change is isolated to diagnostic formatting for edit-tool mismatch errors and keeps the same limit/suffix contract. It does not change edit application, matching, persistence, protocol, UI, or approval behavior.
  • Why acceptable: Counting the truncation limit by Unicode code point prevents malformed diagnostics while preserving bounded output and the existing truncation marker. The only behavior difference is that astral Unicode characters are kept whole at the boundary.
  • Patch quality notes: This is not a fallback or downstream string special-case. It fixes the truncation primitive at the mismatch hint source, keeps the helper private to the edit tool, and adds a regression test for the exact boundary failure mode.

@openclaw-barnacle openclaw-barnacle Bot added channel: imessage Channel integration: imessage channel: matrix Channel integration: matrix channel: telegram Channel integration: telegram app: web-ui App: web-ui gateway Gateway runtime agents Agent runtime and tooling size: L labels Jul 3, 2026
@clawsweeper

clawsweeper Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 4, 2026, 5:16 AM ET / 09:16 UTC.

Summary
This PR updates edit mismatch diagnostics to use truncateUtf16Safe and adds a regression test for a surrogate-pair truncation boundary.

PR surface: Source +1, Tests +17. Total +18 across 2 files.

Reproducibility: yes. from source inspection. Current main and v2026.6.11 slice the mismatch hint at a UTF-16 code-unit boundary, and the PR’s regression case reaches that path through createEditTool(...).execute(...); I did not run tests locally because this review is read-only.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/agents/sessions/tools/edit.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Next step before merge

  • No ClawSweeper repair lane is needed; the branch already contains the focused fix and proof, with only normal maintainer review and merge checks remaining.

Security
Cleared: The diff only changes an internal edit-tool diagnostic and a focused test, with no dependency, workflow, credential, install, package, or supply-chain surface change.

Review details

Best possible solution:

Merge the focused helper reuse and regression test after maintainer review and normal exact-head required checks complete.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection. Current main and v2026.6.11 slice the mismatch hint at a UTF-16 code-unit boundary, and the PR’s regression case reaches that path through createEditTool(...).execute(...); I did not run tests locally because this review is read-only.

Is this the best way to solve the issue?

Yes. Reusing the dependency-free truncateUtf16Safe helper in appendMismatchHint is the narrowest maintainable fix, preserves the 800 UTF-16-unit diagnostic bound, and avoids the duplicate code-point loop from the earlier branch state.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against be95bb72d459.

Label changes

Label justifications:

  • P3: The PR fixes a bounded diagnostic-quality issue in edit-tool error text without changing edit matching, writes, provider routing, config, or persisted state.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied live output from a production-path createEditTool(...).execute(...) invocation showing the boundary emoji omitted whole and no dangling surrogate.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied live output from a production-path createEditTool(...).execute(...) invocation showing the boundary emoji omitted whole and no dangling surrogate.
Evidence reviewed

PR surface:

Source +1, Tests +17. Total +18 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 2 1 +1
Tests 1 17 0 +17
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 19 1 +18

What I checked:

  • Repository policy read: Root AGENTS.md and scoped src/agents/AGENTS.md were read fully; their best-fix, scoped-source, and proof guidance shaped the review. (AGENTS.md:1, be95bb72d459)
  • Current main still has raw slicing: appendMismatchHint on current main builds the over-limit hint with currentContent.slice(0, EDIT_MISMATCH_HINT_LIMIT), so the PR is not redundant with main. (src/agents/sessions/tools/edit.ts:180, be95bb72d459)
  • Latest release has the same shipped behavior: The latest release tag v2026.6.11 also uses raw slicing for this mismatch hint, so the malformed-boundary behavior is shipped. (src/agents/sessions/tools/edit.ts:169, e085fa1a3ffd)
  • PR head reuses the shared helper: The current PR head imports truncateUtf16Safe and applies it only to the over-limit mismatch hint branch, preserving the existing limit branch for shorter content. (src/agents/sessions/tools/edit.ts:178, b41df44ac58e)
  • Helper contract matches the fix: truncateUtf16Safe delegates to sliceUtf16Safe(input, 0, limit), whose boundary logic drops a high surrogate when its low surrogate would fall just outside the limit. (src/shared/utf16-slice.ts:46, be95bb72d459)
  • Runtime path reaches the changed helper: createEditToolDefinition(...).execute(...) catches exact-match mismatch errors and calls appendMismatchHint, so the changed code is on the production edit-tool path. (src/agents/sessions/tools/edit.ts:492, be95bb72d459)

Likely related people:

  • mbelinky: The mismatch-hint recovery behavior appears to date to the edit-tool recovery work in merged pull request fix(agents): harden edit tool recovery #52516. (role: feature-history owner; confidence: medium; commits: 922f4e66ea1a; files: src/agents/pi-tools.host-edit.ts, src/agents/pi-tools.read.host-edit-recovery.test.ts, src/agents/sessions/tools/edit.ts)
  • steipete: Carried the edit recovery surface into the current session tool during the agent-runtime refactor and authored the current PR head follow-up preserving the size bound. (role: recent area contributor; confidence: high; commits: bb46b79d3c14, b41df44ac58e; files: src/agents/sessions/tools/edit.ts, src/agents/sessions/tools/edit.test.ts)
  • Bartok9: Introduced the shared dependency-free UTF-16 safe slicing helper that this PR reuses. (role: helper introducer; confidence: high; commits: e09b9dfc1ba9; files: src/shared/utf16-slice.ts, src/shared/utf16-slice.test.ts, src/agents/tool-display-common.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (4 earlier review cycles)
  • reviewed 2026-07-04T06:37:56.141Z sha 0495dcb :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T08:55:10.726Z sha 81e2d37 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T09:01:03.104Z sha 81e2d37 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-04T09:07:37.322Z sha 99c03e1 :: needs maintainer review before merge. :: none

@mikasa0818 mikasa0818 closed this Jul 3, 2026
@mikasa0818 mikasa0818 reopened this Jul 3, 2026
@mikasa0818
mikasa0818 marked this pull request as draft July 3, 2026 12:39
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jul 3, 2026
@openclaw-barnacle openclaw-barnacle Bot added size: XS and removed channel: imessage Channel integration: imessage channel: matrix Channel integration: matrix channel: telegram Channel integration: telegram app: web-ui App: web-ui gateway Gateway runtime size: L labels Jul 3, 2026
@mikasa0818 mikasa0818 changed the title fix #97069: [Bug]: Approval prompt wrongly says effective policy requires approval every time for non-persistable redirected command fix(edit): keep mismatch hint truncation unicode-safe Jul 3, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. P2 Normal backlog priority with limited blast radius. labels Jul 3, 2026
@clawsweeper clawsweeper Bot added P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. and removed merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jul 3, 2026
@mikasa0818 mikasa0818 closed this Jul 3, 2026
@mikasa0818
mikasa0818 deleted the feat/issue-97069 branch July 3, 2026 15:40
@mikasa0818
mikasa0818 restored the feat/issue-97069 branch July 3, 2026 15:58
@mikasa0818 mikasa0818 reopened this Jul 3, 2026
@mikasa0818
mikasa0818 marked this pull request as ready for review July 3, 2026 16:07
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@steipete steipete self-assigned this Jul 4, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@steipete

steipete commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Land-ready maintainer proof for final rebased head b41df44ac58e86ff687a63bb69c231d7ef1a114b:

  • Replaced the PR-local code-point loop with the canonical truncateUtf16Safe helper, preserving the established 800 UTF-16-unit bound and deleting 18 lines of duplicate logic.
  • node scripts/run-vitest.mjs src/agents/sessions/tools/edit.test.ts — 20/20 passed after the final rebase.
  • Live production-path invocation through createEditTool(...).execute(...)mismatchHeader=true, 799 UTF-16 units before the marker, boundary emoji omitted whole, no dangling surrogate.
  • Fresh auto-review after the final rebase — no findings, confidence 0.99.
  • Final exact-head CI run 28701557955 — all selected build, lint, type, security, boundary, QA smoke, and Node test shards passed.
  • scripts/pr prepare-run 99527 — hosted exact-head gates passed; remote tree verified against the prepared head.

Proof gap: no external channel, Gateway, or UI run; those surfaces are untouched. The live invocation exercises the actual edit-tool mismatch path, not a standalone helper.

@steipete
steipete force-pushed the feat/issue-97069 branch from 81e2d37 to 99c03e1 Compare July 4, 2026 09:01
@steipete
steipete force-pushed the feat/issue-97069 branch from 99c03e1 to b41df44 Compare July 4, 2026 09:11
@steipete
steipete merged commit f27d494 into openclaw:main Jul 4, 2026
96 checks passed
@steipete

steipete commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 5, 2026
* fix(edit): keep mismatch hint truncation unicode-safe

* fix(edit): preserve mismatch hint size bound

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants