fix(markdown): a fenced-code line with trailing text is content, not a closing fence by ly-wang19 · Pull Request #96745 · openclaw/openclaw

ly-wang19 · 2026-06-25T10:59:34Z

What Problem This Solves

scanFenceSpans/parseFenceSpans (packages/markdown-core/src/fences.ts) maps where fenced code blocks start and end so chunkers never split inside one. It accepted any line that begins with ≥3 matching fence markers as a closing fence, ignoring trailing text after the marker. Per CommonMark a closing fence may be followed only by whitespace, so a code-content line like ``` not a close was wrongly treated as a close: the block ended early, the following lines were reported as outside any fence, and the trailing real ``` became a new unclosed opener.

The consequence: isSafeFenceBreak() returns true for offsets inside the actual code block and findFenceSpanAt() returns undefined, so chunkMarkdownText and the embedded-agent block chunker can split inside a fenced code block — the exact thing this module exists to prevent. Code blocks that demonstrate markdown/fences (a line starting with ``` followed by text) are ordinary content.

Why This Change Was Made

The closing branch already checked the marker character and length (its comment cites CommonMark) but missed the trailing-text rule. The fix adds that one condition: a closing fence's trailing text must be whitespace-only. The reference implementation (micromark) confirms the sample input is a single code block. This makes fence detection stricter — the safe direction (it keeps more text inside a fence, so a chunker is less likely to split inside code).

User Impact

Outbound/agent text containing a fenced code block whose body has a ```-prefixed line is no longer mis-parsed, so chunkers won't break the code block at that point. Opening info strings, bare closes, and longer same-marker closes are unaffected. No API change.

Evidence

oxfmt, oxlint, tsgo:core pass. A new colocated fences.test.ts regression test passes and fails when the fix is reverted (fail-before). Both downstream suites — src/auto-reply/chunk.test.ts and src/agents/embedded-agent-block-chunker.test.ts — stay green.

Real behavior proof

Behavior addressed: a fenced-code content line beginning with the fence marker but carrying trailing text (``` not a close) was treated as a closing fence, so isSafeFenceBreak reported an offset inside the code block as a safe chunk break, letting chunkers split inside the block.

Real environment tested: Ran the exported parseFenceSpans / isSafeFenceBreak / findFenceSpanAt from packages/markdown-core/src/fences.ts via tsx on Node v22.22.1 (no mocks), before and after the patch, plus the colocated Vitest test and the two downstream chunker suites via node scripts/run-vitest.mjs.

Exact steps or command run after this patch: parseFenceSpans("```\\ncode\\n``` not a close\\nmore code\\n```\\n") and isSafeFenceBreak(spans, <offset of "more code">); then node scripts/run-vitest.mjs packages/markdown-core/src/fences.test.ts src/auto-reply/chunk.test.ts src/agents/embedded-agent-block-chunker.test.ts, node scripts/run-oxlint.mjs, pnpm tsgo:core.

Evidence after fix:

BEFORE (origin/main):
  parseFenceSpans(...)          -> [{0,24},{35,39}]  (block closed early; "more code" outside any fence)
  isSafeFenceBreak(spans, 27)   -> true   (WRONG: 27 = 'r' in "more code", inside the real code block)
  findFenceSpanAt(spans, 27)    -> undefined  (WRONG)
AFTER:
  parseFenceSpans(...)          -> [{0,38}]   (one code block, as micromark renders it)
  isSafeFenceBreak(spans, 27)   -> false
  findFenceSpanAt(spans, 27)    -> the enclosing span
  control "```\ncode\n```\nafter" -> still one span; "after" is a safe break (true)
fences.test.ts -> exit 0 ; fail-before (revert source) -> the new test fails
chunk.test.ts -> exit 0 ; embedded-agent-block-chunker.test.ts -> exit 0 ; oxlint -> 0 ; tsgo:core -> 0

Observed result after fix: a marker line with trailing text stays inside the code block (one span), interior offsets are no longer reported as safe breaks, and valid closes (bare, longer same-marker, with trailing whitespace) plus opening info strings still behave correctly; the downstream chunkers are unchanged on their existing tests.

What was not tested: a full end-to-end channel render of a chunked message containing such a code block (no live channel in this environment); the fix is a pure parser change proven at the function and unit-test level, cross-checked against the CommonMark reference (micromark) for the single-code-block interpretation.

clawsweeper · 2026-06-25T11:00:36Z

Codex review: needs maintainer review before merge. Reviewed June 26, 2026, 5:27 PM ET / 21:27 UTC.

Summary
The PR tightens shared markdown fence scanning so only same-marker lines with space/tab-only suffixes close a fence, strips trailing CR during scanning, and adds markdown-core regression tests.

PR surface: Source +4, Tests +45. Total +49 across 2 files.

Reproducibility: yes. Source inspection on current main shows the close branch ignores trailing text after a matching marker, and the PR body includes before/after exported-function output for the failing parser case.

Review metrics: none identified.

Root-cause cluster
Relationship: canonical
Canonical: #96745
Summary: This PR is the canonical open landing target for the markdown fence trailing-text scanner fix.

Members:

duplicate: fix(markdown): a fenced-code line with trailing text is content, not a closing fence #96953 - It targets the same scanner bug with the same two-file fix shape, and its ClawSweeper review already identifies this PR as the canonical landing target.
superseded: fix(markdown): a fenced-code line with trailing text is content, not a closing fence #96687 - It carried the same same-author fix first but is closed unmerged with the too-many-PRs label; this open PR preserves and updates that work.
superseded: fix(markdown): reject closing code fences with content after the marker #61300 - It was an older closed-unmerged attempt at the same closing-fence trailing-text rule and CRLF handling.
adjacent_distinct: fix(discord): reserve closing-fence space on fence-closing lines #95661 - It fixed a separate Discord chunk-size bug involving fence-closing lines, but not the shared markdown-core parser root cause handled here.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Next step before merge

No automated repair is needed; the remaining action is maintainer review/landing of the canonical PR and duplicate cleanup.

Security
Cleared: The diff only changes markdown parser logic and colocated tests; it does not touch dependencies, workflows, lockfiles, secrets, install scripts, package metadata, or external code execution.

Review details

Best possible solution:

Land this canonical shared-scanner fix after maintainer review, then close the duplicate salvage PR as redundant.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection on current main shows the close branch ignores trailing text after a matching marker, and the PR body includes before/after exported-function output for the failing parser case.

Is this the best way to solve the issue?

Yes. Fixing scanFenceSpans is the best owner-boundary fix because the downstream chunkers and scanners derive fence safety from that shared parser.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 29680046801d.

Label changes

Label justifications:

P2: This is a real markdown chunk-safety bug with limited blast radius and no data-loss, security, crash-loop, or urgent channel outage signal.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied before/after live output from exported parser functions in Node, plus fail-before regression and downstream test commands, which is sufficient for this pure parser change.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied before/after live output from exported parser functions in Node, plus fail-before regression and downstream test commands, which is sufficient for this pure parser change.

Evidence reviewed

PR surface:

Source +4, Tests +45. Total +49 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	8	4	+4
Tests	1	45	0	+45
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	53	4	+49

What I checked:

Current main scanner behavior: Current main captures the trailing suffix in match[3] but closes solely on same marker character and sufficient marker length, so trailing text is ignored by the close predicate. (packages/markdown-core/src/fences.ts:61, 29680046801d)
PR implementation: The PR head strips a trailing CR before matching and requires the suffix after the closing marker to match only spaces or tabs before closing the fence. (packages/markdown-core/src/fences.ts:44, 7104845b0a43)
Regression coverage: The added tests cover trailing text as content, non-space/tab suffixes, CRLF closing lines, bare closes, longer same-marker closes, and opener info strings. (packages/markdown-core/src/fences.test.ts:6, 7104845b0a43)
Auto-reply caller impact: chunkMarkdownText parses fence spans once and uses isSafeFenceBreak/findFenceSpanAt to choose safe markdown chunk boundaries, so an early false close can expose an unsafe split point. (src/auto-reply/chunk.ts:392, 29680046801d)
Embedded chunker caller impact: EmbeddedBlockChunker also derives paragraph, newline, whitespace, and forced split decisions from parseFenceSpans, confirming the shared scanner is the right owner boundary. (src/agents/embedded-agent-block-chunker.ts:166, 29680046801d)
Spec contract check: CommonMark 0.31.2 says a closing code fence may be followed only by spaces or tabs, matching the PR's stricter suffix rule.

Likely related people:

849261680: Git blame on the current scanner and key chunker caller lines maps to merge commit 7fefc5f from PR 96096; that broad merge carried the current snapshot, so this is a routing signal rather than sole introduction proof. (role: current-main history signal; confidence: medium; commits: 7fefc5ff58de; files: packages/markdown-core/src/fences.ts, src/auto-reply/chunk.ts, src/agents/embedded-agent-block-chunker.ts)
velvet-shark: Live PR metadata for fix: cron stream stalls fail over before job timeout #96096 lists this handle as merger for the current-main snapshot containing the scanner and caller paths. (role: recent merger; confidence: medium; commits: 7fefc5ff58de; files: packages/markdown-core/src/fences.ts, src/auto-reply/chunk.ts, src/agents/embedded-agent-block-chunker.ts)
steipete: Git history shows earlier markdown fence chunking work and quadratic-scan hardening in the shared chunking path under this author. (role: feature-history owner; confidence: medium; commits: 67bda21811b3, 74912037dcbc; files: src/auto-reply/chunk.ts)
vincentkoc: The latest PR-head commit refined the suffix rule to spaces/tabs only and added CRLF/non-space whitespace coverage; the live PR is also assigned to this handle. (role: recent PR-head contributor and assignee; confidence: high; commits: 7104845b0a43; files: packages/markdown-core/src/fences.ts, packages/markdown-core/src/fences.test.ts)
ly-wang19: Beyond authoring this PR, this handle recently landed a related Discord fenced-code chunking fix, making them a credible follow-up owner for this parser/chunking behavior. (role: adjacent chunking contributor; confidence: medium; commits: d84a8b15063a; files: extensions/discord/src/chunk.ts, extensions/discord/src/chunk.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

…a closing fence scanFenceSpans accepted any line starting with >=3 matching fence markers as a closing fence, ignoring trailing text after the marker. Per CommonMark a closing fence may be followed only by whitespace, so a code-content line such as "``` not a close" was wrongly treated as a close: the block ended early, the following lines were reported as outside any fence, and the trailing marker line became a new unclosed opener. That made isSafeFenceBreak() return true for offsets inside the real code block and findFenceSpanAt() return undefined, so chunkers (chunkMarkdownText, the embedded-agent block chunker) could split inside a fenced code block — the exact thing this module exists to prevent. Require the closing fence's trailing text to be whitespace-only. Opening info strings, bare closes, and longer same-marker closes are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>

vincentkoc · 2026-06-26T21:32:16Z

Merged via squash.

Prepared head SHA: 7104845b0a43b8f37801d1361169133a10c3284c
Landed commit: 1b6557dfa25019f70bff53f27eb9b9d0b0d5e25f

…a closing fence (openclaw#96745) * fix(markdown): a fenced-code line with trailing text is content, not a closing fence scanFenceSpans accepted any line starting with >=3 matching fence markers as a closing fence, ignoring trailing text after the marker. Per CommonMark a closing fence may be followed only by whitespace, so a code-content line such as "``` not a close" was wrongly treated as a close: the block ended early, the following lines were reported as outside any fence, and the trailing marker line became a new unclosed opener. That made isSafeFenceBreak() return true for offsets inside the real code block and findFenceSpanAt() return undefined, so chunkers (chunkMarkdownText, the embedded-agent block chunker) could split inside a fenced code block — the exact thing this module exists to prevent. Require the closing fence's trailing text to be whitespace-only. Opening info strings, bare closes, and longer same-marker closes are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(markdown): honor fence suffix whitespace rules Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> --------- Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Vincent Koc <vincentkoc@ieee.org>

openclaw-barnacle Bot added the size: XS label Jun 25, 2026

clawsweeper Bot mentioned this pull request Jun 26, 2026

fix(markdown): a fenced-code line with trailing text is content, not a closing fence #96953

Closed

openclaw-barnacle Bot added size: S and removed size: XS labels Jun 26, 2026

vincentkoc self-assigned this Jun 26, 2026

ly-wang19 and others added 2 commits June 26, 2026 14:22

fix(markdown): honor fence suffix whitespace rules

7104845

Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>

vincentkoc force-pushed the fix/fence-close-trailing-text branch from f63d668 to 7104845 Compare June 26, 2026 21:22

vincentkoc merged commit 1b6557d into openclaw:main Jun 26, 2026
77 checks passed

github-actions Bot mentioned this pull request Jun 26, 2026

📡 Upstream Digest — 2026-06-26 22:59 UTC curtismercier/openclaw-mods#1156

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(markdown): a fenced-code line with trailing text is content, not a closing fence#96745

fix(markdown): a fenced-code line with trailing text is content, not a closing fence#96745
vincentkoc merged 2 commits into
openclaw:mainfrom
ly-wang19:fix/fence-close-trailing-text

ly-wang19 commented Jun 25, 2026

Uh oh!

clawsweeper Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

vincentkoc commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

ly-wang19 commented Jun 25, 2026

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Real behavior proof

Uh oh!

clawsweeper Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vincentkoc commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clawsweeper Bot commented Jun 25, 2026 •

edited

Loading