fix(markdown): a fenced-code line with trailing text is content, not a closing fence#96687
Closed
ly-wang19 wants to merge 1 commit into
Closed
fix(markdown): a fenced-code line with trailing text is content, not a closing fence#96687ly-wang19 wants to merge 1 commit into
ly-wang19 wants to merge 1 commit into
Conversation
…a closing fence scanFenceSpans accepted any line starting with >=3 matching fence markers as a closing fence, ignoring trailing text after the marker. Per CommonMark a closing fence may be followed only by whitespace, so a code-content line such as "``` not a close" was wrongly treated as a close: the block ended early, the following lines were reported as outside any fence, and the trailing marker line became a new unclosed opener. That made isSafeFenceBreak() return true for offsets inside the real code block and findFenceSpanAt() return undefined, so chunkers (chunkMarkdownText, the embedded-agent block chunker) could split inside a fenced code block — the exact thing this module exists to prevent. Require the closing fence's trailing text to be whitespace-only. Opening info strings, bare closes, and longer same-marker closes are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Closing this PR because the author has more than 20 active PRs in this repo. Please reduce the active PR queue and reopen or resubmit once it is back under the limit. You can close your own PRs to get back under the limit. |
Contributor
Author
|
Reopening — I've trimmed my active-PR queue back under the repo limit. This fix is self-contained and verified (reproduce → fix → fail-before-test → green CI), with full proof in the PR body; no live-channel environment needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Problem This Solves
scanFenceSpans/parseFenceSpans(packages/markdown-core/src/fences.ts) maps where fenced code blocks start and end so chunkers never split inside one. It accepted any line that begins with ≥3 matching fence markers as a closing fence, ignoring trailing text after the marker. Per CommonMark a closing fence may be followed only by whitespace, so a code-content line like``` not a closewas wrongly treated as a close: the block ended early, the following lines were reported as outside any fence, and the trailing real```became a new unclosed opener.The consequence:
isSafeFenceBreak()returnstruefor offsets inside the actual code block andfindFenceSpanAt()returnsundefined, sochunkMarkdownTextand the embedded-agent block chunker can split inside a fenced code block — the exact thing this module exists to prevent. Code blocks that demonstrate markdown/fences (a line starting with ``` followed by text) are ordinary content.Why This Change Was Made
The closing branch already checked the marker character and length (its comment cites CommonMark) but missed the trailing-text rule. The fix adds that one condition: a closing fence's trailing text must be whitespace-only. The reference implementation (micromark) confirms the sample input is a single code block. This makes fence detection stricter — the safe direction (it keeps more text inside a fence, so a chunker is less likely to split inside code).
User Impact
Outbound/agent text containing a fenced code block whose body has a
```-prefixed line is no longer mis-parsed, so chunkers won't break the code block at that point. Opening info strings, bare closes, and longer same-marker closes are unaffected. No API change.Evidence
oxfmt,oxlint,tsgo:corepass. A new colocatedfences.test.tsregression test passes and fails when the fix is reverted (fail-before). Both downstream suites —src/auto-reply/chunk.test.tsandsrc/agents/embedded-agent-block-chunker.test.ts— stay green.Real behavior proof
Behavior addressed: a fenced-code content line beginning with the fence marker but carrying trailing text (
``` not a close) was treated as a closing fence, soisSafeFenceBreakreported an offset inside the code block as a safe chunk break, letting chunkers split inside the block.Real environment tested: Ran the exported
parseFenceSpans/isSafeFenceBreak/findFenceSpanAtfrompackages/markdown-core/src/fences.tsviatsxon Node v22.22.1 (no mocks), before and after the patch, plus the colocated Vitest test and the two downstream chunker suites vianode scripts/run-vitest.mjs.Exact steps or command run after this patch:
parseFenceSpans("```\\ncode\\n``` not a close\\nmore code\\n```\\n")andisSafeFenceBreak(spans, <offset of "more code">); thennode scripts/run-vitest.mjs packages/markdown-core/src/fences.test.ts src/auto-reply/chunk.test.ts src/agents/embedded-agent-block-chunker.test.ts,node scripts/run-oxlint.mjs,pnpm tsgo:core.Evidence after fix:
Observed result after fix: a marker line with trailing text stays inside the code block (one span), interior offsets are no longer reported as safe breaks, and valid closes (bare, longer same-marker, with trailing whitespace) plus opening info strings still behave correctly; the downstream chunkers are unchanged on their existing tests.
What was not tested: a full end-to-end channel render of a chunked message containing such a code block (no live channel in this environment); the fix is a pure parser change proven at the function and unit-test level, cross-checked against the CommonMark reference (micromark) for the single-code-block interpretation.