Summary

• preserve UTF-16 boundaries when EmbeddedBlockChunker must split at maxChars
• keep fence close/reopen handling aligned with the adjusted split index
• add production-class regressions for plain, fenced, and one-code-unit budgets

What Problem This Solves

When no natural Markdown break existed, the chunker returned the raw UTF-16 code-unit maxChars index. If that index fell between an astral character's high and low surrogates, the emitted chunk ended with an isolated high surrogate and the buffered remainder began with an isolated low surrogate.

User Impact

Streamed block chunks no longer contain malformed UTF-16 when a forced size boundary lands inside emoji or another astral character.

Origin / follow-up

• Follows: fix(twitch): preserve UTF-16 pairs in outbound chunks #104030, which preserved UTF-16 pairs in Twitch outbound chunks.
• Gap this fills: the shared embedded-agent block chunker independently used forced code-unit indices before its local emit boundary.
• Consistency: forced chunk boundaries now reuse the repository's shared sliceUtf16Safe utility.

Competition / linked PR analysis

A fresh exact-class search found #94216 and #94247 touching the same chunker files, but both address paragraph-separator preservation. Their diffs append or coordinate \n\n around paragraph breaks and do not align forced maxChars indices to UTF-16 boundaries. #68986 does not change the chunker source or target test.

No open PR covers the surrogate-boundary root cause, and the author's open PR scan found no overlap.

Merge risk

• Why it matters / User impact: malformed chunk strings can render as replacement characters or fail later text handling.
• Risk labels considered: session-state and streamed-text.
• Risk explanation: a split can move one code unit earlier when maxChars bisects a surrogate pair. For the otherwise impossible maxChars=1 case with an astral first character, the complete two-unit code point is emitted so the chunker retains forward progress.
• Why acceptable: normal budgets remain hard upper bounds; only a one-unit budget can be exceeded by one code unit because no valid non-empty UTF-16 chunk can contain half of a surrogate pair.
• Maintainer-ready confidence: High; the production class is directly reproducible and all forced branches share one adjusted index.
• Patch quality notes: focused two-file patch with no type escape, fallback serializer, public contract change, dependency change, or unrelated refactor.
• Target test file: src/agents/embedded-agent-block-chunker.test.ts
• Root cause: forced break branches used maxChars directly as a UTF-16 code-unit slice index and propagated that same unsafe index into chunk emission and remainder state.
• Why this is root-cause fix: the source index is normalized once before fence lookup, fence split metadata, raw chunk slicing, emit, and buffered remainder reconstruction.
• What did NOT change: paragraph/newline/sentence preference, natural-break selection, fence parsing, close/reopen syntax, flushOnParagraph, and callback APIs remain unchanged.
• Architecture / source-of-truth check: #pickBreakIndex is the source of truth for forced split positions; downstream emit consumers are not patched to repair malformed strings.
• Scenario locked in: a 20-unit plain split after 19 ASCII units, the same split inside an open Markdown fence, and a one-unit budget beginning with an emoji.
• Why this is the smallest reliable guardrail: one public-class test observes emitted chunks and bufferedText across both forced branches and the minimum-budget progress case.

Real behavior proof

• Behavior or issue addressed: forced block chunks and their buffered remainders retain complete UTF-16 surrogate pairs.
• Canonical reachability path: input → schema/type BlockReplyChunking plus streamed text → ingestion EmbeddedBlockChunker.append → normalization fence parsing and #pickBreakIndex → runtime object BreakResult and internal buffer → emitted chunk and buffered-remainder effect. Concretely, appended assistant text reaches the forced maxChars branch, receives one shared safe index, passes through #emitBreakResult, and is exposed through the production emit callback and bufferedText.
• Boundary crossed: the production class ingests streamed text and passes a chunk string to its local emit callback; the proof terminates at that callback and the retained runtime buffer.
• Shared helper / provider constraint check: reused sliceUtf16Safe through the existing shared utility export; provider-specific constraints are not applicable.
• Fix classification: Root cause fix
• Real environment tested: Linux worktree on Node 22 with the repository pnpm toolchain.
• Exact steps or command run after this patch: direct EmbeddedBlockChunker execution for plain, fenced, one-unit, and natural-paragraph fixtures, plus pnpm test src/agents/embedded-agent-block-chunker.test.ts.
• Evidence after fix:

BEFORE plain: emitted chunk ended with lone high surrogate U+D83D; buffered remainder began with lone low surrogate U+DE00
AFTER plain:  emitted length=19; buffered remainder begins with complete emoji
AFTER fenced: emitted chunk closes the Markdown fence; reopened remainder begins with complete emoji
TINY BUDGET:  complete emoji emitted first and the chunker retained forward progress
CONTROL:      natural paragraph break output and remainder stayed unchanged
TESTS:        1 file passed, 9 tests passed

• Observed result after fix: plain and fenced forced splits emit well-formed strings, the corresponding remainders begin with the complete emoji, the one-unit case drains without stalling, and the natural paragraph control is unchanged.
• What was not tested: provider or channel consumption was not exercised; this defect and proof terminate at the local production emit callback and buffered state.

Regression Test Plan

• run pnpm test src/agents/embedded-agent-block-chunker.test.ts
• verify plain and fenced forced splits keep complete surrogate pairs on both sides
• verify the one-unit budget retains forward progress
• verify existing paragraph, fence, and max-length behavior remains green

Risk / Compatibility

No API, schema, configuration, or dependency changes. Typical forced chunks can be one code unit shorter only when the previous boundary split a surrogate pair. A configured one-unit budget can emit a two-unit astral code point because emitting either half would be malformed and emitting nothing would stall.

Out of scope

No changes to paragraph-separator reconstruction, downstream coalescing, channel formatting, provider output, or delivery transport. The two adjacent open PRs remain responsible for their separate paragraph-boundary behavior.

Root Cause

The forced fallback treated a UTF-16 code-unit budget as a universally safe string boundary. The final source.slice then materialized that unsafe index into both sides of the split, so the boundary must be corrected at #pickBreakIndex before fence and emission logic consume it.