Skip to content

Commit 3ea7af4

Browse files
authored
feat(agent): paste images into conversations
Add native multi-image agent conversations, compact galleries and previews, image copy/download/Media actions, capability-aware provider replay, race hardening, and documentation.
1 parent 10448bc commit 3ea7af4

95 files changed

Lines changed: 7006 additions & 784 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitattributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
src/admin/ai/ModelPicker/ModelPicker.tsx diff

CLAUDE.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,8 +184,9 @@ Detailed patterns: [`docs/reference/typebox-patterns.md`](docs/reference/typebox
184184

185185
Every untyped boundary uses TypeBox. Inside the boundary, code trusts the parsed value.
186186

187-
- **HTTP responses (client):** `@core/http` is a single three-layer stack — there is exactly ONE way to validate a response, expressed at the altitude you need:
187+
- **HTTP responses (client):** `@core/http` is the single client stack — use the entry matching the response kind and altitude:
188188
- **`apiRequest(path, { schema, … })`** — the canonical entry. Does the `fetch` itself: sets `credentials`, serializes a JSON body (FormData passes through untouched), validates the success body against `schema`, and throws a single `ApiError` (carrying the HTTP status) on failure. Detect cancellation with `isAbortError(err)`. **Default to this** — do NOT hand-roll `fetch` + `res.ok` + `res.json()` in admin code.
189+
- **`apiBlobRequest(path, …)`** — the binary-response counterpart for authenticated image/file reads. It shares `apiRequest`'s credential, cancellation, and `ApiError` behavior, then returns a `Blob`; the caller validates the MIME type before use.
189190
- **`readEnvelope(res, Schema, fallbackMessage)`** — for the persistence layer, which performs its own injectable `fetch` (test seam) and then hands the `Response` here. Checks `res.ok` (throws `ApiError` with status + the `{ error }` envelope message), then validates the body. Its no-body sibling is `assertOk(res, fallback)` for `void`/Blob/streaming/text responses.
190191
- **`parseJsonResponse(res, Schema)`** — the low-level body-validation primitive that `apiRequest` and `readEnvelope` are *built on*. It validates a body with NO HTTP-status semantics. Reserved for genuine primitives only: the `@core/http` internals, the XHR upload path (`useUploadQueue`), and server-side fetches of external APIs. Do NOT reach for it in admin/persistence code — `assertOk(res, m); parseJsonResponse(res, S)` is exactly `readEnvelope(res, S, m)`; always write the latter.
191192
- **`JSON.parse` of persisted data:** `safeParseJson(raw, Schema)` for hard, `parseJsonWithFallback(raw, Schema, default)` for soft.

docs/architecture.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -326,7 +326,8 @@ The codebase enforces "validate, then trust": every untyped input goes through a
326326
327327
| Boundary | Helper | Lives in |
328328
|--------------------------------------|-------------------------------------------------------|---------------------------------------|
329-
| HTTP request (client, canonical) | `apiRequest(path, { schema, … })` → throws `ApiError` | `src/core/http/apiClient.ts` |
329+
| HTTP request (client, canonical JSON) | `apiRequest(path, { schema, … })` → throws `ApiError` | `src/core/http/apiClient.ts` |
330+
| HTTP request (client, binary body) | `apiBlobRequest(path, …)` → `Blob` / throws `ApiError` | `src/core/http/apiClient.ts` |
330331
| HTTP response from a held `Response` | `readEnvelope(res, Schema, fallbackMessage)` | `src/core/http/apiClient.ts` |
331332
| Raw JSON response validation | `parseJsonResponse(res, Schema)` | `src/core/utils/jsonValidate.ts` |
332333
| `JSON.parse` of persisted strings | `safeParseJson(raw, Schema)` / `parseJsonWithFallback`| `src/core/utils/jsonValidate.ts` |

docs/features/agent.md

Lines changed: 103 additions & 12 deletions
Large diffs are not rendered by default.

docs/features/media.md

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ The workspace is canvas-style: it uses `AdminWorkspaceCanvasLayout`, the lighter
1313
- **State:** one hook — `useMediaWorkspace()` — orchestrates folders, assets, selection, filters, upload queue, and folder moves. The editor store doesn't grow new slices; the Media page is self-contained.
1414
- **Folders:** folders render as first-class grid/list items in the canvas. Opening a folder filters the canvas to its contents, and nested folders show a parent-folder entry to navigate back.
1515
- **Drag/drop:** assets can be dragged into folders from the canvas or folder tree. A drop replaces the asset's folder memberships with the target folder, matching desktop file-manager move semantics.
16-
- **Floating windows:** Asset viewer, upload queue, bulk edit. Each is `useDraggablePanel('mediaViewer' | 'mediaUploadQueue' | 'mediaBulkEdit')`. Position survives reload via `workspaceLayoutStorage`.
16+
- **Floating windows:** Asset viewer, upload queue, bulk edit. Each uses the shared `@admin/shared/FloatingWindow` shell or its `useDraggablePanel` hook with a unique id (`mediaDetachedInspector`, `mediaUploadQueue`, `mediaBulkEdit`). Position survives reload via `workspaceLayoutStorage`.
1717
- **Auto-open behavior:** upload queue opens when uploads start; bulk-edit opens at 2+ selected; viewer opens on primary selection.
1818
- **Server side:** `media_assets`, `media_folders`, `media_asset_folders` tables. Handlers under `/admin/api/cms/media`, `/admin/api/cms/media/folders`, `/admin/api/cms/media/storage`. Repositories at `server/repositories/media*.ts`.
1919
- **Storage adapters:** built-in local-disk plus plugin-registered adapters. Non-public-url adapters route through `/_instatic/media/<adapterId>/<storagePath>` for signed redirects.
@@ -25,6 +25,7 @@ The workspace is canvas-style: it uses `AdminWorkspaceCanvasLayout`, the lighter
2525
```text
2626
src/admin/pages/media/
2727
├── MediaPage.tsx — top-level component
28+
├── mediaAssetEvents.ts — typed cross-workspace asset-created notifications
2829
├── components/
2930
│ ├── MediaSidebar/ — folder tree + storage + smart folders
3031
│ ├── MediaCanvas/ — file grid / list with FilterBar
@@ -37,7 +38,6 @@ src/admin/pages/media/
3738
│ ├── MediaPickerField/ — embedded picker control
3839
│ ├── MediaFolderPanel/ — folder list panel
3940
│ ├── MediaStoragePanel/ — storage adapter management
40-
│ ├── FloatingWindow/ — shared floating-window shell
4141
│ └── viewers/ — per-media-type viewers (image, video, pdf, etc.)
4242
├── hooks/
4343
│ ├── useMediaWorkspace.ts — orchestrates server state (folders, assets, filters)
@@ -54,6 +54,8 @@ src/admin/pages/media/
5454
├── mediaDragDrop.ts — TypeBox-validated drag/drop payload helpers
5555
├── smartFolders.ts — smart folder IDs, type guard, per-ID predicates
5656
└── variants.ts — image variant URL helpers
57+
58+
src/admin/shared/FloatingWindow/ — shared portal shell + persisted drag hook used across admin workspaces
5759
```
5860

5961
---
@@ -144,7 +146,7 @@ The count badge shown next to each smart folder in the sidebar is computed clien
144146

145147
### Floating windows
146148

147-
Each floating window has a unique `FloatingPanelId` (`'mediaViewer' | 'mediaUploadQueue' | 'mediaBulkEdit'`), uses `useDraggablePanel(id)` for position, and gets its position persisted via `workspaceLayoutStorage.ts`. Visibility differs by window:
149+
Each floating window has a unique `FloatingPanelId` (`'mediaDetachedInspector' | 'mediaUploadQueue' | 'mediaBulkEdit'`), uses `useDraggablePanel(id)` for position, and gets its position persisted via `workspaceLayoutStorage.ts`. Visibility differs by window:
148150

149151
| Window | How visibility is determined |
150152
|-----------------|-------------------------------------------------------------------------------------------------|
@@ -154,7 +156,7 @@ Each floating window has a unique `FloatingPanelId` (`'mediaViewer' | 'mediaUplo
154156

155157
The viewer and bulk-edit are derived rather than stored because "closed" is identical to "no selection" — every close path calls `workspace.clearSelection()`. Deriving avoids an extra render commit and the one-frame open lag that appeared with the old `setState`-in-effect approach.
156158

157-
The shared `FloatingWindow` shell at `components/FloatingWindow/` provides the chrome: drag handle, close button, position bounding.
159+
The shared `FloatingWindow` shell at `src/admin/shared/FloatingWindow/` provides the portal, chrome, drag handle, close button, and position bounding without pulling media-specific editor code into other workspaces. Its dimension-aware clamp measures each panel and keeps at least a 50px header strip reachable on every viewport edge; stored positions are re-clamped when a window mounts, changes size, or the viewport resizes.
158160

159161
---
160162

@@ -250,8 +252,10 @@ Folder routes (`/admin/api/cms/media/folders/...`) are matched **before** asset
250252

251253
### Upload pipeline
252254

255+
Uploads initiated outside the Media page use the same pipeline. In particular, the Agent Panel's explicit **Save to Media** image action resolves the private chat image, wraps it in a MIME-correct `File`, and calls `uploadCmsMediaAsset`; it does not create an AI-specific storage route. On success, `mediaAssetEvents.ts` upserts the new row into an already-mounted Site → Media explorer while the normal media cache is primed for canvas consumers.
256+
253257
```text
254-
POST /admin/api/cms/media/upload
258+
POST /admin/api/cms/media
255259
256260
257261
mediaUpload.ts ← validates upload (size + magic-byte MIME sniff)
@@ -325,8 +329,8 @@ The redirect handler is `tryServeMediaRedirect` in `server/router.ts`. The redir
325329
### Add a new floating window
326330

327331
1. Add a unique id to `FloatingPanelId` in the layout storage module.
328-
2. Create `components/<WindowName>/` with the window React component using the shared `FloatingWindow` shell.
329-
3. Use `useDraggablePanel('newWindowId')` for position; track `open` in `MediaPage` local state.
332+
2. Create `components/<WindowName>/` with the window React component using `FloatingWindow` from `@admin/shared/FloatingWindow`.
333+
3. Pass the id to `FloatingWindow`; track `open` in `MediaPage` local state. Use the exported `useDraggablePanel` directly only for a custom shell such as `MediaViewerWindow`.
330334
4. Add the auto-open rule (selection threshold, upload event, etc.) inside `MediaPage`.
331335

332336
### Add a new sidebar panel

docs/reference/design-tokens.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -418,7 +418,7 @@ The visual editor uses additional raw z-index values that are **not** tokenised.
418418
| 30 | Main toolbar |
419419
| 50 | Floating panels: PropertiesPanel, AgentPanel, DomPanel |
420420
| 55 | LeftSidebar, RightSidebar, PanelRail |
421-
| 70 | Media floating windows (FloatingWindow, MediaViewerWindow) |
421+
| 90 | Shared admin floating windows (`FloatingWindow`, `MediaViewerWindow`, agent image preview) |
422422
| 80 | CodeEditorPanel |
423423
| 201 | Toolbar popovers / dropdowns |
424424
| 400–401 | PreviewOverlay |

docs/reference/typebox-patterns.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Schemas are the **source of truth**. Domain types come from `Static<typeof Schem
1313
- **Helpers live in `src/core/utils/typeboxHelpers.ts`**`Type`, `Value`, `Static`, `withFallback`, `parseValue`, `safeParseValue`, `filterArray`, `formatValueErrors`.
1414
- **Compiled validators live in `src/core/utils/typeboxCompiler.ts`**`compiled`, `compiledCheck`, `compiledDecode`, `compiledSafeParseValue`, `compiledFormatValueErrors`. Hot repeated `Check` / `Decode` / `Errors` paths should use these or helpers that already call them.
1515
- **JSON boundary helpers live in `src/core/utils/jsonValidate.ts`**`safeParseJson`, `parseJsonWithFallback`, `parseJsonResponse`.
16-
- **Canonical client HTTP layer:** `@core/http``apiRequest(path, { schema, … })` (the default for browser→server calls), plus `ApiError`, `isAbortError`, `readEnvelope`, `responseErrorMessage`, `ErrorEnvelopeSchema`. All defined in `src/core/http/apiClient.ts`.
16+
- **Canonical client HTTP layer:** `@core/http``apiRequest(path, { schema, … })` (the default for JSON browser→server calls), `apiBlobRequest(path, …)` for binary responses, plus `ApiError`, `isAbortError`, `readEnvelope`, `responseErrorMessage`, `ErrorEnvelopeSchema`. All defined in `src/core/http/apiClient.ts`.
1717
- **Schemas are source of truth.** `type Foo = Static<typeof FooSchema>` — never a hand-rolled interface beside the schema.
1818
- **Soft fallbacks** for corrupted local storage / optional config use `withFallback(schema, default)` + `parseJsonWithFallback`.
1919
- **Hard fallbacks** for required documents throw and bubble to an error boundary.
@@ -94,10 +94,11 @@ const prefs = parseJsonWithFallback(
9494
| Helper | Purpose |
9595
|-------------------------------------------------|------------------------------------------------------------------|
9696
| `apiRequest(path, { method, body, schema, query, signal, … })` | **The default for browser→server calls.** Sets `credentials`, JSON-serializes `body`, validates the success body against `schema` (returns `Static<schema>`; `void` without one), and throws `ApiError` on a non-OK status. |
97+
| `apiBlobRequest(path, { signal, … })` | Binary-response counterpart for authenticated image/file reads. Uses the same transport and `ApiError` envelope handling, then returns a `Blob`; the caller validates the MIME type before use. |
9798
| `readEnvelope(res, schema, fallbackMessage)` | For code that already holds a `Response` (the persistence layer, which injects its own `fetch`): check `res.ok` (throw `ApiError` with `responseErrorMessage(res, fallback)` if not), then validate body against `schema` |
9899
| `assertOk(res, fallbackMessage)` | No-body counterpart to `readEnvelope`: throw `ApiError` if the `Response` is not OK, otherwise return (for void mutations / bodies parsed separately) |
99100
| `responseErrorMessage(res, fallback)` | Extract a useful error message from a failed `Response` (reads `{ error: string }` envelope if present, then raw text, otherwise the fallback) |
100-
| `ApiError` | The single error type thrown by `apiRequest`/`readEnvelope`; carries `.status` so UI can branch (403, 404, …) |
101+
| `ApiError` | The single error type thrown by `apiRequest`/`apiBlobRequest`/`readEnvelope`; carries `.status` so UI can branch (403, 404, …) |
101102
| `isAbortError(err)` | True for an aborted fetch (user cancellation / superseded request) — the uniform replacement for `(err as Error).name === 'AbortError'` |
102103

103104
---
@@ -286,7 +287,8 @@ Common boundaries already wrapped — extend the same pattern when you add a new
286287

287288
| Boundary | Helper | Lives in |
288289
|--------------------------------------------|-----------------------------------------------------|-----------------------------------------|
289-
| HTTP request (client, canonical) | `apiRequest(path, { schema, … })` | `src/core/http/apiClient.ts` |
290+
| HTTP request (client, canonical JSON) | `apiRequest(path, { schema, … })` | `src/core/http/apiClient.ts` |
291+
| HTTP request (client, binary body) | `apiBlobRequest(path, { signal, … })` | `src/core/http/apiClient.ts` |
290292
| HTTP response from a held `Response` | `readEnvelope(res, Schema, fallback)` | `src/core/http/apiClient.ts` |
291293
| HTTP body-validation primitive (no status semantics; `@core/http` internals, XHR, server-side external APIs) | `parseJsonResponse(res, Schema)` | `src/core/utils/jsonValidate.ts` |
292294
| Request body (server handler) | `readValidatedBody(req, Schema)` → typed value or `null` (return `badRequest` on null) | `server/http.ts` + per-handler |
@@ -333,7 +335,7 @@ Common boundaries already wrapped — extend the same pattern when you add a new
333335
- `src/core/utils/typeboxHelpers.ts` — helper layer (`parseValue`, `withFallback`, `filterArray`, etc.)
334336
- `src/core/utils/typeboxCompiler.ts` — cached TypeCompiler layer for hot validation execution
335337
- `src/core/utils/jsonValidate.ts` — JSON boundary helpers
336-
- `src/core/http/apiClient.ts``apiRequest`, `ApiError`, `isAbortError`, `readEnvelope`, `assertOk`, `responseErrorMessage`, `ErrorEnvelopeSchema`
338+
- `src/core/http/apiClient.ts``apiRequest`, `apiBlobRequest`, `ApiError`, `isAbortError`, `readEnvelope`, `assertOk`, `responseErrorMessage`, `ErrorEnvelopeSchema`
337339
- `src/core/persistence/responseSchemas.ts` — shared CMS HTTP response schemas
338340
- `src/core/persistence/validate.ts``validateSite`, `validatePages`, `ValidatePagesOptions`, `SiteValidationError`
339341
- `src/core/plugins/manifest.ts``parsePluginManifest`

server/ai/conversations/history.ts

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
* preceding tool call in the replayed history.
2424
*/
2525

26-
import type { AiMessage, AiToolOutput } from '../runtime/types'
26+
import type { AiContentBlock, AiMessage, AiToolOutput } from '../runtime/types'
2727
import type { MessageRecord } from './types'
2828

2929
/**
@@ -33,6 +33,9 @@ import type { MessageRecord } from './types'
3333
export const INTERRUPTED_TOOL_RESULT_ERROR =
3434
'Tool call did not complete — the previous turn was interrupted before a result was produced.'
3535

36+
export const NON_VISION_USER_IMAGE_OMITTED =
37+
'[Attached image omitted because the selected model does not support image input.]'
38+
3639
/**
3740
* Reconstruct `AiMessage` history from persisted `MessageRecord` rows.
3841
*
@@ -100,3 +103,36 @@ export function buildMessageHistory(records: MessageRecord[]): AiMessage[] {
100103

101104
return out
102105
}
106+
107+
/**
108+
* Adapt persisted user images to the selected model without mutating history.
109+
*
110+
* Vision models retain every image-bearing turn. Text-only models receive one
111+
* breadcrumb per image-bearing turn, which keeps a conversation usable after
112+
* switching away from a vision model.
113+
*/
114+
export function projectUserImagesForModel(
115+
messages: readonly AiMessage[],
116+
visionInput: boolean,
117+
): AiMessage[] {
118+
if (visionInput) return [...messages]
119+
120+
return messages.map((message) => {
121+
if (message.role !== 'user' || !message.content.some((block) => block.kind === 'image')) {
122+
return message
123+
}
124+
let breadcrumbAdded = false
125+
const content: AiContentBlock[] = []
126+
for (const block of message.content) {
127+
if (block.kind !== 'image') {
128+
content.push(block)
129+
continue
130+
}
131+
if (!breadcrumbAdded) {
132+
content.push({ kind: 'text', text: NON_VISION_USER_IMAGE_OMITTED })
133+
breadcrumbAdded = true
134+
}
135+
}
136+
return { role: 'user', content }
137+
})
138+
}

0 commit comments

Comments
 (0)