AgentForest Platform - Critical Module
Specification
Sub-System 6 (Testing Module): Test
Execution Orchestration
1. Introduction
1.1. Purpose
This document provides a highly detailed specification for the
Test Execution Orchestration module. This module is designed to
enable users to thoroughly validate the functional correctness of
their AI agent flows and multi-flow applications (orchestrations). It
focuses on providing intuitive interfaces for initiating test runs
with custom inputs, triggering backend execution, capturing
results, and centralizing test history for review and debugging.
1.2. Scope
The scope of this module includes:
• Test Initiation UI: The interactive dialogs for triggering tests
from the Flow Canvas and Orchestrator Canvas.
• Input Provisioning: Dynamic generation of input fields based on
flow schemas.
• Backend Execution Invocation: Calling the appropriate
Supabase Edge Functions (run-project-flow, run-project-app) to
perform the actual test computation.
• Result Communication: Displaying the output and status of the
test run to the user.
• Test History Storage: Ensuring that test executions are logged
persistently.
• Test History Listing Page: Displaying a browsable list of past
test runs with filtering and sorting.
• Test Details View: Providing a modal/dialog to inspect the full
inputs, outputs, and status of a specific test run.
• Trace Integration: Linking test runs to detailed execution traces
in the Monitoring & Logging Sub-System.
Excluded from this scope:
• Automated testing frameworks (ee.g., unit/integration tests for
nodes/flows) - considered advanced features.
• Performance testing or load testing features (part of Monitoring &
Logging in future phases).
• The internal interpretation logic of individual nodes/flows (handled
by Flow Execution Engine _shared/flow_engine.ts).
• Deep debugging tools (e.g., step-through execution, breakpoint
support).
1.3. Requirements (Detailed)
1.3.1. Test Initiation & Input Provisioning
• FR-TEO-1: The system SHALL provide a "Test Flow" button on the
Flow Canvas UI.
◦ FR-TEO-1.1: Clicking this button SHALL open
the TestFlowDialog (or inline dialog on the Flow Canvas itself,
like FlowCanvasInstance's test modal).
◦ FR-TEO-1.2: The TestFlowDialog SHALL dynamically generate
input fields for the flow based on the JSON schema defined in the
flow's Input Node.
FR-TEO-1.2.1: It SHALL
support string, number, boolean, textarea (for long strings),
and enum (for select dropdowns) input types.
FR-TEO-1.2.2: It SHALL
display title, description, placeholder from the schema.
FR-TEO-1.2.3: It SHALL mark required fields and prevent
submission if they are empty.
FR-TEO-1.2.4: It SHALL pre-populate fields with default values
from the schema, if provided.
• FR-TEO-2: The system SHALL provide a "Test App" button on the
Orchestrator Canvas UI.
◦ FR-TEO-2.1: Clicking this button SHALL open the TestAppDialog.
◦ FR-TEO-2.2: The TestAppDialog SHALL dynamically generate
input fields based on the JSON schema of the Input Node within
the orchestrated application's entry_node_id's flow.
• FR-TEO-3: Users SHALL be able to input custom data into the
generated fields in the test dialogs.
1.3.2. Test Execution & Result Communication
• FR-TEO-4: Upon user submission in the test dialog:
◦ FR-TEO-4.1: The system SHALL display a loading indicator in the
UI.
◦ FR-TEO-4.2: The system SHALL make a fetch call to the
appropriate Supabase Edge Function:
For Flow Tests: [Link]("run-project-flow",
{ body: { project_id, initial_flow_inputs: providedInputs,
session_id_override: flowId } }).
For App Tests: [Link]("run-project-app", {
body: { project_id, app_id, initial_inputs: providedInputs }
}).
• FR-TEO-5: The test dialog SHALL display the raw JSON
output/result returned by the Edge Function execution.
• FR-TEO-6: The test dialog SHALL display any error messages
returned by the Edge Function, along with a distinct visual style
(e.g., red text).
• FR-TEO-7: Test execution SHALL be logged
to public.execution_logs (handled by run-project-flow/run-
project-app Edge Functions). These logs SHALL
include details_json containing
the initial_inputs and final_result/error.
1.3.3. Test History Listing
(src/app/(dashboard)/projects/[projectId]/tests/[Link])
• FR-TEO-8: The system SHALL provide a dedicated "Testing List"
page displaying a paginated table of past test executions for the
current project.
• FR-TEO-9: The test history list SHALL display the following
columns:
◦ Timestamp: When the test was run.
◦ Run Type: "Flow Test" or "App Test".
◦ Status: "Success" or "Error".
◦ Flow/App Name: The name of the flow or application tested.
◦ Result Summary: A truncated preview of the final output or error
message.
◦ Duration: Total execution time in milliseconds.
◦ Cost: Estimated LLM cost for the run.
◦ Action: Buttons for "View Details".
• FR-TEO-10: The UI SHALL provide filtering options:
◦ Run Type: Select dropdown for "Flow Test" or "App Test".
◦ Status: Select dropdown for "Success" or "Error".
◦ Search: Input field to search by flow/app name,
or run_session_id.
◦ Date Range: Input fields for "Date From" and "Date To".
• FR-TEO-11: The UI SHALL allow sorting
by Timestamp (ascending/descending).
• FR-TEO-12: The page SHALL allow refreshing the list.
1.3.4. Test Details View
• FR-TEO-13: Clicking "View Details" from the test history list
SHALL open a modal/dialog (reusing LogDetailModal).
• FR-TEO-14: The detailed view SHALL display:
◦ All attributes of the corresponding LogEntry from execution_logs.
◦ The full JSON of initial_inputs provided for the test.
◦ The full JSON of the final_result or
detailed error from details_json.
• FR-TEO-15: The detailed view SHALL include a button/link to
"View Logs/Timeline" which navigates to the Monitoring & Logging
Sub-System's Logs page, filtered to the
specific run_session_id and potentially highlighting its trace.
1.4. Non-Functional Requirements
• NFR-TEO-1 (Performance): Test execution (client-side
submission and result display) SHALL be highly responsive.
• NFR-TEO-2 (Data Integrity): Test run records
in execution_logs SHALL be complete and accurate, representing
the exact inputs, outputs, and status.
• NFR-TEO-3 (Scalability): The execution_logs table and its
query performance shall be scalable to handle a high volume of
test runs.
• NFR-TEO-4 (Usability): Test interfaces SHALL be intuitive for
quickly understanding test outcomes and diagnosing issues.
• NFR-TEO-5 (Traceability): Every test run SHALL generate a
unique run_session_id that links all associated execution logs,
enabling full traceability in the Monitoring & Logging Sub-System.
2. Specification Design Document (SDD)
2.1. High-Level Architecture
The Testing Module Sub-System provides specific UI entry points
for testing and then relies on the established backend execution
Edge Functions (run-project-flow, run-project-app) and
the execution_logs database table for persistence and history. It
largely reuses components from the Monitoring & Logging Sub-
System.
graph TD
A[Flow Canvas UI] --1a. "Test Flow" Click --> B(Flow
Test Dialog)
C[Orchestrator Canvas UI] --1b. "Test App" Click -->
D(App Test Dialog: `[Link]`)
B --2. Inputs, FlowID --> E{Supabase Edge Function:
`run-project-flow`}
D --2. Inputs, AppID --> F{Supabase Edge Function: `run-
project-app`}
E --3. Logs Execution Events --> G(Supabase DB:
`execution_logs`)
F --3. Logs Execution Events --> G
B --4. Display Result/Error --> B
D --4. Display Result/Error --> D
H[Testing List Page UI:
`projects/[projectId]/tests/[Link]`] --5. Query
`execution_logs` (filtered by test-related `event_type`) -->
G
H --6. Display Paginated Table --> H
H --7. "View Details" --> I[LogDetailModal Component]
H --8. "View Logs/Timeline" Button (from Modal) -->
J[Monitoring & Logging Sub-System: Logs Page]
G --9. Retrieve AI Keys --> K[Project Settings & Secrets
Sub-System: Secrets Vault]
content_copy
download
Use code with caution.
Mermaid
2.2. Component Breakdown & Detailed
Design
2.2.1. Flow Test Dialog (Integrated
into src/app/(dashboard)/projects/[projectId]/[Link] - FlowCa
nvasInstance)
• Purpose: Allow users to test a flow directly from the Flow
Canvas.
• Type: Client Component (part of the main FlowCanvasInstance).
• Design:
◦ Uses a Shadcn Dialog for the modal.
◦ Input Fields: Dynamically generated
using inputNodeSchemaForTestModal (derived from the
flow's Input Node [Link]). This object provides schema
properties, required fields, default values, titles, and descriptions.
◦ Input Types: Renders Input for string/number, Textarea for long
strings, Checkbox for booleans, Select for enums.
◦ Submission (handleTestFlow):
Collects inputs.
Performs basic client-side validation (required fields).
Calls [Link]("run-project-flow", { body: {
project_id: projectId, initial_flow_inputs: testInputs,
session_id_override: [Link] } }).
The session_id_override is used to link logs to this specific test
run.
Updates isTesting state, displays Loader2.
Displays testResult or testError in a <Textarea> within the
dialog.
2.2.2. App Test Dialog (src/components/project/test-app-
[Link])
• Purpose: Allow users to test an orchestrated application.
• Type: Client Component.
• Design:
◦ Uses Shadcn Dialog.
◦ Input Fields: Fetches the orchestration_definition_json for
the appId to determine the entry_flow_id. Then fetches that
entry flow's Input Node schema to dynamically generate input
fields.
◦ Submission (handleSubmit):
Collects inputs.
Calls [Link]("run-project-app", { body:
{ project_id: projectId, app_id: appId, initial_inputs:
testInputs } }).
Updates isLoading state, displays Loader2.
Displays result or error in a <Textarea>.
2.2.3. Test History Listing Page
(src/app/(dashboard)/projects/[projectId]/tests/[Link])
• Purpose: Display a historical list of all test runs for the project.
• Type: Client Component ("use client").
• State: testRuns, isLoading, error, selectedEventType, searchTer
m, dateFrom, dateTo, currentPage, totalTestRuns, sortOrder, selec
tedLogEntry, isDetailModalOpen.
• Data Fetching (fetchTestRuns useCallback):
◦ Queries public.execution_logs filtered
by project_id, event_type (restricted to TEST_EVENT_TYPES), and
applying UI filters.
◦ Uses count: "exact" for total rows, and range for pagination.
◦ Populates testRuns and totalTestRuns.
• UI:
◦ Filter controls: Select for Run Type, Select for Status, Input for
search, Input type date for date range, Button for sort order
toggle.
◦ Table (Shadcn Table): Displays testRuns data with columns
for Timestamp, Run
Type (Flow/App), Status (Success/Error), Flow/App Name, Result
Summary, Duration, Cost, and Action.
◦ Badge components for visual status.
◦ Pagination controls (Button for Previous/Next).
• Actions:
◦ "View Details"
(handleViewDetails): Calls LogDetailModal component.
◦ "Refresh" Button: Calls fetchTestRuns(1).
2.2.4. Test Detail View (Reuses src/components/project/log-
[Link])
• Purpose: Display full details of a specific test run.
• Type: Client Component.
• Design: Reuses LogDetailModal. The LogEntry structure
from execution_logs provides all necessary data,
including details_json for initial_inputs and final_result/erro
r.
2.3. Interconnections
• To UI & Navigation Sub-System:
◦ The "Testing List" page is accessible via the Project Sidebar.
◦ "Test Flow" and "Test App" buttons are integrated into the main
canvas headers.
• To Flow Canvas Sub-System:
◦ Provides the TestFlowDialog for direct interaction on the Flow
Canvas.
◦ The Input Node schema from Flow Canvas dictates the test input
fields.
• To Orchestrator Canvas Sub-System:
◦ Provides the TestAppDialog for direct interaction on the
Orchestrator Canvas.
◦ The entry flow's Input Node schema from Orchestrator Canvas
dictates the app test input fields.
• To Monitoring & Logging Sub-System:
◦ This is the primary integration point. All test execution data is
stored in execution_logs (managed by Monitoring & Logging).
◦ The LogDetailModal component and TraceTimeline component
(for deep debugging) are directly reused from Monitoring &
Logging.
◦ The Testing List page essentially provides a filtered view of
the execution_logs.
• To Project Settings & Secrets Sub-System:
◦ The execution functions (run-project-flow, run-project-app)
retrieve LLM API keys and other secrets from the Project Secrets
Vault at runtime to execute tests.
• To Deployment Factory Sub-System:
◦ Successful tests are a prerequisite for confidence in deployment.
The Testing Module provides the validation output that feeds into
deployment decisions.
This completes the detailed requirements and design specification
for the Testing Module Sub-System.