Loop Engineering
Engineering Implementation Notes
Detailed technical spec for Loop 1 (Case Closure) and Loop 2 (Agent Improvement)
Audience: engineering team. Status: draft for review before Phase 1 kickoff.
How to use this doc: Section 1 is shared infrastructure everyone depends on — build this first, in one sitting, as a
team. Sections 2 and 3 are implementation detail per loop. Section 6 is the actual sprint-sized task breakdown — start
there if you just want the checklist.
0. Non-negotiables Before Writing Any Code
• No loop touches production traffic until the reason-code taxonomy (1.1) is frozen v1.0.0. Everything
downstream — clustering, eval, dashboards — joins on this. Changing it later means a backfill migration across
every table that references it.
• Every state transition is written by exactly one code path (a Postgres function, not application code in three
different services). This is what makes the audit trail trustworthy.
• Loop 2 never writes to a live case. It only ever produces a new policy version artifact. This separation is
architectural, not a convention — enforce it with permissions (Loop 2's service role has no INSERT/UPDATE grant
on the cases or state_transitions tables).
1. Shared Foundations
1.1 Reason-code taxonomy
This is a controlled vocabulary, versioned like a schema migration, not a free-text field.
{
"taxonomy_version": "1.0.0",
"codes": [
{
"code": "NO_PICKUP",
"category": "non_contact",
"terminal": false,
"description": "Call placed, no answer within ring timeout"
},
{
"code": "SOFT_NO_RETRY",
"category": "engaged_negative",
"terminal": false,
"description": "Borrower declined but did not close the door;
eligible for retry after cooldown"
},
{
"code": "DISPUTE_RAISED",
"category": "engaged_negative",
"terminal": false,
"requires_handoff": true,
"description": "Borrower disputes amount or validity of debt"
},
{
"code": "PTP_GIVEN",
"category": "engaged_positive",
"terminal": false,
"description": "Promise-to-pay committed with date/amount"
},
{
"code": "PAYMENT_VERIFIED",
"category": "resolution",
"terminal": true,
"description": "Payment gateway confirms receipt matching PTP"
},
{
"code": "COMPLIANCE_HOLD",
"category": "system",
"terminal": false,
"requires_handoff": true,
"description": "DND/NCPR window or retry cap prevents further contact"
}
]
}
Rules for maintaining this: Additive changes (new codes) are a minor version bump. Renaming or removing a code is
a major version bump and needs a backfill plan. Every code declares terminal and requires_handoff — these are
read directly by the state machine, not re-derived. Own this file in a git repo, not a spreadsheet — treat changes like a
database migration (PR review, changelog).
1.2 Core data model (Postgres / Supabase)
-- One row per case (collections account, lead, campaign target)
create table cases (
id uuid primary key default gen_random_uuid(),
product_line text not null, -- 'collections' | 'lead_qual' | 'campaign'
external_ref text not null, -- client's account/lead id
current_state text not null default 'NEW',
taxonomy_version text not null,
policy_version text not null, -- which NBA policy version drives this case
created_at timestamptz not null default now(),
updated_at timestamptz not null default now(),
metadata jsonb not null default '{}' -- product-specific fields
);
-- Append-only ledger. Never updated, never deleted.
create table state_transitions (
id bigserial primary key,
case_id uuid not null references cases(id),
from_state text not null,
to_state text not null,
reason_code text not null,
action_taken text, -- 'CALL' | 'SMS' | 'WHATSAPP' | 'ESCALATE' | 'WAIT'
channel_meta jsonb default '{}', -- call duration, transcript_id, sms status, etc.
actor text not null, -- 'policy_engine' | 'human_agent' | 'system_timeout'
created_at timestamptz not null default now()
);
create index on state_transitions (case_id, created_at);
create index on cases (product_line, current_state);
-- One row per candidate policy version (Loop 2 output, consumed by Loop 1)
create table policy_versions (
id text primary key, -- e.g. 'collections-v14'
product_line text not null,
status text not null default 'draft', -- draft|shadow|canary|promoted|retired
parent_version text references policy_versions(id),
change_summary text not null,
created_by text not null, -- named human approver, see Stage C
eval_report_ref text, -- link to Stage D eval artifact
created_at timestamptz not null default now(),
promoted_at timestamptz
);
The transition function — the single write path
create or replace function apply_transition(
p_case_id uuid,
p_reason_code text,
p_action_taken text,
p_channel_meta jsonb,
p_actor text
) returns void as $$
declare
v_current_state text;
v_next_state text;
v_terminal boolean;
v_requires_handoff boolean;
begin
select current_state into v_current_state
from cases where id = p_case_id for update;
select terminal, requires_handoff into v_terminal, v_requires_handoff
from reason_codes where code = p_reason_code;
v_next_state := case
when v_requires_handoff then 'HUMAN_HANDOFF'
when v_terminal then 'CLOSED_WON' -- or CLOSED_LOST, per category
else next_state_lookup(v_current_state, p_reason_code)
end;
insert into state_transitions
(case_id, from_state, to_state, reason_code, action_taken, channel_meta, actor)
values
(p_case_id, v_current_state, v_next_state, p_reason_code,
p_action_taken, p_channel_meta, p_actor);
update cases set current_state = v_next_state, updated_at = now()
where id = p_case_id;
end;
$$ language plpgsql;
This function is the only place cases.current_state is ever written. n8n calls it via an RPC (Supabase exposes
Postgres functions as callable endpoints); no service should ever run a raw UPDATE cases SET current_state
= ....
1.3 Event/telemetry contract
Every transition writes a Langfuse trace (or updates an existing one) tagged with:
• case_id, product_line, policy_version, reason_code
• Link to the raw transcript/interaction artifact (voice transcript, SMS thread)
This is what makes Stage A of Loop 2 a query, not a manual export.
2. Loop 1 — Case Closure Loop, Implementation Detail
2.1 Architecture
[Scheduler: cron every N min]
|
v
[State Re-eval Job] -- reads cases where current_state NOT terminal
| and next_eligible_at <= now()
v
[Eligibility Gate] -- checks DND/NCPR window, retry cap, cooldown
| (fail -> log COMPLIANCE_HOLD, skip)
v
[NBA Policy Layer] -- decision table / ranking model -> action
|
v
[n8n: Execution Router] -- CALL -> voice agent platform
SMS/WHATSAPP -> messaging integration
ESCALATE -> human queue
|
v
[Outcome Webhook] -- voice/messaging platform posts result
|
v
[apply_transition() RPC] -- writes to state_transitions, updates cases
2.2 Eligibility gate — implement as a pure function, testable in isolation
is_eligible(case, now) -> { eligible: bool, reason_if_not: code }
checks, in order (short-circuit on first failure):
1. DND/NCPR window (per TRAI calling-hours rules)
2. retry_count < max_retries_for(product_line)
3. cooldown elapsed since last contact (per-case, configurable per
reason_code -- e.g. SOFT_NO_RETRY cooldown != NO_PICKUP cooldown)
4. not currently in HUMAN_HANDOFF or terminal state
Write this as a standalone module with 100% unit test coverage — it's the piece regulators and client legal teams will
ask to see directly. Do not embed this logic inline in an n8n node; keep it in versioned, testable code (even if invoked
from n8n via an HTTP call to a small service).
2.3 NBA policy layer — v1 spec (rules/decision table, not ML)
Input: { case_state, reason_code_history, channel_success_stats,
time_of_day, case_age }
Output: { action: CALL|SMS|WHATSAPP|ESCALATE|WAIT, policy_version }
v1 implementation: a literal decision table, e.g.:
current_state last_reason_code case_age_days → action
CONTACTED NO_PICKUP <3 CALL (different time-of-day)
CONTACTED NO_PICKUP >= 3 SMS
NEGOTIATING SOFT_NO_RETRY any CALL after cooldown
any DISPUTE_RAISED any ESCALATE
Keep this table in the policy_versions artifact (versioned, reviewed, git-tracked) — this is the thing Loop 2 Stage C
produces new versions of. Do not hardcode it in n8n; load it at runtime from the policy_versions row marked
status = 'promoted' for that product line.
v2 (later, once you have 2–3 months of reason-coded data): replace the decision table lookup with a scored ranking
model behind the same interface ({action, policy_version} in, same shape out) — this is why the interface
contract matters more than the implementation on day one.
2.4 Execution routing (n8n)
n8n's job is thin: given {action, case_id}, call the right downstream system and post the result back. Keep n8n
workflows to:
• One workflow per action type (CALL, SMS, WHATSAPP, ESCALATE)
• A single shared "outcome webhook receiver" workflow that all of them funnel into, which calls apply_transition()
This keeps business logic (eligibility, NBA selection) out of n8n and execution/integration logic in n8n, which is what
n8n is actually good at.
2.5 Idempotency & retry handling
• Every action dispatch carries an idempotency_key (case_id + attempt_number) so a webhook retry from the
voice/messaging platform doesn't double-write a transition.
• apply_transition() should be safe to call twice with the same key — check for an existing transition with that key
before inserting.
2.6 Testing strategy
Layer Test type
Eligibility gate Unit tests, one per rule, including edge cases (exact boundary of DND
window)
apply_transition() pgTAP or equivalent — test every (state, reason_code) combination
produces the expected next state
NBA policy layer Table-driven unit tests against the decision table itself
End-to-end Synthetic case run through the full loop in staging, asserting final state
Layer Test type
matches expected
3. Loop 2 — Agent Improvement Loop, Implementation Detail
3.1 Stage A — Ingestion & labeling
[Trigger: case reaches terminal state, or nightly batch]
|
v
[Fetch transcript + full state_transitions history for case_id]
|
v
[LLM labeling pass] -> structured output:
{
sentiment_arc: [...],
objection_types: [...],
compliance_flags: [...],
off_script_turn_index: int | null,
confidence: float
}
|
v
[Write to case_labels table, linked to case_id + transcript_id]
|
v
[Human calibration sample: 5-10% random sample routed to human QA
queue, compare against LLM label, track agreement rate over time]
create table case_labels (
id bigserial primary key,
case_id uuid not null references cases(id),
transcript_ref text not null,
labels jsonb not null,
labeler text not null, -- 'llm-judge-v2' | 'human:<name>'
agreement_check boolean, -- null unless part of calibration sample
created_at timestamptz not null default now()
);
Prompt design note: the labeling prompt should output against the same reason-code taxonomy from 1.1 wherever
possible (e.g. objection_types should be a subset of taxonomy codes, not free-form categories the LLM invents) —
this is what keeps Stage A and Stage B joinable.
3.2 Stage B — Failure clustering
Two complementary passes, run on a schedule (weekly):
• Taxonomy-driven aggregation — simple GROUP BY on case_labels joined to state_transitions, e.g. "% of cases
with DISPUTE_RAISED that reached HUMAN_HANDOFF more than 2 turns late." Catches known failure modes,
feeds a metrics dashboard directly.
• Embedding-based clustering — embed transcript segments where off_script_turn_index is non-null, cluster
(HDBSCAN or similar), surface clusters above a minimum size to a human for naming. Catches failure modes you
haven't taxonomized yet. Feed newly-named clusters back into 1.1 as taxonomy candidates.
Output artifact: a ranked list of failure clusters with {cluster_id, description, case_count,
sample_case_ids, first_seen} — this is the queue Stage C works from.
3.3 Stage C — Hypothesis → change
[LLM drafts candidate change]:
- proposed prompt/config diff
- rationale referencing the cluster's sample cases
- risk flag (touches compliance-sensitive language? yes/no)
|
v
[Human review -- named approver, required]
|
v
[New row in policy_versions, status='draft', created_by=<approver>]
Do not let this stage auto-promote. The created_by field on policy_versions is what makes this defensible in an
audit — every change traces to a named human decision, even though an LLM drafted the candidate.
3.4 Stage D — Offline eval harness
Build this as an extension of your existing Langfuse setup, not a parallel system — Langfuse already has dataset and
experiment-tracking primitives, and your traces are already tagged with case_id/policy_version.
Three-layer eval, run in this order (fail fast on the cheap check):
1. Deterministic compliance gate (hard pass/fail, must be 100%)
- required disclosures present
- no prohibited language/promises
- implemented as regex/rule checks against transcript, fast and cheap
2. LLM-as-judge quality scoring (soft, trend metric)
- score against a rubric: negotiation quality, objection handling,
tone -- same rubric Stage A's labeling uses, for consistency
3. Simulated borrower replay (for larger changes / model swaps)
- an LLM plays a borrower persona (cooperative, disputing,
hostile, non-responsive)
- new policy_version handles the simulated conversation end-to-end
- score close-rate proxy against personas run through the current
promoted version
Held-out set: maintain a frozen regression set of ~50–100 representative real transcripts (anonymized) per product
line, refreshed quarterly, that every candidate policy version is replayed against before touching Stage E. This is your
regression suite — treat it like a test suite in CI, not a one-off spreadsheet.
Promotion criteria — decide and record before running the eval, not after, e.g.:
promote_if:
compliance_gate == 100% pass
AND llm_judge_score >= control_score - 0 (no regression)
AND simulated_close_rate >= control_close_rate * 1.0
Write the eval report artifact (eval_report_ref on policy_versions) as a structured JSON blob, not just a chat
summary — this is what gets pulled into a client audit package later.
3.5 Stage E — Canary / promotion
Maturity ladder, each gate a config flag on the policy_versions.status field:
draft -> shadow -> canary -> promoted -> (eventually) retired
• shadow: policy runs in parallel on live cases, output logged (what action would this version have taken), never
dispatched. Zero production risk. Minimum 1–2 weeks or N cases, whichever is larger.
• canary: policy_version actually drives NBA selection for a small % of new cases (start with a low-stakes segment,
not your top-tier client). Control group runs the currently-promoted version on a matched cohort.
• promoted: canary beat control on the pre-declared criteria with zero compliance regressions. Old version moves
to retired but is never deleted (needed for audit trail / rollback).
Rollback plan: because policy_versions is versioned and cases.policy_version is stamped per case, rolling
back is just promoting the previous version — no data migration needed. This is why versioning policy as data (not as
a code deploy) matters.
4. Interface Contract Between Loop 1 and Loop 2
This is the seam — get this right and the two loops can be built by different people in parallel.
Loop 1 provides → Loop 2 consumes Loop 2 provides → Loop 1 consumes
state_transitions (reason-coded, append-only) policy_versions rows with status='promoted'
Linked transcripts/traces (Langfuse) Eval reports (for audit, not consumed
programmatically by Loop 1)
cases.policy_version (which version drove each —
case)
Loop 1 only ever reads the latest promoted policy_versions row for its product line at the start of each NBA decision.
It never reads draft/shadow/canary rows directly during normal execution (shadow-mode execution is a separate,
explicit parallel run, not a branch inside the main loop).
5. Observability & Audit Requirements
Build these in from day one, not bolted on later.
• Every transition traceable to a specific policy version and reason code — no exceptions, no "system" catch-all
reason codes in production.
• Every policy_version traceable to a named human approver (Stage C) and an eval report (Stage D).
• Dashboards (extend the existing agent performance dashboard rather than building new): close rate by
policy_version, compliance flag rate by policy_version, time-in-state distribution per product line.
• Retention: state_transitions and policy_versions are append-only and should never be purged — this is your
evidence package for any future audit.
6. Engineering Task Breakdown by Phase
Phase 1 — Foundations (shared, blocks everything else)
☐ Freeze reason-code taxonomy v1.0.0 (cross-functional session, not solo)
☐ Create cases, state_transitions, policy_versions, reason_codes tables in Supabase
☐ Write and unit-test apply_transition()
☐ Wire Langfuse tagging convention (case_id, policy_version, reason_code) into existing trace logging
Phase 2 — Loop 1 MVP (pick one product line, recommend collections)
☐ Build eligibility gate as standalone tested module
☐ Build v1 NBA decision table + loader from policy_versions
☐ Build n8n execution router workflows (CALL / SMS / WHATSAPP / ESCALATE) + shared outcome webhook
☐ Build state re-eval scheduler job
☐ End-to-end staging test with synthetic cases
Phase 3 — Instrumentation
☐ Confirm every transition produces a Langfuse-linked trace
☐ Build the case_labels table + LLM labeling job (nightly batch)
Phase 4 — Loop 2 Stage A/B
☐ LLM labeling pipeline + human calibration sampling (5–10%)
☐ Taxonomy-driven aggregation dashboard
☐ Embedding clustering pass (weekly job)
Phase 5 — Loop 2 Stage C/D
☐ Candidate-change drafting workflow + human approval UI (can be as simple as a reviewed PR-style flow
initially)
☐ Compliance gate checks (deterministic)
☐ LLM-judge quality scoring against frozen held-out set
☐ Simulated borrower replay harness (personas: cooperative, disputing, hostile, non-responsive)
Phase 6 — Loop 2 Stage E
☐ Shadow-mode execution path (parallel, non-dispatching)
☐ Canary cohort splitting + control comparison reporting
☐ Promotion/rollback tooling wired to policy_versions.status
☐ Wire results into agent performance dashboard (replace mock data)
7. Tech Stack Decisions Summary
Component Choice Why
State store Supabase/Postgres, single transition Atomicity, audit trail, no service
function owns state directly
Orchestration n8n Already in use, good for thin
execution routing, not business
logic
Eligibility/NBA logic Standalone versioned service/module, Testability, auditability
not embedded in n8n
Eval harness Extend Langfuse Already in stack, tagged traces,
datasets/experiments avoids a parallel system
Policy versioning Data rows (policy_versions table), not Instant rollback, no migration
code deploys needed
Clustering Taxonomy aggregation (SQL) + Covers known and unknown failure
embedding clustering (batch) modes
8. Open Questions for the Team to Resolve Before Phase 1
• Who has final sign-off authority on taxonomy changes post-v1 (single owner vs. review committee)?
• What's the actual DND/NCPR ruleset we encode in the eligibility gate — is there an existing compliance doc to
source this from, or does it need to be authored?
• What's the minimum case volume before Stage B's embedding clustering produces meaningful clusters (worth a
quick data check before committing engineering time)?
• Do we build the human approval UI for Stage C, or is a lightweight git-PR-based review sufficient for v1?