Enterprise AI Multi-Agent Platform

Complete Build → Deploy → Runtime Guide

All company/product names masked
Reference document compiled from project notes

Platform Overview
10 microservices deployed to Cloud Run, communicating via A2A (Agent-to-Agent) JSON-RPC protocol. All
services are private (--no-allow-unauthenticated), VPC-internal only. Identity bridged from Azure AD to GCP via
Workforce Identity Federation.

# Service Purpose / Key Details

1 BACKEND_API_SERVICE Auth token exchange + RAG search backend. Deploy FIRST. Called by
knowledge, asset-recovery, and DL agents.
2 KNOWLEDGE_AGENT RAG Q&A over enterprise docs. Calls Service 1. Supports text (SSE) and
voice (non-streaming) modalities.
3 ITSM_SUPERVISOR IT ticket management. 17 files. In-process sub-agents: incident_builder,
snow_ticket, trending_issue, csat. Unique cache-scheduler workflow.
4 HR_BENEFITS_AGENT Employee benefits list read/write. Calls Identity Graph API + HR Gateway.
Ethical guardrails embedded in agent instructions.
5 APP_CATALOG_AGENT Enterprise software search/install/status. Voice blocked at entry. Uses
IntentDetectionSubAgent (LLM-in-LLM) for routing.
6 ASSET_RECOVERY_AG Device return + TAR requests. Dynamic instruction injection
ENT (InstructionProvider). skip_summarization bypasses LLM re-summary.
7 DL_AGENT Email distribution list management. Uses LLM_MODEL_LITE (cost
optimization). Dynamic prompting with dept DL context.
8 SUPERVISOR_AGENT Central conversational router. Deploy LAST. Generates own OAuth2
tokens. Routes all user requests to domain agents.
9 UTILITIES_API Config, DL feature flags. (Details in subsequent pages.)
10 JD_CREATION_SERVICE HR job description generation. (Details in subsequent pages.)
Phase 0 — Platform-Wide One-Time Setup
Do this ONCE before any service is built. Creates all shared infrastructure.

0.1 — Cloud Provider Projects

• [CLOUD_PROJECT_NONPROD] → dev / tst / stg environments
• [CLOUD_PROJECT_PROD] → prod environment
Enable in both: Cloud Run API, Artifact Registry API, Secret Manager API, Firestore API, Memorystore (Redis)
API, Vertex AI API, Discovery Engine API, IAM API, Cloud Storage API.

0.2 — Identity Provider Setup (Azure AD)

• [PLATFORM]_NONPROD: Client ID, Client Secret, Tenant ID, Scope → [AZURE_WORKFORCE_SCOPE]
• [PLATFORM]_PROD: separate app registration, same pattern
• ITSM team uses separate registration: [ITSM]_NONPROD with own Client ID + Secret

0.3 — GCP Workforce Identity Federation

• Create workforce pool: gcloud iam workforce-pools create [WORKFORCE_POOL_ID] --
organization=[ORG_ID] --location=global
• Create OIDC provider linked to Azure AD: issuer-uri=[Link]

0.4 — Service Accounts (one per agent)

• Pattern: [env]-[service-name]-sa@[PROJECT].[Link]
• BACKEND_API_SERVICE: [Link], [Link], [Link],
[Link]
• KNOWLEDGE_AGENT: [Link] (to call BACKEND_API), [Link], [Link]
• ITSM_SUPERVISOR: [Link], [Link], [Link], [Link],
[Link]
• SUPERVISOR_AGENT: [Link], [Link], [Link], [Link],
[Link] (to call ALL domain agents)

0.5 — Secret Manager

• itsm-api-key → [ITSM_SYSTEM] Basic Auth base64
• hr-gateway-api-key → [HR_GATEWAY] API key
• identity-client-secret → Azure AD client secret
• search-engine-id → [CLOUD_SEARCH_ENGINE] engine ID
• agent-registry-api-key → internal agent registry key

0.6 — Redis (Cloud Managed Cache)

• Nonprod: Standard tier, standalone mode | Prod: Enterprise tier, cluster mode
• Used by: ITSM_SUPERVISOR, SUPERVISOR_AGENT, BACKEND_API_SERVICE
• Env vars: REDIS_HOST, REDIS_PORT

0.7 — Firestore (Session Database)

• Native mode. Connection URL: A2A_SESSION_DB_URL
• Used by: ITSM_SUPERVISOR, SUPERVISOR_AGENT

0.8 — VPC Networking

• VPC: [PLATFORM]-network-vpc-[env] with 2 subnets: agent subnet + backend subnet
• All services deploy with --no-allow-unauthenticated (internal VPC only, no public internet)
0.9 — Artifact Registry
• JFrog Artifactory SaaS: build + push (nonprod). Registry: [REGISTRY_URL]/[PLATFORM]-docker-np-loc
• GCP Artifact Registry: promotion path for prod.
[CLOUD_REGION]-[Link]/[PROJECT]/[PLATFORM]-containers

0.10 — IaC State Bucket

• gsutil mb gs://[IaC_TOOL]-state-[PLATFORM]-nonprod
• gsutil mb gs://[IaC_TOOL]-state-[PLATFORM]-prod

0.11 — CI/CD Repository Setup

• Each service = separate GitHub repo
• GitHub Environments: dev, tst, tst2, stg, prd
• Per-env variables: GCP_PROJECT_ID, ARTIFACT_REPO, TF_STATE_BUCKET,
GCP_DEPLOYER_SA_EMAIL
• GitHub Secrets (per team):
[PLATFORM]_NONPROD_AZURE_TENANT_ID/CLIENT_ID/CLIENT_SECRET/WORKFORCE_SCOPE +
GCP_PROJECT_NUMBER/WORKFORCE_POOL_ID/PROVIDER_ID/DEPLOYER_SA_EMAIL (mirror set
for _PROD)
CI/CD Standard Pattern (All Services)
Every service uses the same 3 GitHub Actions workflows unless noted otherwise.

Workflow 1 — [Link]
• on: workflow_dispatch | inputs: environment (dev/tst/stg/prd), apply (bool)
• Steps: POST [IDENTITY_PROVIDER]/oauth2/token → AZURE_JWT → gcloud auth login (workforce pool)
→ add-iam-policy-binding (impersonate SA) → TF init → plan → apply -auto-approve

Workflow 2 — [Link]
• Image tags: branch → [name]:[branch]-[TIMESTAMP] | semver tag → [name]:[semver]
• Build + scan via shared CI workflow. CVSS ≥ 9 → block (CVE gate)
• Push to JFrog Artifactory (nonprod)

Workflow 3 — [Link]
• Copy image: JFrog → GCP Artifact Registry → gcloud run deploy
• --no-allow-unauthenticated --no-traffic → then update-traffic 100%
• Prod gate: only runs on refs/tags/vX.Y.Z
• Promotion: dev → tst → stg → prd

Execution Order (per service)

• 4.1 Phase 0 complete • 4.2 Add secrets • 4.3 Write code files in order • 4.4 Write CI/CD files
• 4.5 Workflow 1 plan → review • 4.6 Workflow 1 apply • 4.7 Workflow 2 build • 4.8 Workflow 3 deploy (dev)
• 4.9 Test: curl [SERVICE_URL]/health → {"status":"healthy"} • 4.10 Promote dev → tst → stg → prd
Service 1 — BACKEND_API_SERVICE
Purpose Auth token exchange + RAG search backend
Deploy order FIRST — must be live before Knowledge, Asset-Recovery, DL agents
Stack FastAPI + uvicorn, redis[asyncio], httpx, opentelemetry-*, google-auth
Port 8080
Instances min 1 / max 10

Key Code Files

2.5 app/services/auth_service.py — CRITICAL
get_gcp_token(azure_token) → str:
• redis_key = f"gcp_token:{hash(azure_token)}" — check Redis cache
• HIT → return cached token
• MISS → POST [AUTH_PROXY_ENDPOINT] { subject_token: azure_token, grant_type: "urn:ietf:...token-
exchange", audience: [WORKFORCE_POOL_AUDIENCE] }
• → STS token → impersonate [SERVICE_ACCOUNT] → access_token
• redis_set(redis_key, access_token, TTL=3600) → return

2.6 app/services/knowledge_service.py — CORE

load_datastore_config() — 3-layer cache: Redis → in-memory → Secret Manager (search-engine-id)
search(query, azure_token, conversation_id, user_location):
• get_gcp_token(azure_token) → preamble = "User is located in {user_location}"
• POST [SEARCH_API]/v1alpha/projects/.../collections/default_collection/engines/[ENGINE_ID]/
servingConfigs/default_search:search
• { query, pageSize: 10, queryExpansion: {condition: "AUTO"}, session:
"projects/.../sessions/{conversation_id}" }
• build_context_sources_from_references(results) → inject [refLink:N] markers → return { text,
context_sources, conversation_id }

API Endpoints
• POST /knowledge — full JSON response
• POST /knowledge/stream — SSE streaming
• GET /health

Phase 5 — Runtime Decision Tree

• azure_token present? NO → 401 STOP
• YES → [Link]("gcp_token:{hash}") — HIT → use cached token
• MISS → POST AUTH_PROXY → STS exchange → SA impersonate → cache TTL=3600
• Load datastore config (3-layer) → streaming=True? → SSE | NO → full JSON
• SEARCH_ENGINE OK? NO → { error: "search unavailable" } → 500
• YES → build_context_sources → inject [refLink:N] → return { text, context_sources, conversation_id }
Service 2 — KNOWLEDGE_AGENT
Purpose RAG Q&A over enterprise docs
Calls BACKEND_API_SERVICE (/knowledge and /knowledge/stream)
Stack google-adk, httpx, tenacity, opentelemetry-*
Framework Google ADK — get_fast_api_app(a2a=True) publishes agent card

Key Code Files

2.2 src/agent/[Link]
attempt_with_retry(cfg, fn): @retry stop=3 attempts, wait=exponential(min=1s, max=10s), reraise=False
All retries fail → raise KnowledgeFetchError("All retries exhausted")

2.3 src/agent/[Link] — CORE

• State keys: AZURE_TOKEN_STATE_KEY="atoken", MODALITY_TYPE_STATE_KEY="modality"
• class KnowledgeAgent(BaseAgent) — _run_async_impl(ctx) → AsyncGenerator[Event]
• voice (modality=="voice"): attempt_with_retry → _fetch_knowledge(streaming=False) →
_sanitize_voice_response(text) → yield Event(is_final=True)
• text: async for chunk in _handle_streaming_request() → yield Event(content=chunk)
• Write-back: [Link]["conversation_id"] + [Link]["context_sources"]

_sanitize_voice_response(text)
• Remove [refLink:N] markers, remove URLs ([Link] remove **bold** markdown → [Link]()

2.5 src/agent/[Link] — agent card

• "name": "[KNOWLEDGE_AGENT]", "capabilities": { "streaming": true }

Phase 5 — Runtime Decision Tree

• A2A JSON-RPC → extract atoken + modality from session state
• modality=="voice" → non-streaming path with retries (max 3, exponential backoff)
• else → SSE streaming path → yield chunks → join
• Write session state: conversation_id, context_sources → yield Event(is_final=True) → A2A response to
[SUPERVISOR_AGENT]
Service 3 — ITSM_SUPERVISOR
Purpose IT ticket management with in-process sub-agents
Complexity 17 code files — most complex service
Extra IAM roles/[Link] (Firestore) + roles/[Link]
([ITSM_SYSTEM] key)
Unique CI/CD Workflow 4: cache-scheduler (cron every 30 min)

Sub-agent Architecture
• Root: itsm_supervisor (LlmAgent) → orchestrates 4 sub-agents
• incident_builder_agent: create/open/file tickets — tools: viewTickets, createTicket, docTicket,
validateAssignment, createDirectAssignment
• snow_ticket_agent: manage existing tickets — tools: view_open_incidents, add_comment, escalate,
withdraw, reopen, send_file, check_status
• trending_issue_agent: pre-warmed from Redis cache
• csat_agent: satisfaction survey — terminal state of all conversations

Key Code Files

2.3 session_state.py — BUILD FIRST
State key constants: invocation_id, user_info, atoken, msid, employee_id, livechat_active, modality,
conversation_id
Methods: get/set_livechat_active(ctx), is_voice_modality(ctx), get_user_info(ctx)

2.6 custom_patching.py — IMPORT BEFORE ADK

Monkey-patches ADK user_id extraction. Priority chain:
• 1. call_context.user.user_name • 2. [Link]["user_id"] • 3. [Link]["user_id"] • 4.
fallback: "SYSTEM_USER_{ctx}"

2.7 viewTickets_tool.py
check_duplicate_incidents(): GET [ITSM_SYSTEM]/api/now/table/incident?caller_id=[MSID]^state<7
Internal LLM call (SEMANTIC_CHECK): strip verbs → extract noun anchor → compare similarity against open
tickets

2.8 createTicket_tool.py
Validate: contact_time empty → error | method=="phone" → extract digits only, len<7 → phone_retry_count++
POST [ITSM_SYSTEM]/api/now/table/incident → { number: "INC...", sys_id: "[UUID]" }

2.16 src/agent/[Link] — ROOT SUPERVISOR

• before_tool_run: [Link]["active_agent"] = [Link]
• before_model_run: deduplicate history, bypassLLM check, modality → load voice/text instructions
• itsm_supervisor = LlmAgent(sub_agents=[incident_builder, snow_ticket, trending_issue, csat])

2.17 src/[Link] — ENTRY POINT

• import custom_patching — MUST be before any ADK import
• get_fast_api_app(session_service_uri=A2A_SESSION_DB_URL,
memory_service_uri=A2A_MEMORY_URL, artifact_service_uri=A2A_ARTIFACT_URL, a2a=True)

Phase 5 — Runtime Decision Tree

• custom_patching resolves user_id (4-priority chain)
• before_model_run: deduplicate → bypassLLM? → voice/text instructions
• LLM intent: "create/open/file" → incident_builder | "view/check/comment/escalate/update/attach" →
snow_ticket
• incident_builder: routing phrase → WF 1.0 | "doc ticket" → WF 1.2 | default → WF 1.1 (check_duplicates →
contact method → phone validation up to 3x → contact time → create)
• snow_ticket: ownership gate → action (VIEW/COMMENT/ESCALATE/WITHDRAW/REOPEN/ATTACH) →
"anything else?" → csat_agent
• Workflow 4 (cache-scheduler): every 30 min → warm Redis with trending issues + sessions. Redis lock
nx=True prevents concurrent runs
Service 4 — HR_BENEFITS_AGENT
Purpose Employee benefits list read/write
APIs Identity Graph API (user resolution) + HR Gateway (benefits operations)
Files 5 code files
Key feature Ethical guardrails embedded as code in behavior_guidelines.py

Key Code Files

2.1 tools/[Link] — BUILD FIRST
resolve_user_identity(name_or_email_or_id):
• email format → GET [IDENTITY_GRAPH]/v1.0/users?$filter=mail eq '{email}'
• digits only → direct employee ID lookup
• name string → GET users?$search="displayName:{name}" ConsistencyLevel: eventual
• Multiple results → multi-turn: LLM asks user to pick one | Single → return { object_id, employee_id, email,
display_name }

2.2 tools/[Link] — CORE (14 tools)

• check_group_membership(user_object_id, group_id) → POST checkMemberGroups → returns
[ADMIN_GROUP_ID] if member, [] if not
• get_benefits_list(requestor_id) → GET [HR_GATEWAY]/benefits/v2.0/supportListOf/{requestor_id} Auth:
Bearer [CLOUD_SA_TOKEN]
• update_benefits_list(requestor_id, add_ids): self-add guard (add_ids contains requestor_id? → reject) →
admin check (NOT member → reject) → PUT
• remove_from_benefits_list(requestor_id, remove_ids): same guards → PUT with remove: [ids]
• get_employee(employee_id) → GET [HR_GATEWAY]/employee/v2.0/profile/{id}
• search_departments(query) → GET [HR_GATEWAY]/org/v2.0/department/search?q={query}
• + 8 more tools following same HR_GATEWAY pattern

2.3 tools/behavior_guidelines.py
• ETHICAL_GUARDRAILS constant embedded in agent instructions
• Cannot add person without consent rules | Cannot view another person's list without authorization | Audit
trail requirements

Phase 5 — Runtime Decision Tree

• LLM identifies action: show/view → get_benefits_list | add [person] → resolve_user_identity → update |
remove [person] → resolve_user_identity → remove | find employee → get_employee | find dept →
search_departments
• Write action: self-add guard → check_group_membership → NOT admin → reject | admin → PUT →
confirm
• Read action: GET → format response → return
• All paths → A2A response → [SUPERVISOR_AGENT] → User
Service 5 — APP_CATALOG_AGENT
Purpose Enterprise software search, install, status
Voice BLOCKED at entry point — yields unsupported response immediately
Key pattern IntentDetectionSubAgent: internal LLM call with structured JSON output
schema
Intents search | install | status | generic

Key Code Files

2.1 [Link]
• IntentDetectionOutput(BaseModel): intent: str, application_query: str
• AppSearchResponseBody(BaseModel), AppInstallResponse(BaseModel)

2.3 catalog_client.py
• search_app(query) → GET [APP_CATALOG_API]/apps/search?q={query} Auth: Bearer
[CLOUD_SA_TOKEN]
• submit_install(app_id, user_id) → POST [APP_CATALOG_API]/install { app_id, user_id, device_id }
• get_install_status(request_id) → GET [APP_CATALOG_API]/requests/{request_id}

2.5 src/agent/[Link]
• IntentDetectionSubAgent: internal LlmAgent(model=[LLM_MODEL],
system=intent_detection_instruction.txt, output_schema=IntentDetectionOutput)
• AppCatalogAgent(BaseAgent): voice? → yield create_voice_unsupported_response(); return
• intent_result = await intent_sub_agent.detect(user_message) → # { intent: "install", application_query:
"Zoom" }
• search → catalog_client.search_app() | install → search + multi-turn confirm + submit_install | status →
extract_request_id → get_install_status

Phase 5 — Runtime Decision Tree

• voice? → "Sorry, app installation not supported via voice" → STOP
• IntentDetectionSubAgent classifies: search → GET catalog/search → format | install → search + confirm
(yes/no) + POST install → { request_id, status: "pending" }
• status → extract_request_id → GET requests/{id} → show state + ETA | generic → LLM generates general
app catalog help
Service 6 — ASSET_RECOVERY_AGENT
Purpose Device return + tech asset return (TAR) requests
Key patterns InstructionProvider (dynamic instruction injection) + skip_summarization
Backends ASSET_MGMT_API (returns/TAR) + HR_GATEWAY (employee lookup) +
BACKEND_API_SERVICE (RAG)

Key Code Files

2.2 instruction/[Link] — InstructionProvider
get_instruction(context): merge STATIC_INSTRUCTION + user's device list + return eligibility dates
Instructions are dynamically generated per-request, not static.

2.4 tools/asset_recovery_api.py
• get_employee_name_by_id(id) → GET [HR_GATEWAY]/employee/v2.0/profile/{id}
• submit_asset_return(employee_id, asset_tag, return_type, shipping_address) → POST
[ASSET_MGMT_API]/returns → { request_id, confirmation_number }
• submit_tar_request(employee_id, reason, approval_code) → POST [ASSET_MGMT_API]/tar-requests
• validate_and_format_date(date_string): parse → ISO 8601 | future → valid | past → error

2.5 tools/knowledge_tool.py
• asset_recovery_knowledge_base_response(query) → calls [BACKEND_API_SERVICE]/knowledge (same
RAG backend)
• asset_recovery_box_label_form() → GET [ASSET_MGMT_API]/forms/box-label → PDF URL
• tar_knowledge_base_response(query) → same RAG with TAR-specific preamble

2.6 [Link] — modify_output_after_tool

skip_summarization enabled? → YES: return tool_response directly (bypass LLM re-summary) | NO: return None
(let LLM summarize)
asset_recovery_agent = LlmAgent(instruction=InstructionProvider(),
after_tool_callback=modify_output_after_tool)

Phase 5 — Runtime Decision Tree

• InstructionProvider injects device list + eligibility dates into prompt
• "return my device" → collect asset_tag, shipping_address, date → validate_and_format_date →
submit_asset_return → show confirmation + box label link
• "TAR request" → collect reason, approval_code → submit_tar_request → { request_id }
• "box label" → box_label_form() → PDF URL | "what is the process" → knowledge_base_response → RAG
(skip_summarization → emit directly)
Service 7 — DL_AGENT
Purpose Email distribution list management
Model LLM_MODEL_LITE (cost optimization — simpler task)
Key pattern Dynamic prompting: pre-loads user's current DL memberships + dept context
per call

Key Code Files

2.1 dynamic_prompting.py
get_dynamic_instruction(context): inject user's current DL memberships + department + common DL suggestions
for dept

2.2 tools/dl_api.py
• search_distribution_lists(query) → GET [UTILITIES_API]/dl/search?q={query} → [{ id, name, email,
member_count }]
• get_user_distribution_lists(user_id) → GET [UTILITIES_API]/dl/members/{user_id}
• add_user_to_dls(user_id, dl_ids) → POST [UTILITIES_API]/dl/members { dl_ids, user_id }
• remove_user_from_dls → DELETE [UTILITIES_API]/dl/members { dl_ids, user_id }

2.3 tools/knowledge_tool.py
dl_knowledge_base_response(query) → [BACKEND_API_SERVICE]/knowledge (same RAG backend)

2.4 [Link]
dl_agent = LlmAgent(model=[LLM_MODEL_LITE], instruction=get_dynamic_instruction,
after_agent_callback=modify_output_after_agent)

Phase 5 — Runtime Decision Tree

• dynamic_prompting injects user's memberships + dept
• [LLM_MODEL_LITE] identifies: "show my lists" → get_user_distribution_lists → display
• "add me to [name]" → search → not found: reject | found: multi-turn confirm → "yes" → add_user_to_dls →
confirm
• "remove me from [name]" → search → membership check → multi-turn confirm → remove
• "what lists exist for [dept]" → search → show | "how do I manage DLs" → dl_knowledge_base_response →
RAG
Service 8 — SUPERVISOR_AGENT (partial)
Purpose Central conversational router — deploy LAST
Deploy order LAST — all domain agents (1-7, 9-10) must be live first
Key difference Generates own OAuth2 tokens (client credentials flow) rather than receiving
them from users

Phase 2 — Code Files (partial)

2.1 models/[Link]
• LlmAgentConfig(BaseModel), CallbackConfig(BaseModel)
• AccessControlConfig(BaseModel): enabled: bool, access_denied_message: str
• get_settings() → Settings (loaded from env vars)

2.2 authentication/[Link]
get_auth_header(audience: str) → dict:
• Step 1: POST [IDENTITY_PROVIDER]/oauth2/token { client_id, client_secret, scope, grant_type } →
azure_jwt
• Step 2: Exchange azure_jwt → [CLOUD_SA_TOKEN] (STS workforce pool exchange)
• return { Authorization: "Bearer [CLOUD_SA_TOKEN]" }
(Remaining files, CI/CD, and runtime decision tree to be documented from subsequent uploads)
Cross-Cutting Patterns & Key Design Decisions
Authentication Flow (Domain Agents)
• User presents Azure AD JWT in session state key "atoken"
• BACKEND_API_SERVICE exchanges it → GCP SA token via 3-hop: Azure JWT → STS token → SA
impersonation → GCP access_token
• Token cached in Redis with TTL=3600 seconds

Authentication Flow (SUPERVISOR_AGENT)

• Supervisor performs client-credentials OAuth2 itself (not user-facing)
• POST [IDENTITY_PROVIDER]/oauth2/token → azure_jwt → STS exchange → CLOUD_SA_TOKEN
• Used to call all downstream domain agents with Authorization: Bearer header

Modality Handling
• All agents receive modality from session state key "modality" = "text" | "voice"
• Voice path: non-streaming, sanitize response (strip [refLink:N], URLs, **bold**)
• Text path: SSE streaming, full markdown including citations
• Exception: APP_CATALOG_AGENT blocks voice entirely at entry

RAG Pattern (shared across Services 2, 6, 7)

• All call BACKEND_API_SERVICE/knowledge (or /knowledge/stream)
• Each passes its own azure_token + query + conversation_id
• Service 6 uses skip_summarization to emit RAG response directly without LLM re-summary
• Service 7 uses dl_knowledge_base_response for DL policy questions

A2A Response Chain

• User → SUPERVISOR_AGENT → routes to domain agent → domain agent calls tools/backends → A2A
response → SUPERVISOR_AGENT → User
• All domain agent responses are A2A JSON-RPC format

Dynamic Instructions vs Static

• Most services: static instructions loaded from [Link] or [Link]
• ITSM_SUPERVISOR: loads voice_instructions.md or text_instructions.md based on modality at runtime
• ASSET_RECOVERY_AGENT: InstructionProvider generates instructions per-request with user's device
context
• DL_AGENT: get_dynamic_instruction injects user's current DL memberships per-call

Cost Optimization
• DL_AGENT uses LLM_MODEL_LITE (lighter model for simple, well-scoped tasks)
• Redis token caching (TTL=3600) avoids repeated 3-hop auth exchanges
• BACKEND_API_SERVICE has 3-layer config cache: Redis → in-memory → Secret Manager
• ITSM_SUPERVISOR cache-scheduler pre-warms Redis every 30 min with trending issues

Security Guards
• HR_BENEFITS_AGENT: self-add guard + admin membership check before any write
• ITSM snow_ticket: ownership check (MSID must match caller_id) before all ticket actions
• APP_CATALOG_AGENT: voice blocked (install requires confirmation, unsuitable for voice)
• All services: --no-allow-unauthenticated, VPC-internal only
• Secret Manager: all credentials stored there, never in code or env vars