📝 Study Notes: AI Agents in Production
1. Infrastructure Options (Where to Host)
Choosing a compute platform is a trade-off between customization
control and management overhead.
Agent Runtime (Gemini Enterprise Agent Platform)
o Best for: Code-first agents built via the Agent Development
Kit (ADK).
o Key Specs: Fully managed; features native session state
management and an Agent Memory Bank (long-term
semantic memory).
o Deployment: Fast, single-action deployments using the
agents-cli.
Cloud Run
o Best for: Custom agent architectures, microservices, or
specific runtime dependencies.
o Key Specs: Serverless container environment; automatically
scales down to zero when idle (pay-per-use); easily exposes
agents via REST APIs or web UIs.
Google Kubernetes Engine (GKE)
o Best for: Multi-component agent systems or hosting your own
open-source models.
o Key Specs: Massive container orchestration; native support for
specialized AI hardware (GPUs/TPUs); features AI-aware
autoscaling with the GKE Inference Gateway.
App Engine
o Best for: Backends supporting standard Conversational Agents
(Dialogflow).
o Key Specs: Easy environment configuration via [Link]; ideal
for embedding directly into simple frontends like Dialogflow
Messenger.
Compute Engine
o Best for: Migrating legacy systems or stateful, long-running
processes.
o Key Specs: Raw Virtual Machines (IaaS); complete control over
the OS and kernel, but requires you to manage patching,
networking, and manual scaling.
2. The Production Security Architecture
Security in agentic systems focuses on restricting what the agent can do
and access.
Identity (IAM): Human users are authenticated via standard IAM
rules to access frontends, while the agent itself is granted a Service
Account to run background tools under the principle of least
privilege.
Perimeter Security (VPC Service Controls): Enforces a virtual
wall around critical enterprise infrastructure (like vector databases
and document repositories) to eliminate data exfiltration risks.
Scoped Credentials: Constraints the agent's external tool usage.
When an agent calls a database or a third-party webhook, its access
token is scoped only to the data mandatory for that specific sub-
task.
3. Production Observability Stack
Because LLM outputs are inherently non-deterministic, monitoring needs
to watch both the infrastructure and the logic.
Cloud Logging: Tracks a structured audit trail of every single user
prompt, tool call, and failure code.
Cloud Trace: Maps end-to-end user latency. It breaks down
timelines visually so you can spot whether a delay is coming from
the LLM generation, an API webhook, or a vector retrieval.
Cloud Monitoring: Measures infrastructure health metrics like
Queries Per Second (QPS) and server response latencies.
Agent Analytics: Measures business metrics unique to AI
workloads, such as human escalation rates, conversation resolution
success, and tool failure frequencies.
4. Enterprise Lifecycle Framework
Crucial Rule: Treat prompts, system instructions, and model selections
exactly like high-stakes software code.
1. Strict Versioning: Never tweak prompts "live" in production. Use
Agent Runtime or Cloud Run version snapshots to lock your code,
prompt configurations, and model versions to fixed baselines.
2. Segregated Environments: Deploy updates to isolated staging
areas first. Give internal QA and adversarial red-teams time to test
for behavioral drift or prompt-injection vulnerability.
3. Instant Rollbacks: Keep a zero-friction rollback path ready. If a
model update introduces unexpected hallucinations or crashes, you
need to be able to drop back to the previous stable state instantly