Here’s the uncomfortable truth almost every RAG tutorial skips: the standard RAG pipeline doesn’t actually work beyond the demo.

You know the recipe, embed your documents, retrieve the top-k chunks for a query, stuff them into a prompt, hope for the best. It looks magical on a curated example. Then you put it in front of real users and it falls apart: someone asks a two-part question and the single retrieval grabs context for half of it and makes up the rest. Someone uses an exact term your embedding model doesn’t understand and gets nothing. Someone asks something the docs don’t cover and the model confidently invents an answer with a citation that doesn’t exist.

These aren’t bugs you can tune away. They’re structural. Naive RAG retrieves once, blindly, before the model has thought about anything, and then it’s stuck with whatever it got. The system never gets to decide it should search again, search differently, read more, or admit it doesn’t know.

The fix isn’t a better embedding model or a bigger context window. It’s a different architecture: agentic RAG, where retrieval itself becomes agentic. The model decides when to search, rewrites its own queries, runs as many retrieval rounds as the question needs, pulls more context only where it’s thin, and, the part I care about most, refuses to cite a source it didn’t actually retrieve.

This post builds one end to end so you can see exactly why the agentic version wins. The project runs on LiteLLM (multi-provider, searching for a replacement though ;) ), LanceDB (vectors), full text search (BM25), and a local embedding + reranking stack. Let’s build it up piece by piece.

The shape of the thing

Before any code, here’s the mental model. There are two pipelines: an offline one that gets documents into searchable shape, and an online one that answers questions.

The dashed lines show where the two halves meet: the indexes built offline are what the agent’s execute tool step searches at question time. And the loop label matters, the model can cycle LLM → tool → result → LLM up to 10 times before it commits to an answer.

The key difference lives in that loop on the right. Naive RAG is a straight line: retrieve → generate, once, and you live with it. Agentic RAG is a cycle the model drives, and that single architectural change is what turns retrieval from a blind guess into a deliberate search.

Part 1, Ingestion: getting documents in

Nothing exotic here, but two decisions matter.

Chunk with overlap, and give every chunk a positional ID. I split text into 512-character chunks with 64 characters of overlap, and ID them sequentially:

def _chunk_id(doc_id: str, index: int) -> str:
    return f"{doc_id}__c{index:04d}"   # paper__c0000, paper__c0001, ...

That c0000, c0001 ordering isn’t cosmetic, it’s what lets the agent later say “give me the chunks around this one” to read more of a document. The ID encodes position.

Embed locally, and build two indexes. Each chunk is embedded with BAAI/bge-small-en-v1.5 (a small, fast, 384-dim model that runs on your machine, no embedding API bill), stored in LanceDB, and also indexed by BM25:

vectors = embed(texts, "BAAI/bge-small-en-v1.5")   # normalized, local
insert_chunks(db, chunks)                            # → LanceDB (semantic)
rebuild_index(collection, all_ids, all_texts)        # → BM25  (keyword)

Why two indexes? Because semantic search and keyword search fail in opposite ways, and we’re about to use that. Notice: LanceDB also has full text search, but I decided to implement it separately.

Part 2, Retrieval that doesn’t suck: hybrid search

Pure vector search is great at meaning (”car” matches “automobile”) and terrible at specifics, exact names, error codes, acronyms like “GNN-RAG” that don’t sit near anything in embedding space. Keyword search (BM25) is the mirror image: nails the exact term, misses the paraphrase.

The fix is to run both and merge the rankings. The merge trick is Reciprocal Rank Fusion (RRF), and it’s beautifully simple. You don’t try to reconcile the two scoring systems (cosine similarity vs. BM25 scores aren’t comparable). You throw the scores away and only use rank position:

def _rrf(semantic, keyword, k: int = 60):
    """Reciprocal Rank Fusion over two ranked lists, keyed by chunk_id."""
    scores = {}
    for rank, r in enumerate(semantic):
        scores[r.chunk_id] = scores.get(r.chunk_id, 0.0) + 1.0 / (k + rank + 1)
    for rank, r in enumerate(keyword):
        scores[r.chunk_id] = scores.get(r.chunk_id, 0.0) + 1.0 / (k + rank + 1)
    return scores

A chunk that ranks #1 in both lists rises to the top. A chunk that’s #1 in one and absent from the other still scores respectably. The k=60 constant dampens the contribution of low-ranked items, it’s the standard value from the original paper and you rarely need to touch it.

Then one more pass: a cross-encoder reranker (ms-marco-MiniLM). The bi-encoder embeddings we used for retrieval encode the query and document separately, fast, but it can’t model how they relate. A cross-encoder reads the query and a candidate chunk together and scores the pair directly. It’s too slow to run over the whole corpus, but perfect for re-ranking the ~10 candidates RRF handed us down to the best 5.

This whole stack, embed, search, fuse, rerank, is one tool call from the agent’s point of view. Which brings us to the interesting part.

Part 3, The agent loop

Here’s the heart of the system. Instead of retrieving once, the agent runs a ReAct loop: reason, act, observe, repeat. Stripped down, it’s this:

for iteration in range(MAX_ITERATIONS):          # cap at 10
    stream = await stream_completion(messages, tools, ...)

    # ... collect the model's response: either text or tool calls ...

    if finish_reason == "stop":                  # model wrote an answer
        correction = validate_citation(answer, retrieved_ids)
        if correction:                           # caught a bad citation
            messages.append(correction)          # tell it to fix, loop again
            continue
        return answer                            # clean — we're done

    if tool_calls:                               # model wants to use a tool
        for call in tool_calls:
            result = dispatch_tool(call.name, call.args)
            messages.append(result)     

That’s the entire control flow. Every iteration, the model looks at the conversation so far, including the results of any searches it already ran, and decides the next move. Search again with a better query? Pull more context around a promising hit? Or does it finally have enough to answer?

This is what “agentic” actually means in practice. Not a personality, not magic, just a loop where the model chooses the next action and sees the consequences before choosing again. Retrieval stops being a preprocessing step and becomes a decision the model makes, repeatedly, with feedback. That’s agentic retrieval, and it’s the whole ballgame.

A real trace for “What is GNN-LLM and what is query rewriting?” looks like:

Two searches, not one, because it’s a two-part question. Then a third call to read around the best hit, because one chunk wasn’t enough. A naive pipeline physically cannot do any of this. It gets one shot at one query and is done. This is the exact failure I opened with, and here’s the architecture that removes it.

Part 4, The tools: the agent’s hands

The agent can only do what its tools let it. I gave it four, deliberately small:

get_context is the quiet hero. When a search hit is relevant but too short to fully answer from, the agent fetches the surrounding chunks (using those positional IDs from Part 1) instead of dumping the entire 40-page document into the prompt. It reads deeply, but only where it’s worth it, which keeps the context window (and your token bill) under control.

There’s a hard budget enforced around every retrieval, too: once retrieved content crosses ~8,000 tokens, further searches are blocked and the model is told to answer with what it has. Agents left unsupervised will happily search forever.

Part 5, The part that builds trust: citation validation

This is the feature I’d fight to keep. An agent that retrieves well is useful. An agent that can’t lie about its sources is something you can put in front of users.

After the model writes an answer, a post-hook checks every [chunk_id] citation against the set of chunks actually retrieved during this session:

def validate_citation(answer, retrieved_ids):
    cited = parse_cited_ids(answer)               # IDs in [brackets]
    unknown = cited - retrieved_ids               # cited but never retrieved
    if unknown:
        return (f"Citation error: {unknown} were not in your retrieved results. "
                f"Valid chunk IDs you can cite: {sorted(retrieved_ids)[:6]}. ")
    return None   

If the model invents a citation, or cites something it merely “remembered” from training instead of retrieved, the answer is rejected. The correction goes back into the loop and the model has to revise, up to a few attempts before it gives up gracefully. The result: every bracketed citation in a final answer is guaranteed to point at real, retrieved source text. No silent hallucinated references.

(Getting this right took a few tries. My first version’s regex also matched footnote markers like [1] and [Smith 2023] inside the document text, poisoning the set of “valid” IDs and sending the agent into an apology spiral. Validation logic that runs on model output is worth testing carefully, the failure modes are subtle.)

Multi-provider, almost for free

One design choice paid off repeatedly: every LLM call goes through LiteLLM, so the provider is just a config string.

PROVIDERS = {
    "gemini-fast":     ProviderConfig(model="gemini/gemini-2.5-flash"),   # default
    "anthropic-smart": ProviderConfig(model="claude-sonnet-4-6", enable_thinking=True),
    "openai-fast":     ProviderConfig(model="gpt-4o-mini"),
    # ...
}

python main.py --provider anthropic-smart "What are the key risks?"

Same agent loop, same tools, same validation, swap the brain underneath. This is also how you A/B a cheap-and-fast model against a smart-and-slow one on your actual workload, which is the only benchmark that matters. (Each provider has its quirks, Gemini, for instance, tacks hidden “thought signature” tokens onto tool-call IDs mid-stream, which you have to strip before the conversation history will round-trip. The abstraction smooths the API, not the behavior.)

Seeing it think

A loop you can’t observe is a loop you can’t debug or trust. The agent streams every step, tool calls, arguments, results, and the final answer, to whatever UI is attached: a Rich-powered terminal view, or a Chainlit browser UI that renders each tool call as an expandable step above the streamed answer.

It matters more than it sounds. Watching the agent decide to run a second search because the first didn’t cover the whole question is the moment “agentic RAG” stops being a buzzword and starts being an obviously-better design.

What I’d tell you before you build your own

A few takeaways from building this:

• The loop is the whole idea. Everything else, hybrid search, reranking,

  budgets, is quality tuning. The thing that makes retrieval agentic is ten

  lines: decide, act, observe, repeat. That’s the line between “doesn’t work in

  production” and “does.”

• Hybrid search is non-negotiable for real documents. Vector-only retrieval

  will embarrass you the first time someone searches for an exact term or an

  acronym.

• Validate the model’s output, not just its input. Citation checking turned a

  “usually grounded” system into a “provably grounded” one, and it’s maybe 30 lines.

• Make it observable from day one. Most of my hardest bugs were invisible until I could watch the steps stream by.

• Keep the tools small and sharp. Four focused tools beat one giant

  do-everything tool the model has to reason about.

Stop shipping naive RAG

If there’s one thing to take from this: the one-shot retrieve-then-generate pipeline is a dead end for anything real. It demos beautifully and fails quietly, wrong on multi-part questions, blind to exact terms, and happy to hallucinate a citation when the docs come up empty. You can’t tune your way out of a structural flaw.

Agentic RAG fixes it at the root by making retrieval a decision the model makes, not a step that happens to it. The model searches when it needs to, searches again when the first attempt falls short, reads deeper only where it pays off, and can’t claim a source it didn’t pull. None of it is exotic, it’s a loop, two indexes, and a validator. But that combination is the difference between a demo and a system you’d actually put in front of users.

If you’re still building one-shot RAG, you’re building the version that breaks. Build the agent instead.

Built with LiteLLM, LanceDB, BM25 + cross-encoder reranking, and a strict citation validator. The full project is on GitHub mallahyari/rag-agent-harness, and if you want the annotated, step-through version of the example trace above, there’s an interactive walkthrough in the project repo, interactive walkthrough.

In Part 2, we built a working agent harness with three real tools, read_file, write_file, and run_bash. The feedback loop worked. Errors came back as structured signals. The model self-corrected.

But I left something unaddressed on purpose, and it is time to confront it directly.

Our run_bash function was a subprocess.run() call on your local machine. No container. No isolation. No boundary between the model’s generated code and your host filesystem, environment variables, and network. If the agent wrote import os; os.environ into a script and executed it, it would see your API keys. If it ran curl to an attacker-controlled server, nothing would stop it.

This is not a hypothetical. In October 2024, security researcher Johann Rehberger demonstrated what he called the ZombAIs attack against Claude’s Computer Use feature. Using prompt injection embedded in a webpage the agent was browsing, he caused the agent to download and execute the Sliver C2 framework, effectively turning the AI into a remotely controlled zombie on the host machine, with no malicious intent from the user who launched it.

Giving an AI agent a shell is handing it a loaded weapon. Today we are going to look at how a harness builds a secure playground around that weapon.

The Golden Rule: The Harness Belongs Outside the Sandbox

Before containers or virtualization, one architectural principle must be non-negotiable: the agent harness must never run in the same execution context as the agent’s actions.

If your harness runs inside the sandbox, a compromised model execution can access your environment variables, modify your history logic, or exfiltrate your API keys. The harness is the warden; the sandbox is the cell. The warden must command from outside — passing instructions in, receiving exit codes and stdout back out.

The Sandboxing Spectrum: Four Isolation Models

Every sandboxing decision involves the same trade-off: isolation security vs. startup latency. Here is where the four primary models land in practice today:

A few clarifications worth calling out explicitly:

WebAssembly & Pyodide: The common claim that Wasm gives you sub-10ms startup is wrong for Python. Pyodide (Python compiled to Wasm) requires downloading ~6 MB of runtime and takes 1–3 seconds to initialize. Once running, it is also 3–5x slower than native CPython. The real advantage is mathematical isolation. Pyodide code cannot escape its Wasm memory boundary by design. It is a good fit for lightweight, browser-based execution but not for general-purpose agent tool calls.

Docker + gVisor: gVisor intercepts system calls in user space rather than passing them to the host kernel directly. This eliminates the biggest Docker security risk (kernel-sharing) while keeping Docker’s ergonomics. Google runs gVisor in production for Cloud Run. The tradeoff is ~10–20% runtime overhead and some syscall compatibility gaps.

Firecracker MicroVMs: Used by AWS Lambda and E2B (a cloud sandbox platform for AI agents processing ~15 million sandboxes/month as of 2025). Each agent gets its own kernel, not just a container namespace. Cold boot is ~90–200ms, and with VM snapshotting it drops to ~150ms for pre-warmed states. This is the production standard for hosted coding agents.

Filesystem Isolation: Setting Clear Boundaries

An agent refactoring a codebase needs file access. The question is which files.

Three rules make this work in practice:

1. Mount explicitly, never from root. Never bind-mount / or ~. Only mount the specific directory the agent is assigned to work in.

2. Use ephemeral copies for high-risk tasks. Copy the target repository into a temporary path (/tmp/agent-run-xyz) and mount that instead. When the agent finishes, diff the changes, present them to the user, then destroy the container. The original is never touched directly.

3. Run as a non-root user. Always run the container with a non-privileged user (--user 1000:1000). This prevents model-generated code from installing kernel modules, modifying network routes, or writing to system directories even if a container escape is attempted.

From Subprocess to Docker: Upgrading Your Part 2 Harness

In Part 2, the run_bash function was a bare subprocess.run(). Here is what the upgrade looks like — a drop-in Docker replacement that applies all the isolation rules above:

import subprocess

def run_in_docker_sandbox(command: str, workspace_path: str) -> tuple[str, bool]:
    """
    Execute a command inside an ephemeral Docker container.
    Drop-in replacement for the bare subprocess run_bash from Part 2.
    """
    docker_cmd = [
        "docker", "run",
        "--rm",                                      # destroy container on exit
        "--network", "none",                         # no network access by default
        "--memory", "512m",                          # hard memory cap
        "--cpus", "1.0",                             # hard CPU cap
        "--read-only",                               # root filesystem is read-only
        "--tmpfs", "/tmp:size=100m",                 # writable scratch space only
        "-v", f"{workspace_path}:/workspace:rw",    # mount only the workspace
        "-w", "/workspace",                          # set working directory
        "--user", "1000:1000",                       # non-root user
        "python:3.11-slim",
        "bash", "-c", f"timeout 30 {command}",      # inner timeout
    ]

    try:
        result = subprocess.run(
            docker_cmd,
            capture_output=True,
            text=True,
            timeout=35,    # outer timeout slightly longer than inner
        )
        output = (result.stdout + result.stderr).strip()
        return output or "(no output)", result.returncode != 0
    except subprocess.TimeoutExpired:
        return "Error: sandbox timed out", True
    except Exception as e:
        return f"Error: {e}", True

Notice what each flag does:

• --rm ensures the container is destroyed after each tool call, no state leaks between runs

• --network none cuts off all external network access entirely

• --read-only + --tmpfs means the agent can only write to /tmp and the mounted workspace, nothing else on the filesystem

• --user 1000:1000 ensures model-generated code runs without root privileges

• The double timeout (inner timeout 30 + outer timeout=35) guarantees the harness is never blocked by a runaway process

To plug this into the AgentHarness from Part 2, replace the run_bash function in TOOL_REGISTRY with a lambda that calls run_in_docker_sandbox with your workspace path.

Ready-Made Sandboxing Libraries

If you do not want to manage Docker configuration yourself, several open-source libraries handle the heavy lifting. Here are three worth knowing, each covering a different point on the isolation spectrum:

E2B (e2b-code-interpreter)

E2B is the most production-ready option for AI agent sandboxing. Under the hood it uses Firecracker microVMs — each sandbox gets its own kernel. The Python SDK makes it a near drop-in replacement for the Docker function above:

from e2b_code_interpreter import Sandbox

def run_in_e2b_sandbox(command: str) -> tuple[str, bool]:
    with Sandbox() as sandbox:
        result = sandbox.run_code(command)
        output = "\n".join(str(o) for o in result.logs.stdout)
        error  = "\n".join(str(e) for e in result.logs.stderr)
        is_error = bool(result.error)
        return (error if is_error else output) or "(no output)", is_error

Install with pip install e2b-code-interpreter. Requires an E2B API key. Best choice if you are building a cloud-hosted agent and want hardware-level isolation without managing infrastructure.

RestrictedPython

RestrictedPython takes a different approach — rather than isolating at the OS level, it restricts what Python code is allowed to do at parse time. You define exactly which builtins, imports, and operations are permitted before the code ever runs.

from RestrictedPython import compile_restricted, safe_globals

def run_restricted_python(code: str) -> tuple[str, bool]:
    try:
        byte_code = compile_restricted(code, "<string>", "exec")
        local_vars = {}
        exec(byte_code, safe_globals, local_vars)
        return str(local_vars.get("result", "(no result)")), False
    except Exception as e:
        return f"{type(e).__name__}: {e}", True

Install with pip install RestrictedPython. No containers needed — useful when Docker is overkill and you only need to prevent agents from importing os, subprocess, or sys. Not a replacement for OS-level isolation for untrusted code, but a solid lightweight layer for constrained use cases.

Monty (pydantic-monty)

Monty is the most interesting new entrant in this space — a minimal, secure Python interpreter written in Rust by the Pydantic team, designed specifically for running LLM-generated code. It starts in under a microsecond, requires no containers, and completely blocks access to the host filesystem, environment variables, and network by default. You control exactly which host functions the agent can call.

import pydantic_monty

def run_in_monty(code: str) -> tuple[str, bool]:
    try:
        result = pydantic_monty.run(code)
        return result.stdout or "(no output)", False
    except pydantic_monty.ExecutionError as e:
        return str(e), True

Install with pip install pydantic-monty. The tradeoff is intentional scope — Monty runs a subset of Python and does not support third-party libraries like NumPy or Pydantic itself. It is designed for agents that express logic in pure Python rather than calling into ecosystem packages. Worth watching closely: Pydantic plans to use it as the foundation for code execution in PydanticAI, and it is still marked experimental at the time of writing.

Deno (for JavaScript/TypeScript agents)

If your agent executes JavaScript or TypeScript, Deno has a built-in permission model that makes sandboxing a flag, not an architecture decision:

# Agent-generated script runs with explicit, minimal permissions only
deno run --allow-read=/workspace --allow-write=/workspace/output --no-prompt agent_script.ts

Deno denies all filesystem, network, and environment access by default. You opt-in to exactly what the agent needs. Install with curl -fsSL https://deno.land/install.sh | sh. Best fit for TypeScript-first agent stacks.

Network Egress: The Most Overlooked Attack Surface

Even with a containerized sandbox, unlimited internet access is a liability. A prompt-injected agent can participate in DDoS attacks, exfiltrate data, or, as ZombAIs demonstrated, phone home to a C2 server.

Two practical controls:

Allowlist over blocklist. Rather than trying to block malicious domains, restrict outbound traffic to a known-good set: pypi.org, npmjs.com, github.com, your internal registry. Everything else is denied by default. This is far more defensible than maintaining a blocklist.

Block cloud metadata endpoints. If your sandbox runs in a cloud environment (AWS, GCP, Azure), the instance metadata service at 169.254.169.254 is a prime target for SSRF attacks — a compromised agent can use it to steal IAM credentials. Block this at the network level and enforce IMDSv2 (require token-authenticated requests) at the cloud provider level. This was an active exploitation target as recently as early 2025.

Process Guardrails: Bounding Cost and Blast Radius

Even inside an isolated container, an agent can write an infinite loop or spawn thousands of subprocesses:

• CPU and memory caps: The --memory and --cpus Docker flags are your first line. Set them conservatively for agent tasks — 512 MB RAM and 1 CPU is sufficient for most code execution workloads.

• Process limit: Add --pids-limit 100 to cap the number of processes the container can spawn. This stops fork bombs and runaway test runners.

• Disk quota: Use Docker’s --storage-opt size=1G (with a supported storage driver) to cap how much the agent can write to the mounted workspace.

Human-in-the-Loop as a Security Gate

Some commands should never run without explicit approval, regardless of sandbox isolation. The harness is the right place to intercept them:

REQUIRES_APPROVAL = [
    r"curl\s+.*\|\s*(bash|sh)",   # curl pipe to shell
    r"wget\s+.*\|\s*(bash|sh)",   # wget pipe to shell
    r"git\s+push",                # pushing code
    r"pip\s+install\s+--index",   # installing from non-standard index
]

def requires_human_approval(command: str) -> bool:
    import re
    return any(re.search(pattern, command) for pattern in REQUIRES_APPROVAL)

Wire this into the pre-execution hook from Part 2. If requires_human_approval() returns True, the harness pauses and prints the command for the user to approve or deny before the sandbox sees it. The agent never knows the gate exists — it just receives a response or a timeout.

What’s Next?

A secure, sandboxed harness is now capable of running code safely over many iterations. But as tasks grow in complexity — refactoring a large codebase, researching a topic across dozens of sources, debugging a multi-file system — the agent will start to hit a different wall: context window bloat.

After 50 tool calls and thousands of lines of output, the conversation history becomes unwieldy, expensive, and eventually truncated. The agent starts to “forget” earlier decisions.

In Part 4 of this series, we will tackle Managing the Long-Running Agent: context compaction, dynamic history compression, and strategies for keeping an agent coherent across hours of execution without burning through your entire token budget in the first 20 minutes.

In Part 1, I argued that the harness is more important than the model, that the teams shipping reliable autonomous agents win by obsessing over the infrastructure around the LLM, not just the weights inside it.

That was the “why.” This post is the “what” and “how.”

We will deconstruct the anatomy of a production-grade agent harness, map out its execution lifecycle, and walk through a complete, working Python implementation you can run today. By the end, you will have a blueprint you can extend into your own projects.

The 5 Architectural Pillars of a Harness

An agent harness is more than an infinite while loop calling an API. In a production environment, a resilient harness consists of five distinct components working in unison:

1. The State Controller & Continuation Loop

An agent operating in the wild cannot be ephemeral. If a network request drops or a rate limit is hit, the agent must resume exactly where it left off. The State Controller maintains a durable, append-only conversation history, every prompt sent, every tool result received, so that each new step picks up with full context intact.

2. The Tool Registry and Schema Manager

Models do not natively understand APIs, file systems, or shells. The Tool Registry translates between model intent and real-world system interfaces by maintaining a catalogue of declarative tool schemas, structured definitions the model reads to understand what a tool does and what parameters it expects. The registry also validates inputs before execution, rejecting malformed calls at the harness level and saving expensive API roundtrips.

3. The Execution Sandbox

A raw terminal is a liability. A robust harness isolates every tool call: commands run inside constrained environments (Docker containers, microVMs, or at minimum a subprocess with strict timeouts and blocked patterns) rather than directly on your host. In our implementation we use a subprocess with a blocklist and timeout, and we are honest about what a production sandbox looks like in Part 3.

4. Deterministic Middleware & Lifecycle Hooks

Hooks intercept every tool call before and after execution, the same pattern HTTP frameworks use for authentication and logging middleware. Pre-execution hooks can block or rewrite a call (a linter catching bad syntax before wasting a compile cycle). Post-execution hooks can run tests and return pass/fail as feedback.

5. Boundary Controls & Guardrails

The harness is the last line of defense against runaway costs and infinite loops. Hard iteration caps guarantee a termination point regardless of what the model decides internally. Token budgets and timeout limits enforce financial boundaries.

The Lifecycle of a Single Step

Before reading the code, it helps to see how these five pillars interact during a single execution step:

The critical insight is in the bottom path: errors are not crashes. The harness intercepts them, formats them as structured feedback, appends them to the conversation history, and loops, giving the model the information it needs to self-correct on the next step.

The Complete Implementation

Here is a fully working, minimal agent harness in Python. It uses the Anthropic API with real tool execution, a proper tool registry, lifecycle hooks, and boundary controls, but you can replace it with any other LLM provider or even your own local model. You can run this with pip install anthropic.

Before running, set your API key:

export ANTHROPIC_API_KEY=your-key-here

#!/usr/bin/env python3
"""
Minimal Agent Harness — Educational Implementation

Demonstrates the 5 architectural pillars of a production harness:
  1. State Controller & Continuation Loop
  2. Tool Registry & Schema Manager
  3. Execution Sandbox (subprocess with blocklist + timeout)
  4. Middleware & Lifecycle Hooks (pre/post interceptors)
  5. Boundary Controls & Guardrails (iteration cap, timeout)
"""

import json
import subprocess
from pathlib import Path
import anthropic


# ─── PILLAR 3: EXECUTION SANDBOX ────────────────────────────────────────────
# These are the actual tool implementations. In a production harness these
# would run inside Docker or a microVM. Here we use subprocess with a blocklist
# and a hard timeout as a minimal safety layer.

BLOCKED_PATTERNS = ["rm -rf", "sudo", ":(){:|:&};:", "> /dev/", "mkfs", "dd if="]

def read_file(path: str) -> str:
    try:
        return Path(path).read_text()
    except FileNotFoundError:
        return f"Error: file not found: {path}"
    except Exception as e:
        return f"Error: {e}"

def write_file(path: str, content: str) -> str:
    try:
        Path(path).parent.mkdir(parents=True, exist_ok=True)
        Path(path).write_text(content)
        return f"Wrote {len(content)} characters to {path}"
    except Exception as e:
        return f"Error: {e}"

def run_bash(command: str) -> str:
    for pattern in BLOCKED_PATTERNS:
        if pattern in command:
            return f"Blocked: command contains disallowed pattern '{pattern}'"
    try:
        result = subprocess.run(
            command,
            shell=True,
            capture_output=True,
            text=True,
            timeout=30,   # Hard cap: no command runs forever
        )
        return (result.stdout + result.stderr).strip() or "(no output)"
    except subprocess.TimeoutExpired:
        return "Error: command timed out after 30 seconds"
    except Exception as e:
        return f"Error: {e}"


# ─── PILLAR 2: TOOL REGISTRY & SCHEMA MANAGER ───────────────────────────────
# Maps tool names to implementations and exposes Anthropic-compatible schemas.
# The model reads these schemas to understand what tools exist and how to call them.

TOOL_REGISTRY = {
    "read_file":  read_file,
    "write_file": write_file,
    "run_bash":   run_bash,
}

TOOL_SCHEMAS = [
    {
        "name": "read_file",
        "description": "Read the full contents of a file at the given path.",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "Relative or absolute file path"}
            },
            "required": ["path"],
        },
    },
    {
        "name": "write_file",
        "description": "Write text content to a file, creating it if it does not exist.",
        "input_schema": {
            "type": "object",
            "properties": {
                "path":    {"type": "string", "description": "File path to write to"},
                "content": {"type": "string", "description": "Text content to write"},
            },
            "required": ["path", "content"],
        },
    },
    {
        "name": "run_bash",
        "description": (
            "Execute a shell command and return its output. "
            "Use for running scripts, installing packages, or checking system state. "
            "Destructive commands are blocked."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "command": {"type": "string", "description": "The bash command to run"}
            },
            "required": ["command"],
        },
    },
]


# ─── THE AGENT HARNESS ───────────────────────────────────────────────────────

class AgentHarness:
    """
    A minimal, self-correcting agent harness.

    All five pillars are implemented:
      1. history[] maintains durable state across iterations
      2. TOOL_SCHEMAS + TOOL_REGISTRY handle registration and dispatch
      3. _dispatch_tool() runs tools in the sandbox
      4. _handle_tool_calls() wraps each call with pre/post hooks
      5. max_iterations enforces a hard termination boundary
    """

    def __init__(self, system_prompt: str, max_iterations: int = 10):
        self.client = anthropic.Anthropic()
        self.system_prompt = system_prompt
        self.max_iterations = max_iterations

        # PILLAR 1: durable state — the full conversation lives here
        self.history: list[dict] = []
        self.iteration = 0

    # ── PUBLIC INTERFACE ─────────────────────────────────────────────────────

    def run(self, task: str) -> str:
        """Run the agent on a task and return its final text response."""
        print(f"\n[Harness] Task: {task[:80]}...")
        self.history.append({"role": "user", "content": task})

        while self.iteration < self.max_iterations:
            self.iteration += 1
            print(f"\n[Harness] ── Step {self.iteration}/{self.max_iterations} ──")

            response = self.client.messages.create(
                model="claude-opus-4-7",
                max_tokens=4096,
                system=self.system_prompt,
                tools=TOOL_SCHEMAS,
                messages=self.history,
            )

            if response.stop_reason == "end_turn":
                print("[Harness] Goal reached.")
                return self._extract_text(response)

            if response.stop_reason == "tool_use":
                # PILLAR 4: middleware intercepts every tool call
                self._handle_tool_calls(response)

        # PILLAR 5: hard boundary — guaranteed termination
        print("[Harness] Iteration limit reached. Halting.")
        return "Stopped: maximum iteration limit reached."

    # ── PRIVATE METHODS ──────────────────────────────────────────────────────

    def _handle_tool_calls(self, response) -> None:
        """Execute all tool calls in a response and feed results back into history."""

        # Save the full assistant message (including tool_use blocks) to history
        self.history.append({"role": "assistant", "content": response.content})

        tool_results = []
        for block in response.content:
            if block.type != "tool_use":
                continue

            # PRE-EXECUTION HOOK: log, validate, or block before anything runs
            print(f"[Hook:pre]  → {block.name}({json.dumps(block.input)[:100]})")

            result, is_error = self._dispatch_tool(block.name, block.input)

            # POST-EXECUTION HOOK: log outcome; extend here for test runners, linters, etc.
            status = "error" if is_error else "ok"
            print(f"[Hook:post] ← [{status}] {result[:100]}")

            tool_results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": result,
                "is_error": is_error,   # tells the model this is an error to recover from
            })

        # Append results as a user message — this closes the feedback loop.
        # The model sees tool outputs (or errors) on the next iteration and
        # decides whether to retry, adjust, or proceed.
        self.history.append({"role": "user", "content": tool_results})

    def _dispatch_tool(self, name: str, args: dict) -> tuple[str, bool]:
        """Look up a tool from the registry and call it. Returns (output, is_error)."""
        tool_fn = TOOL_REGISTRY.get(name)
        if tool_fn is None:
            return f"Unknown tool: '{name}'", True
        try:
            return str(tool_fn(**args)), False
        except Exception as e:
            # Exceptions become structured feedback — not application crashes
            return f"{type(e).__name__}: {e}", True

    @staticmethod
    def _extract_text(response) -> str:
        for block in response.content:
            if hasattr(block, "text"):
                return block.text
        return ""


# ─── ENTRYPOINT ──────────────────────────────────────────────────────────────

SYSTEM_PROMPT = """\
You are a coding assistant with access to a filesystem and a bash shell.
Complete tasks step by step. Always verify your work by running it.
If something fails, read the error carefully and fix it before moving on.\
"""

if __name__ == "__main__":
    harness = AgentHarness(system_prompt=SYSTEM_PROMPT, max_iterations=10)

    result = harness.run(
        "Write a Python function that returns the nth Fibonacci number, "
        "save it to fibonacci.py, then run it with n=10 to verify it works."
    )

    print(f"\n[Final Answer]\n{result}")

What Makes This a Harness, and Not Just a Wrapper?

Looking at the code above, three behaviors stand out that elevate it beyond a simple API call:

1. The application never crashes on tool failure.

Any exception from read_file, write_file, or run_bash is caught inside _dispatch_tool and returned as a structured string with is_error=True. The model receives it as feedback, not as a Python traceback that kills the process. If the agent writes broken code and run_bash returns a SyntaxError, the harness feeds that error back and the model fixes it on the next step, without any human intervention.

2. Errors are formatted as instructions.

Setting is_error=True in the tool result is not cosmetic. The Anthropic API treats it as a signal that the model should analyze what went wrong and decide how to recover. This is the self-correction loop made concrete: the model is not told how to fix the error, but it is given the error precisely, and its next action should be a correction.

3. Termination is structurally guaranteed.

The while loop has a hard ceiling at max_iterations. No matter what logic the model follows internally, retrying the same broken approach five times, generating new subtasks, calling tools in unexpected orders, the harness guarantees a termination point. This is what makes agent systems safe to run unsupervised.

Extending the Blueprint

The implementation above is intentionally minimal. Here is where you would add more production pillars:

What’s Next?

The sandbox in our implementation, subprocess calls with a blocklist and a timeout, is a good starting point for learning, but it is not what you would ship. A production harness needs genuine isolation: the agent’s code must not be able to read your SSH keys, saturate your CPU, or exfiltrate environment variables.

In Part 3 of this series, we will tackle Sandboxing & Execution Security in Harness Engineering. We will look at how to wrap agent tool calls in Docker containers, handle filesystem mounts safely, and build a security layer that lets your agent run with real permissions without putting your host at risk.

Over the past couple of years, I have spent most of my working hours building agent systems in production, multi-agent pipelines for enterprise clients, agentic platforms for startups, and educational content for a community of over 100,000 people learning to work with LLMs. Across all of that work, one pattern kept showing up: the teams that shipped reliable, autonomous agents were not the ones with the biggest models. They were the ones who had thought hardest about the infrastructure wrapped around those models.

That observation is the starting point for this series. Over the next few posts, I want to share what I have learned from the ground up about agent harness engineering, the discipline of building the execution environment, safety rails, and feedback loops that turn a raw LLM into a system you can actually trust. Each post will move from concepts to concrete code, so whether you are just getting started with agents or already shipping them to production, there will be something here for you.

This is Part 1. Let’s start with the most important question: what even is a harness, and why does it matter more than the model inside it?

If you have been building with LLMs over the last couple of years, you have likely participated in two distinct waves of optimization:

1. Prompt Engineering: Writing carefully crafted instructions, system prompts, and few-shot examples to make a model output the correct format.

2. Context Engineering: Building retrieval-augmented generation (RAG) pipelines, managing vector databases, and injecting relevant context at the exact right time.

But as we pivot toward fully autonomous AI systems, agents that can write code, manage filesystems, and execute workflows over several hours, we are hitting the limits of prompts and context alone.

Today, a third wave has taken center stage: Harness Engineering.

The core realization of this era is simple: An LLM is not an agent. An LLM is a reasoning engine. To make that engine do useful work safely, repeatably, and autonomously, we must surround it with software.

In the developer community, a new rule of thumb has emerged: “If you’re not the model, you’re the harness.”

In this post, we will demystify what an agent harness actually is, why it holds more leverage than the underlying model weights, and how you should think about building one.

The Metaphor: The Engine vs. The Car

To understand why a harness is necessary, consider a high-performance sports car engine.

If you bolt a 600-horsepower engine to a wooden pallet in your garage and turn it on, you do not have a vehicle. You have a loud, vibrating hazard that will quickly burn through its fuel or damage itself. To make that engine useful, you need a chassis, a steering column, brakes, a dashboard, a fuel pump, and a seatbelt

In the AI agent stack:

• The LLM is the engine. It provides raw reasoning, language processing, and planning capabilities.

• The Harness is the car. It is every piece of code, configuration, filesystem access, state management, and execution logic wrapped around that model.

Without the harness, a model cannot safely write a file, execute a test, or recover from a compile error. The harness is what turns static, one-shot token generation into dynamic, continuous action.

What Lives Inside an Agent Harness?

By definition, the harness is the entire software ecosystem built to constrain, guide, and support the LLM. If we look under the hood of modern agent runtimes (like Claude Code or LangChain’s Deep Agents), a standard harness typically provides several core services:

1. The Execution Loop: The basic state machine (while active: run_step()) that keeps the agent iterating toward its goal.

2. Context Compaction: Programmatic logic that summarizes historical messages, archives oversized tool outputs, and manages the model’s memory limits over long-running sessions.

3. The Sandbox: A secure, isolated runtime (like a Docker container or microVM) where the agent can run bash commands or execute code without compromising your local machine.

4. Lifecycle Hooks: Code that intercepts the agent’s actions before or after they execute (e.g., checking code with a linter before running a compiler).

5. Human-in-the-Loop Gates: Permission layers that halt execution and ask a human to approve potentially destructive commands (e.g., deleting a folder or executing a database write).

The Core Paradigm: Feedforward vs. Feedback

A useful way to think about how a harness works is to divide its responsibilities into two major systems: Feedforward Controls (Guides) and Feedback Controls (Sensors).

1. Feedforward: Setting the Rails Before the Model Acts

Feedforward mechanisms set the model up for success before it generates a single token. This includes:

• System Prompt Assembly: Dynamically constructing the prompt based on the agent’s current workspace.

• Project-Level Rules: Injecting repository-specific guidelines (often read from files like CLAUDE.md or AGENTS.md) directly into the agent’s memory to prevent common style or architectural mistakes.

2. Feedback: Reacting After the Model Acts

Feedback mechanisms are where the harness truly shines. Instead of forcing a human to read every error, the harness acts as a sensor.

• The Self-Correction Loop: If the agent writes bad code, the harness attempts to compile it, catches the syntax error, and passes that error back to the model as a new message: “Here is the compiler output; please fix this syntax error.”

• By automating this “sensor-loop”, the agent corrects its own mistakes in the background, only surfacing to the human user once a clean, verified solution is found.

Harness vs. Orchestration Framework: What’s the Difference?

A common point of confusion is how a harness differs from existing orchestration libraries like LangChain or LangGraph.

• An Orchestration Framework is a toolbox. It provides the building blocks—the graph runtimes, the token counters, the schema definitions—needed to connect models to code.

• An Agent Harness is an opinionated, batteries-included application wrapper built with those blocks. It dictates exactly how files are read, how sandboxes are managed, how memory is pruned, and how users approve tasks.

You use a framework to build a harness. Your harness is the custom, end-to-end environment that handles your business logic.

The Real Leverage: The Harness Beats the Model

When an agent fails a complex coding benchmark, developers are often tempted to blame the underlying model. The standard response is to wait for the next, larger model to release.

However, the practical evidence points in a clear direction: tuning the harness is often more effective than upgrading the model.

The SWE-bench leaderboard, the industry benchmark for autonomous coding agents, illustrates this well. Teams that reached the top did not do it by swapping in a bigger model. They did it by obsessing over the harness: how errors were formatted before being fed back, how long context was pruned, how the agent was given permission to retry. The same model weights, run inside a smarter harness, produced dramatically better results.

This is the Harness Engineering Mindset. Every time an agent makes a mistake, do not just tweak your system prompt. Instead, ask: What sensor, sandbox rule, or execution hook could I add to my harness so the system prevents or self-corrects this mistake next time?

What’s Next?

Now that we have established the “what” and “why” of the agent harness, it’s time to look at how these systems are built.

In Part 2 of this series, we will zoom in and examine the Anatomy of a Modern Agent Harness. We will sketch out a minimal Python implementation of an agent loop, look at how to build a simple tool registry, and write code that captures execution errors to feed them back into the model.

Have you started building custom wrappers around your LLMs? What constraints or feedback loops have made the biggest difference in your projects? Let’s discuss in the comments below.

I was watching a news broadcast being translated live on screen and had one thought: why isn’t this just built into the browser?

The technology clearly exists. Transcription, translation, real-time, all of it. But using it meant switching apps, copying text, losing the thread of what you were watching.

I started thinking about who actually needs this. Language learners trying to follow native content. Students watching university lectures in a second language. Anyone who’s ever given up on a video because the captions were too broken to follow.

So I built it.

Overline is a Chrome extension that overlays real-time AI captions and translation directly on any video you’re watching in your browser. YouTube, news sites, university lecture portals, anything with video.

You click Start. Captions appear on the video itself, in your language, in real time.

No switching tabs. No copy-pasting. No lag that breaks your focus. Just the video, with understanding layered on top.

Who it’s for:

• Language learners who want to watch native content without losing the thread

• Students accessing lectures or educational videos in a foreign language

• Anyone who’s ever given up on a video because the captions were too bad to follow

How to try it:

Overline is free and available now on the Chrome Web Store. Install it, sign in, pick your target language, and click Start Captions on any video page.

Install Overline →

How it works (for the curious)

Building this taught me a few interesting things about what’s actually possible inside a Chrome extension.

The high-level architecture looks like this:

The flow has three stages:

1. Capture

The extension captures your tab’s audio directly, the video playing in Chrome, without touching your microphone. This is done through a Chrome API designed specifically for tab audio, routed through a background process that handles the stream.

2. Transcribe & Translate

Audio is streamed in real time to a backend that pipes it through a speech-to-text API, then optionally through an LLM for translation. The system distinguishes between partial (in-progress) transcripts and finalized ones, which is what gives the captions their live, flowing feel rather than appearing in sudden chunks.

3. Overlay

The caption text is injected directly into the tab as an overlay on the video, no separate window, no UI clutter. It appears where you’re already looking.

Here’s the sequence of a single caption appearing from the moment audio leaves your browser:

The whole round trip, audio out, caption back, typically lands under a second

The extension is built with WXT (a TypeScript-first framework for Chrome extensions), React for the popup, and a Python + FastAPI backend. Auth is handled by Clerk.

This is v1. I’m actively working on making it faster, supporting more languages, and reducing setup friction. If you try it, reply here or reach out directly, I’d genuinely love to know what’s working and what isn’t.

Install Overline on the Chrome Web Store

There’s a story I keep hearing in the ML community right now, and it goes something like this: a team spins up a fleet of coding agents, ships features at 10x speed, and celebrates. Two months later, nobody on the team understands the codebase anymore. Tests pass but production breaks. The agents are called in to fix the mess, but the codebase is now so large, so tangled, and so far beyond anyone’s context window that even the agents can’t get their bearings.

This isn’t a hypothetical. Mario Zechner, an independent software developer, who just shipped PI, his own coding agent harness, watched this pattern emerge and decided to say something about it out loud. His talk, “Building PI in a World of Slop,” is one of the most honest and technically grounded takes I’ve seen on the current state of AI coding agents. I agree with nearly everything he says. Here’s my take from his talk.

Why a Veteran Developer Walked Away from Claude Code

Mario didn’t build PI out of spite. He started using Claude Code in April 2025, loved it, and credits the Anthropic team as genuinely talented. But over time, something shifted. The tool grew faster than it improved.

His specific complaints are worth naming:

• Context he couldn’t control. Claude Code manages the context window on your behalf, and it does things behind your back. System prompts change on every release, including tool definitions. Tools get removed or modified without notice.

• System reminders injected at the worst moments. The tool would insert reminders into the context mid-task, sometimes with language like “this may or may not be relevant to what you’re doing”, which, as Mario notes, is exactly the kind of hedging that confuses a language model trying to maintain coherence.

• Zero observability. You can’t see what your agent is doing or why. For anyone who cares about understanding their systems, this is a serious problem.

• Shallow extensibility. Hooks exist, but they’re process-level, a new shell command spawned on each trigger. Not deep integration. Not programmable.

These aren’t complaints about Anthropic’s intentions. They’re complaints about what happens when a tool optimizes for feature velocity at the cost of transparency and stability. Mario’s analogy is perfect: if your hammer breaks every day on a construction site, you get mad. Developer tools are no different.

He looked at alternatives, shoutout to AMP and Factory Droid as the “Porsche and Lamborghini” of coding harnesses, and also dug into OpenCode’s internals, where he found similar issues: context pruning that effectively lobotomizes the model, LSP error injection mid-edit (checking for errors after every line, not after you finish your work), and a server that by default exposes itself to any website open in your browser.

So he built PI. And what he built teaches us something important.

Big Idea #1: Minimal Beats Maximal

PI ships with four tools. Its system prompt fits on a single slide. That’s the whole thing.

Before you dismiss this as toy-project minimalism, consider Terminal Bench, a coding agent benchmark that gives the model exactly one capability: send keystrokes to a tmux session and read the output. No file tools. No sub-agents. No search. Just a terminal. As of December 2025, Terminal Bench scores higher on the leaderboard than most full-featured harnesses, including native model harnesses. The simplest possible environment outperforms the most feature-rich ones.

This should give us pause.

We’ve been operating under an assumption that more context management, more tools, more agentic scaffolding equals better performance. Mario’s argument, backed by benchmark data, is that we’re in what he calls the “fuck around and find out” phase of coding agents. We don’t actually know what the right harness looks like yet. And in that uncertainty, the minimal approach wins because it introduces fewer confounding variables, gives the model cleaner signal, and is easier to reason about.

This maps directly to something ML practitioners already know: simpler models with strong inductive biases consistently outperform overparameterized ones in low-data regimes. The principle holds in agent design too. A model that’s been RLHF-trained extensively on coding tasks already knows what a coding agent is, you don’t need 10,000 tokens to remind it. It knows because it was trained to be one.

PI’s extensibility philosophy reinforces this: instead of shipping every feature, ship a minimal core and let the agent extend itself. Users describe what they need; PI builds the extension. The agent adapts to your workflow, not the other way around.

Big Idea #2: Agent Velocity Without Discipline Is Technical Debt on Steroids

This is the part of Mario’s talk that I think every ML team lead and practitioner needs to hear.

Agents compound errors. Mario calls these “boooos”, mistakes that accumulate silently, without the friction that would normally alert a human developer. A human feels pain. They get confused, frustrated, they slow down, they refactor. Agents don’t feel pain. They will happily keep generating code into a broken codebase indefinitely.

The math here is brutal. With one human developer, you have a natural bottleneck on how many errors can enter the codebase per day. Add ten agents running in parallel, and that bottleneck disappears. The error rate scales. The review burden scales with it, but your human capacity to review does not.

And here’s the structural problem with what agents learn: they were trained on the internet, which is overwhelmingly our old, mediocre code. Not the pearls, the garbage. Every blank left in a spec gets filled in by the agent using patterns learned from that garbage. You get abstraction layers nobody asked for, duplicated logic, backwards-compatibility shims for scenarios that don’t exist. Enterprise-grade complexity generated in two weeks by two humans and ten agents. Congratulations.

“But we have a detailed spec.” Mario’s response to this is worth quoting directly: a sufficiently detailed spec is a program. If there are gaps in your spec, the model fills them. You don’t get to control what it fills them with.

“But we have a review agent.” Also not enough. Review agents catch some issues. They miss the architectural ones, the decisions that seem locally reasonable but cause global damage, because those require understanding the whole system, which is exactly what gets lost when no human is reading the code.

Mario’s framework for responsible agent use is practical and I endorse it:

• Scope tightly. Give the agent only the context it needs and guarantee it can find everything relevant to the task. If you can’t scope it, don’t delegate it.

• Modularize your codebase. Smaller, well-bounded modules make agentic tasks tractable and reviewable.

• Use agents for the right tasks. Reproduction cases for bugs, boring boilerplate, non-critical automation, research, these are good. Core business logic, security-sensitive code, architectural decisions, these are not.

• Read the critical lines yourself. Every one of them. If you don’t know what’s critical, that’s your answer: read more code.

The last point is uncomfortable, and I think that’s the point. The discipline to read your own code is not a tax on productivity, it’s what makes you the developer rather than the reviewer of whatever the agent decided.

The OSS Problem: Clankers Are Degrading the Commons

Briefly, because it deserves mention: AI-generated GitHub issues and pull requests are burning out open source maintainers. Mario showed his own tracker, half of it is garbage from agent instances running autonomously, submitting PRs without reading the contribution guidelines, flooding issue queues with low-quality reports.

His response, auto-closing agent PRs, asking contributors to write issues in their own human voice under a screen’s worth of text, building a filter that only lets through accounts that actually read the instructions, is clever. But the fact that he has to spend time building abuse filters for his own open source project is a problem the community created.

If you’re using agents to contribute to OSS, please configure them with appropriate constraints. Or better yet, read the repo and write the issue yourself.

Slow Down. Build Less. Understand More.

Mario ends his talk with something that sounds almost contrarian in 2025: slow down. Think about what you’re building and why. Learn to say no. Fewer features, but the ones that matter, and then use your agents to polish those features until they’re excellent.

I think he’s right. The race to maximize agent output is a race to maximize technical debt, user confusion, and maintainer burnout. The practitioners who will build the most durable, trustworthy AI-powered systems are the ones who treat agents as precision instruments, not autonomous co-developers.

Mario’s closing line is the one I keep coming back to:

“Friction is the thing that builds understanding of the system in your head, and it’s also where you learn new things.”

We’ve been treating friction as the enemy. It’s not. It’s the mechanism by which understanding transfers from code to developer. Remove it entirely and you end up with a codebase you can’t debug, can’t extend, and can’t trust.

Use your agents. Use them well. But keep your hands in the code.

Mario Zechner’s talk “Building PI in a World of Slop” was presented at AI Engineer conference. PI is open source, you can find it on GitHub. If you found this useful, forward it to a colleague who’s been talking about going “full agents.”

I’ve been working with semantic web technologies since 2012. RDF, OWL, SPARQL, Protege, Jena, triplestores -- these have been part of my toolkit for over a decade. I’ve built ontologies for research projects, designed knowledge graphs for NLP pipelines, and written more SPARQL than I’d care to admit.

And throughout all of it, one frustration has remained constant: the gap between what knowledge graphs can do and what people are willing to put up with to use them.

The technology is sound. The W3C stack is well-designed. Ontological reasoning is genuinely powerful. But the barrier to entry has always been too high. You need to understand RDF serialization, SPARQL syntax, OWL semantics, URI design, and named graph management before you can do anything useful. For most teams, the learning curve kills adoption before the value becomes visible.

LLMs have finally given us the tools to close that gap. So I built KeplAI -- an open-source platform that puts an AI layer on top of the standards-based semantic web stack, making knowledge graphs accessible without sacrificing the rigor that makes them worth using in the first place.

The Semantic Web’s Adoption Problem

If you’ve worked with knowledge graphs, you know this story. You show someone a beautifully reasoned inference chain -- how (Mehdi, founded, BrandPulse) plus the ontology’s domain/range constraints automatically infers that Mehdi is a Person and BrandPulse is a Company. They’re impressed. Then you show them the SPARQL they’d need to write to query it, and their eyes glaze over.

The semantic web has always had a marketing problem disguised as a usability problem. The underlying ideas -- triples, ontologies, inference, linked data -- are elegant. But the tooling has historically assumed that everyone using it has a PhD in knowledge representation.

I don’t think the answer is to dumb it down. Stripping away the ontology layer to make things “simpler” defeats the purpose. Without formal semantics, you just have a labeled property graph -- fine for some use cases, but you lose inference, auto-typing, and schema validation. You lose the reason knowledge graphs are more powerful than a relational database in the first place.

The answer is to keep the semantic rigor underneath and put a natural-language interface on top.

What Knowledge Graphs Actually Give You

For readers who haven’t worked with these technologies before, let me explain why anyone would bother with all this machinery instead of just using a regular database.

A knowledge graph stores facts as triples: subject, predicate, object.

(Mehdi, founded, BrandPulse)

(BrandPulse, industry, AI)

(Alice, worksAt, BrandPulse)

Each triple is a fact. Together, they form a graph you can traverse. Want to know who works at companies Mehdi founded? Follow the edges. No JOINs, no pre-designed queries, no schema migration when you add a new relationship type.

But the real power comes from ontologies -- formal descriptions of what types of things exist and how they can relate.

Why Ontologies Matter

An ontology doesn’t just describe data. It enables reasoning.

Define this:

Class: Person

Class: Company

Property: founded (domain: Person, range: Company)

Now when someone adds (Mehdi, founded, BrandPulse), the system automatically infers that Mehdi is a Person and BrandPulse is a Company. This is called auto-typing, and after working with it for over a decade, I can tell you it’s one of those capabilities that seems minor until you realize how much manual annotation it eliminates at scale.

Ontologies also give you constraint validation (is this triple even valid given the schema?), cross-domain reasoning (if a Person can found a Company, and a Company has Employees, what can we infer about the relationship between founders and employees?), and interoperability (your ontology and mine can be linked through shared upper ontologies).

The Multi-Ontology Problem

Real-world knowledge doesn’t fit into one neat schema. This is something I’ve dealt with repeatedly across projects. A hospital needs a medical ontology, an organizational one, and a geographic one. A research lab might combine domain-specific ontologies with FOAF, Dublin Core, and Schema.org.

These ontologies overlap. Both FOAF and Schema.org define “Person.” When your data says “Person,” which one do you mean? When two ontologies define different properties with the same label, which takes precedence?

In KeplAI, each imported ontology lives in its own named graph inside Apache Jena Fuseki. URI resolution searches across all loaded ontologies. If there’s a unique match, it resolves. If two ontologies claim the same label, the system raises a conflict rather than silently picking one. This is the kind of design decision that comes from having been burned by silent ontology collisions more than once.

What LLMs Finally Make Possible

I’ve watched the semantic web community struggle with two problems for years. LLMs don’t solve them perfectly, but they solve them well enough to fundamentally change the equation.

Populating the Graph

Knowledge graph construction has always been the bottleneck. I’ve used every approach:

• Manual curation -- accurate but impossibly slow. I’ve spent weeks annotating corpora for research projects that produced graphs with a few thousand triples.

• Rule-based extraction -- fragile. Change the sentence structure and your regex pipeline breaks.

• Supervised NER/RE models -- requires labeled training data, which requires... manual curation. A chicken-and-egg problem.

LLMs break this cycle. Give GPT-4 a paragraph and an ontology schema, and it extracts structured triples that actually conform to the schema:

triples = await kg.extract_and_store(

"Mehdi founded BrandPulse in 2024. The company focuses on AI.",

mode="strict" # Constrained to ontology schema

)

In “strict” mode, extraction is constrained to properties defined in your ontology -- the LLM can only produce triples that match your schema. In “open” mode, it discovers relationships freely, which is useful for exploratory graph building.

But extraction alone isn’t enough. Text says “M. Allahyari” and the graph already has “Mehdi Allahyari” -- are they the same entity? This entity disambiguation problem is something I’ve worked on extensively. KeplAI uses vector embeddings (OpenAI’s embedding model + Qdrant vector store) to match extracted entities against existing ones. The user sees candidates ranked by similarity score and can confirm or reject. It’s not perfect -- “Apple” the company vs. “apple” the fruit still requires contextual reasoning that embeddings alone can’t fully capture -- but it handles the 80% case that used to require manual reconciliation.

Querying the Graph

SPARQL is a powerful query language. I write it fluently after years of practice. But I also know that asking a domain expert to write this is unreasonable:

PREFIX entity: <http://keplai.io/entity/>
PREFIX ontology: <http://keplai.io/ontology/>
SELECT ?company WHERE {
  GRAPH ?g {
    entity:Mehdi ontology:founded ?company .
    ?company ontology:industry "AI" .
  }
}

They want to ask: “What AI companies did Mehdi found?”

Translating natural language to SPARQL is where I’ve spent the most engineering effort in KeplAI, because this is where naive LLM usage falls apart spectacularly.

The core problem: LLMs invent predicates. Ask GPT-4 to generate SPARQL for “when was Tom Hanks born?” and it’ll confidently produce ontology:birthDate -- even when your graph uses ontology:bornOn. The LLM’s world knowledge overrides the schema context you provided. It’s the same hallucination problem, just manifesting in a structured output.

KeplAI’s NLQ engine addresses this with a multi-stage pipeline:

1. Relation mapping -- Before generating SPARQL, an LLM call maps natural-language phrases (”works at,” “born in”) to actual predicates in your graph. This produces explicit mappings that get injected into the generation prompt.

2. Entity resolution -- Entity mentions are extracted from the question and resolved against the graph via vector similarity. “Tom Hanks” becomes entity:ThomasJeffreyHanks (or whatever your graph uses).

3. Schema-grounded generation -- The SPARQL generation prompt includes the full property list (with descriptions), the entity mappings, the relation mappings, and explicit instructions to use only listed predicates.

4. Predicate validation -- After generation, every URI in the output SPARQL is checked against the allowed set. If the LLM invented ontology:birthDate when only ontology:bornOn exists, it’s flagged.

5. Self-repair -- Invalid predicates trigger a repair call: “Your query uses these invalid predicates. Here are the allowed ones. Fix it.” This catches the 10-15% of queries where the LLM ignores the schema despite explicit instructions.

Is it perfect? No. Complex multi-hop queries still trip up sometimes. But for the vast majority of questions a domain expert would ask, it produces correct SPARQL -- and it shows the generated query alongside the results, so power users can verify and edit.

Use Cases Where This Actually Matters

Research Knowledge Management

This is closest to my own experience. A researcher imports domain ontologies (biomedical, legal, environmental), then feeds papers and documents through the extraction pipeline. The knowledge graph grows semi-automatically. They can ask “Which genes are associated with both diabetes and cardiovascular disease?” and get answers grounded in extracted data, with provenance tracking showing which paper each fact came from.

Enterprise Knowledge Unification

Organizations have knowledge trapped in wikis, Confluence pages, Slack threads, and people’s heads. A knowledge graph can unify this. Extract triples from documentation, connect them with organizational ontologies, and you have a queryable map of institutional knowledge. When someone leaves the company, their knowledge doesn’t leave with them.

Compliance and Auditing

Every triple in KeplAI can carry provenance: source document, extraction timestamp, method (manual vs. AI-extracted). In regulated industries -- healthcare, finance, legal -- being able to trace every fact back to its source isn’t a nice-to-have. It’s a requirement.

AI-Powered Applications

This is where I think the biggest opportunity lies. RAG (Retrieval-Augmented Generation) typically uses vector search over chunks of text. But knowledge graphs offer something vector search can’t: structured reasoning. Instead of retrieving “similar text,” you retrieve facts and their logical connections. An LLM grounded in a knowledge graph doesn’t just find relevant passages -- it follows relationship chains and provides answers with explicit provenance.

Hard-Won Technical Insights

A few things I want to share from building this that aren’t obvious from the outside:

Named graphs are essential but the ecosystem barely supports them. Almost every SPARQL tutorial teaches you to query the default graph. In practice, if you’re managing multiple ontologies with data isolation, everything lives in named graphs. This means every query needs a GRAPH ?g { ... } wrapper. Forget it, and you get zero results with zero helpful error messages. KeplAI auto-injects GRAPH clauses when the LLM forgets them -- a safety net born from painful debugging sessions.

The LLM-to-SPARQL “last mile” is harder than NL-to-SQL. SQL has a relatively flat structure -- tables, columns, JOINs. SPARQL has PREFIX declarations, URI resolution, named graphs, blank nodes, OPTIONAL patterns, FILTER expressions, and the RDF data model underneath. The LLM needs to get all of this right simultaneously. The relation mapping and predicate validation layers exist because a single wrong URI means the query silently returns nothing.

Ontologies compound in value over time. This is something you only appreciate after maintaining a knowledge graph for months or years. Early on, defining an ontology feels like overhead. Six months in, when auto-typing has correctly classified thousands of entities and inference has surfaced relationships you never explicitly stated, the upfront investment pays for itself many times over.

Entity disambiguation remains the hardest unsolved problem. After years of working on this, I’m convinced that pure vector similarity gets you ~80% of the way. The remaining 20% -- contextual ambiguity, cross-language entities, temporal references -- requires hybrid approaches combining embeddings with graph structure and contextual reasoning. This is active work.

The Architecture

For those who want to look under the hood:

KeplAI is SDK-first. The Python SDK (keplai) is the core -- it manages the graph, ontologies, and AI features and can be used standalone. The REST API (FastAPI) and Web UI (React + TypeScript) are layers on top.

The stack:

• Apache Jena Fuseki -- standards-compliant RDF triplestore with SPARQL endpoint and OWL reasoning. The SDK manages Fuseki’s Docker lifecycle automatically.

• OpenAI GPT-4 -- powers extraction, NL-to-SPARQL generation, and result explanation.

• Qdrant -- vector similarity search for entity disambiguation.

• OWL/RDFS reasoning -- built into the triplestore, so inference happens at the data layer, not the application layer.

The web UI provides a dashboard, triple management with batch operations, an ontology editor with multi-ontology support, a text extraction interface, a natural language query page, and an interactive force-directed graph explorer.

Open source under Apache 2.0: github.com/mallahyari/keplai.

The Bigger Picture

I’ve been in the semantic web world long enough to have seen multiple hype cycles come and go. Linked Data. The original Semantic Web vision. Knowledge graphs in enterprise. Each wave brought real progress but never achieved mainstream adoption.

This time feels different, and the reason is LLMs.

The fundamental thesis: LLMs are great at understanding language but hallucinate facts. Knowledge graphs are great at storing facts but are hard to query naturally. They’re complementary in a way that’s almost too clean.

LLMs as the interface to structured knowledge -- with the knowledge graph providing ground truth, provenance, and reasoning, and the LLM providing the natural-language bridge -- is, I believe, how the next generation of knowledge-intensive AI applications will be built. Not chatbots that make things up, but systems that reason over verified facts and can show their work.

KeplAI is my attempt to make that vision practical today. It’s not finished -- the NLQ pipeline needs more work on complex queries, the disambiguation could be smarter, and there are a dozen features on the roadmap. But it’s usable now, and the best way to improve it is for people to use it and push against the edges.

If you’ve been curious about knowledge graphs but put off by the tooling, or if you’re a semantic web veteran wondering how LLMs fit into the picture, give it a try.

KeplAI is open source and available at github.com/mallahyari/keplai.

If you’ve been paying attention to the open-source AI space lately, you’ve probably noticed the explosion of personal AI assistants. It started with OpenClaw — a powerful, feature-rich agent that took GitHub by storm with 240k+ stars. Then came the inevitable: everyone wanted a lighter version.

ZeroClaw rewrote it in Rust. PicoClaw rebuilt it in Go for $10 hardware. IronClaw focused on security with WebAssembly sandboxing. NanoClaw stripped it to 500 lines.

I tried several of them. And then I found Nanobot — a 4,000-line Python alternative — and stopped looking.

Here’s why.

The Problem With OpenClaw

Don’t get me wrong — OpenClaw is impressive. It supports dozens of messaging platforms, has 100+ skills, and can genuinely automate complex workflows. But it comes at a cost:

• 430,000+ lines of TypeScript

• 70+ dependencies

• 52+ modules and 53 config files

• Individual files that run thousands of lines deep

I wanted a personal assistant I could own — not just use, but understand end-to-end, modify confidently, and extend without fear of breaking some distant module I’d never read. OpenClaw wasn’t that.

The Alternatives Didn’t Quite Fit Either

The community responded with lighter alternatives, and they’re genuinely impressive:

But here’s my issue: I think in Python. My entire ML/AI workflow is Python. When I look at a Rust codebase, I can appreciate the engineering — but I can’t casually add a new tool on a Saturday morning. And the 500-line options, while elegant, sacrifice too many features to be practical as a daily driver.

Enter Nanobot: The Sweet Spot

Nanobot delivers the core functionality of OpenClaw in roughly 4,000 lines of Python. That’s 99% smaller. But the line count isn’t the real story — what matters is that you can read the entire codebase in an afternoon and understand every decision.

Here’s what’s packed into those 4,000 lines:

• 13 chat channels — Telegram, Discord, Slack, WhatsApp, Email, Matrix, and more

• 19 LLM providers — Anthropic, OpenAI, DeepSeek, Groq, Gemini, OpenRouter...

• 13 built-in tools — file ops, shell, web search, scheduled tasks, background agents

• Persistent memory — the bot remembers you across conversations

• A skill system — drop a markdown file in a folder and your bot gains new abilities

• MCP support — plug into the Model Context Protocol ecosystem

And it’s not just small — it’s clean. The architecture follows a simple principle: channels don’t know about the AI, and the AI doesn’t know about channels. Everything talks through a message bus.

The Architecture (in 30 Seconds)

The whole system is three layers connected by async queues:

A message comes in from any platform. The bus routes it to the agent loop. The agent builds a prompt (with your conversation history, long-term memory, and any active skills), calls the LLM, executes tools if needed, and sends the response back through the bus to whichever platform you’re on.

Here’s what a single request looks like end-to-end:

That’s it. The entire flow.

What Makes It Easy to Customize

Adding a tool is ~30 lines

Want your assistant to do something new? Every tool is a small class with one method:

class MyTool(Tool):
    name = "check_stocks"
    description = "Check current stock prices"
    parameters = {
        "type": "object",
        "properties": {
            "symbol": {"type": "string", "description": "Stock ticker symbol"}
        },
        "required": ["symbol"]
    }

    async def execute(self, params: dict) -> str:
        symbol = params["symbol"]
        # Your logic here — call an API, query a database, anything
        price = await fetch_stock_price(symbol)
        return f"{symbol}: ${price}"

Register it in one line, and your bot can now check stock prices. No plugin system to learn, no config files to wrestle with, no build step.

Adding a skill is just a markdown file

Skills inject knowledge into your bot’s system prompt. Want your assistant to be an expert at managing your home automation?

Create ~/.nanobot/workspace/skills/smart-home/SKILL.md:

# Smart Home Assistant

You can control the user's home devices using the exec tool.
- Lights: `hue-cli set <room> <brightness>`
- Thermostat: `nest-cli temp <degrees>`
- Always confirm before changing settings.

That’s a skill. Drop the file in the folder, and your bot knows how to manage your smart home on the next message.

The memory system is beautifully simple

Nanobot keeps two files:

• MEMORY.md — long-term facts (”User prefers Python. Lives in San Francisco. Works on ML projects.”)

• HISTORY.md — timestamped event log

MEMORY.md gets injected into every prompt, so your assistant always knows who you are. When conversations get long, the agent summarizes old messages and saves the important bits. No vector database, no embeddings, no infrastructure — just markdown files you can read and edit yourself.

The Honest Trade-offs

Nanobot isn’t trying to replace OpenClaw for everyone. Here’s where it intentionally makes different choices:

• No GUI — it’s channels + CLI. If you want a web dashboard, look elsewhere.

• Fewer built-in skills — OpenClaw has 100+; Nanobot has 8 (but adding your own is trivial).

• Single-process — it’s one Python process, not a distributed system. For a personal assistant, that’s a feature, not a bug.

• File-based storage — no database. Sessions are JSONL, memory is markdown, config is JSON. Simple and portable, but not built for multi-user scale.

For a personal assistant that you run for yourself, these trade-offs are exactly right.

Why This Matters

There’s a broader point here that goes beyond any single project.

We’re in an era where AI tools are getting more powerful every month. The models improve, the APIs get cheaper, the capabilities expand. In this environment, the best AI assistant isn’t the one with the most features — it’s the one you can evolve alongside the models.

When Claude gets better at reasoning, I want to adjust my agent’s tool execution strategy. When a new MCP server drops for my favorite service, I want to plug it in. When I have a weird personal workflow that no one else would build a plugin for, I want to code it myself in an hour.

That requires a codebase I can hold in my head. For me, that’s 4,000 lines of Python — not 430,000 lines of TypeScript or a Rust binary I’d need to recompile.

Getting Started

If this resonates, here’s the quickest path:

# Install
pip install nanobot

# First-time setup (creates config + workspace)
nanobot onboard

# Chat interactively
nanobot agent

# Or run as a multi-channel gateway
nanobot gateway

Edit ~/.nanobot/config.json to add your API keys and enable channels.

Final Thought

The claw family has given us incredible options. OpenClaw proved the concept. ZeroClaw and IronClaw showed us what’s possible in Rust. PicoClaw proved AI agents can run on tiny hardware.

Nanobot showed me that sometimes the most powerful tool is the one simple enough to be truly yours.

If you’re building your own personal AI assistant or have thoughts on the claw ecosystem, I’d love to hear from you. Drop a comment or reach out, I’m always happy to talk about this stuff.

Most tutorials show you a single LLM call that converts a question into SQL. That works on toy examples. It falls apart the moment a real user touches it.

Real databases have dozens of tables. Real users ask vague, ambiguous questions. And real SQL agents need to handle errors gracefully, retry intelligently, stream responses in real time, and never, ever run a DROP TABLE on your production database.

In this post, I’ll walk you through how I designed and built a production-ready SQL agent, covering the architecture, the key design decisions, and the lessons learned along the way. We’ll go light on boilerplate and heavy on the why.

The full code (backend + frontend with live visualizations) is on GitHub: github.com/mallahyari/langgraph-sql-agent

The Stack

• LangGraph: multi-node agent orchestration with conditional routing and retry loops

• GPT-4o: reasoning, SQL generation, and natural language synthesis

• FastAPI + Server-Sent Events: async streaming API

• SQLite: database backend (the architecture is database-agnostic)

• React + Vega-Lite: frontend with auto-generated charts

Why a Multi-Agent Approach?

The naive approach is one LLM call: “Here’s my schema. Here’s the user’s question. Write SQL.”

This breaks in predictable ways:

• The LLM picks the wrong tables when your database has many of them

• Vague questions like “show me sales” generate ambiguous queries

• When a query fails at runtime, there’s no recovery path

• You have zero visibility into what went wrong or why

The insight is that text-to-SQL isn’t one problem, it’s several smaller ones chained together. When you decompose it into specialized nodes, each with a single job, you get a system that’s easier to debug, easier to improve, and resilient to failure.

Think of it as a pipeline of domain experts:

Each of these is a node in a LangGraph StateGraph. The graph connects them with conditional edges that handle routing, retries, and error recovery automatically.

The Architecture

Here’s the full workflow:

The retry loop is the key architectural decision: if the SQL fails validation or fails at runtime, the graph routes back to the SQL generator with the error message injected into the conversation. The LLM sees what went wrong and corrects itself , up to 3 times before giving up gracefully.

This is what separates a demo from a system you can actually deploy.

How LangGraph Orchestrates It

LangGraph is built on the concept of a StateGraph, a directed graph where each node reads from and writes to a shared state object. Here’s the shape of that state:

class AgentState(TypedDict):
    user_query: str
    refined_query: Optional[str]
    relevance: str
    selected_tables: List[str]
    generated_sql: str
    query_result: List[Dict[str, Any]]
    query_error: Optional[str]
    is_valid_sql: bool
    retry_count: int
    validation_error: Optional[str]
    natural_response: str
    needs_visualization: bool
    visualization_spec: Optional[Dict[str, Any]]
    logs: Annotated[List[str], operator.add]
    steps: Annotated[List[str], operator.add]

One thing worth calling out: the Annotated[List[str], operator.add] on logs and steps. This tells LangGraph to append to these lists rather than overwrite them. Every node contributes its own entries, and you end up with a full audit trail of what happened at each step, invaluable for debugging.

Routing between nodes is handled by plain Python functions:

def route_after_execution(state: AgentState) -> str:
    if state.get("query_error") and state.get("retry_count", 0) < MAX_RETRIES:
        return "sql_generator"
    return "response_synthesizer"

Simple, readable, and easy to change. The orchestration logic lives in the graph definition, completely separate from the agent implementations.

Key Design Decisions

1. Use Function Calling for Structured Decisions

Any time a node needs to produce a categorical output, “is this relevant?”, “which tables?”, “does this need a chart?” — use OpenAI’s function calling with an enum constraint instead of parsing free text.

Free text parsing is fragile. The model might say “Yes, this is relevant” or just “Relevant” or “I believe the query relates to...”, you end up writing brittle string matching. Function calling gives you guaranteed structured output:

"relevance": {
    "type": "string",
    "enum": ["relevant", "irrelevant"]
}

One call, guaranteed valid result. Use this pattern for every decision node in your graph.

2. Always Ground-Truth Check LLM Outputs

The table selector asks the LLM which tables are needed, then does this:

valid_tables = [t for t in args["tables"] if t in all_tables]

That one line prevents a surprisingly common failure: LLMs confidently hallucinate table names that don’t exist. Always cross-reference any LLM output that refers to real-world entities (table names, column names, file paths) against the actual ground truth before passing it downstream.

Same principle applies to column names in the SQL generator, feed the LLM the exact CREATE TABLE schema, not a description of it.

3. Raise Temperature on Retries

At temperature=0, GPT-4o is deterministic. If the first SQL query fails and you retry with the same temperature, you get the exact same broken query. Set temperature to 0.3 on retries, just enough variation that the model explores a different approach, without going off the rails.

4. Inject Error Context Into the Retry Conversation

On retry, don’t just run the node again. Pass the failed query and the error message back to the SQL generator as prior conversation turns:

System: [schema + instructions]

User: [original query]

Assistant: [the broken SQL that failed]

User: "That query failed with: no such column 'TotalRevenue'. Please fix it."

The LLM now has the full context of what it tried and why it failed. This dramatically improves retry success rate compared to starting fresh.

5. Validate SQL Before It Touches Your Database

A safety layer that checks for destructive keywords (DROP, DELETE, TRUNCATE, UPDATE, etc.) using regex word boundaries is non-negotiable. This blocks both accidental and malicious writes before anything reaches the database.

pattern = r'\b' + keyword + r'\b'

Word boundaries matter, without them, a column named dropship_count would trigger a false positive.

6. Truncate Large Results Before Synthesis

Some queries return thousands of rows. Feeding all of them to the synthesizer is expensive and often exceeds context limits. Truncate to a reasonable character limit before synthesis. The synthesizer’s job is to summarize, not to enumerate every row, so you rarely lose anything meaningful.

7. Thread IDs for User Session Isolation

LangGraph’s MemorySaver checkpointer scopes conversation history to a thread_id. Pass a unique ID per user session, and multiple concurrent users share one deployment with fully isolated state. No session collisions, no state bleed-through.

For production, swap MemorySaver for a persistent checkpointer backed by Redis or Postgres so history survives server restarts.

Streaming: Why It Matters More Than You Think

A SQL query can take 3-5 seconds end-to-end. Without streaming, your user stares at a blank screen and assumes something broke.

The architecture uses two streaming modes simultaneously via LangGraph’s astream:

• updates mode: fires after each node completes, sending structured data (SQL generated, rows returned, etc.) that the frontend can display progressively

• custom mode: fires on every LLM token during synthesis, enabling word-by-word text streaming

The result is a UI that feels responsive and alive even during a multi-second pipeline. The user sees the SQL appear, then the row count, then the explanation streaming in, then the chart, all without waiting for the full workflow to finish.

This is delivered over Server-Sent Events (SSE) via a FastAPI StreamingResponse. SSE is simpler than WebSockets for this use case: unidirectional, HTTP-native, no handshake overhead.

One gotcha: if you run behind Nginx, add X-Accel-Buffering: no to your response headers. Without it, Nginx buffers the entire response and defeats the purpose of streaming entirely.

The Visualization Layer

One of the more interesting parts of the system is the automatic chart generation. After synthesizing a text response, a visualization planner node decides whether a chart would add value, checking for explicit requests (”plot this”, “show a chart”) and analyzing the data shape (time series, categorical breakdown, single values).

If a visualization makes sense, a visualization generator node calls GPT-4o in JSON mode to produce a Vega-Lite spec directly from the query results. The frontend renders it instantly.

This means a query like “show me monthly revenue over the past year” automatically produces a line chart, with no hardcoded chart types and no user configuration required. The LLM figures out the right encoding, axes, and title from the data itself.

Vega-Lite is a great fit here because it’s declarative JSON, the LLM can produce it reliably in JSON mode, and the frontend renders it without needing to interpret any code.

What the Frontend Does

The backend streams JSON events. The React frontend listens and progressively renders:

1. A “thinking” indicator while nodes are running

2. The generated SQL in a syntax-highlighted code block (so users can inspect it)

3. The row count and raw data in a table

4. The natural language explanation, streaming token by token

5. The Vega-Lite chart, rendered automatically when the visualization spec arrives

Showing the SQL is a deliberate choice. It builds trust, users can see exactly what query was run, verify it makes sense, and catch errors. It also makes the system feel transparent rather than like a black box.

Common Failure Modes (And How to Avoid Them)

Hallucinated column names. The LLM makes up a column that doesn’t exist. Fix: always pass the exact CREATE TABLE DDL as schema context, not a paraphrased description.

Overly broad table selection. The table selector includes every table “just in case,” bloating the schema context. Fix: require the LLM to justify each selected table, or use strict function calling with an explicit count limit.

ORDER BY placement in UNION queries. SQLite requires ORDER BY to appear after the final SELECT in a UNION ALL. The LLM often puts it in the wrong place. Fix: add this as an explicit rule in the SQL generator’s system prompt.

Markdown in SQL output. The model wraps its SQL in ```sql ``` fences. Always strip these before executing.

Retry storms. Without a hard retry cap, a broken query can loop indefinitely. Enforce a maximum retry count in state and route to a graceful error response when exceeded.

Going Further

This architecture is a foundation. A few directions worth exploring:

Scale to larger schemas. With hundreds of tables, feeding even table names to the LLM becomes unwieldy. Add a vector similarity search step to retrieve semantically relevant tables before the table selector node runs.

Row-level security. The executor currently runs all queries with the same database credentials. In a multi-tenant system, inject user-specific filters or use separate database roles per user.

Human-in-the-loop. LangGraph has native support for breakpoints and interrupt_before, you can pause the graph before execution and require a human to approve the generated SQL. Useful in high-stakes environments.

Swap the LLM. The OpenAI client lives in a single service file. Replace it with Anthropic’s Claude, a local Ollama model, or any OpenAI-compatible endpoint. The rest of the system stays the same.

Persistent memory. Right now, conversation history lives in memory and resets on restart. Swap MemorySaver for LangGraph’s AsyncPostgresSaver for durable multi-session memory.

The Full Picture

What makes this system work in practice isn’t any single clever trick, it’s the combination:

• Specialization: each node does one thing and does it well

• Retry with context: the graph corrects its own mistakes with full error awareness

• Safety by design: SQL is validated before it ever reaches the database

• Streaming at every layer: users see progress, not spinners

• Grounded outputs: LLM results are always checked against reality before use

• Separation of concerns: orchestration logic lives in the graph, not in the nodes

The LangGraph pattern, decompose into nodes, connect with conditional edges, add retry loops, generalizes far beyond SQL. Any problem where a single LLM call is too brittle is a candidate for this architecture.

The complete code for both the backend and the React frontend with live Vega-Lite visualization is available here:

github.com/mallahyari/langgraph-sql-agent

If you build something with this or adapt it for your own database, I’d love to hear what you changed and why.

"From Stanford professors to OpenAI scientists, unicorn founders to quantum-tech pioneers, the future of AI is gathering in one place— and you're invited."

Mark your calendars for July 13-17, 2025, because the epicenter of the AI world will be at the Santa Clara Convention Center. GenAI Week Silicon Valley 2025 is shaping up to be the largest, most influential AI gathering of the decade. With over 30,000 attendees expected, this will be a glimpse into the future of artificial intelligence and its transformative impact across every industry.

An Unprecedented Lineup of AI Luminaries

The speaker list reads like a who's who of AI innovation:

• Jason Wei,, OpenAI scientist behind GPT-3

• Arvind Jain, CEO of Glean ($7.2B valuation)

• Edward Oates, Co-Founder of Oracle

• Soumya Batra, core author of Llama 2 & 3

• Richard Socher, CEO of You.com ($900M valuation)

And that's just scratching the surface. From startup founders pushing the boundaries of what's possible, to leaders at tech giants like Microsoft, Google, and Meta, the collective brainpower at this event is staggering.

For startups, this is your chance to shine on a global stage.

The Founder's Pitch Package offers the opportunity to present your vision to a room full of top-tier investors and decision-makers.

📢 Do you want to pitch your startup on stage?

Apply here:

Apply here!

（Reminder, purchase the FOUDNERS PITCH PACKAGE at the same time)

📈 Are you a VC attending the event?

If you'd like curated intros, private sessions, or exclusive access — fill this:

For VCs

📚 Still deciding which day(s) to attend? Here more reasons:

100+ Sessions Covering the Future of AI

With over 100 tactical sessions, you'll gain practical insights into:

• Advanced agent frameworks

• AI monetization strategies

• Ethical considerations in AI development

• Industry-specific AI applications

• The intersection of AI and quantum computing

Whether you're a developer, entrepreneur, investor, or simply AI-curious, there's something here to expand your horizons and spark new ideas.

Full Speaker Lineup (One-Stop Guide)：

All speakers listed here:

Coolest demo - Alef flying cars

Global Networking on an Unparalleled Scale

"With attendees from over 30 countries, it'll be buzzing with energy, ideas, and opportunities." - Jimmy, Cofounder of GPTDAO

The value of connections made at events like this cannot be overstated. GenAI Week 2025 brings together a truly global community of AI enthusiasts, practitioners, and visionaries. The potential for collaboration, investment, and career advancement is immense.

Exclusive Discount for Our Readers

We're excited to offer our TwoSetAI audience an exclusive 15% discount on tickets. Use coupon code TWOSETAI when registering through these links:

• Luma: https://lu.ma/genaiweek2025?coupon=TWOSETAI

• Eventbrite: www.eventbrite.com/e/1301648458579/?discount=TWOSETAI

• QR codes for easy registration:

  

Limited spot free ticket!

For our MLNotes and TwoSetAI audience, we have several free tickets available. For a chance at a free ticket, email angelina@oscr.tech before 6pm PST on 7/6. Limited quantities available, so act fast!

Hope to see you there!

Claim your spot now with our exclusive discount!

🛠️✨ Happy practicing and happy building! 🚀🌟

Thanks for reading our newsletter. You can follow us here: Angelina Linkedin or Twitter and Mehdi Linkedin or Twitter.

Source of image:
SWR: https://milvus.io/docs/v2.5.x/assets/advanced_rag/sentence_window.png

🌈 Our RAG course: https://maven.com/angelina-yang/mastering-rag-systems-a-hands-on-guide-to-production-ready-ai

 📚 Also if you'd like to learn more about RAG systems, check out our book on the RAG system: You can download for free on the course site:
https://maven.com/angelina-yang/mastering-rag-systems-a-hands-on-guide-to-production-ready-ai

🦄 Any specific contents you wish to learn from us? Sign up here: https://noteforms.com/forms/twosetai-youtube-content-sqezrz

🧰 Our video editing tool is this one!: https://get.descript.com/nf5cum9nj1m8

📽️ Our RAG videos: https://www.youtube.com/@TwoSetAI

📬 Don't miss out on the latest updates - Subscribe to our newsletter:

The MLnotes Newsletter

MLnotes delivers bite-sized content covering diverse aspects of AI and ML, from real-world applications to entrepreneurship and careers in AI. Together, we can foster knowledge sharing and alleviate information anxiety amidst the rapid advancements in AI!

By Mehdi Allahyari

(He shared a backlink/SEO hack that he personally is using, which is totally thinking outside of the box, check it out below!) 👇
(He also shared deep insight around whether SEO is for you OR NOT! see below!) 👇

How’s SEO evolving in the age of AI?

"I think there's a few factors. One we can get into is building a foundation about actually thinking about starting to rank on LLMs," says Sam Dunning, founder of Breaking B2B.

This statement captures the current state of SEO -

it's not dying, it's evolving.

The rise of AI and Large Language Models (LLMs) has undoubtedly shaken up the world of search engine optimization. Some have even proclaimed the death of SEO as we know it.

However, the reality is far more nuanced.

SEO isn't dead; it's undergoing a transformation, adapting to the new landscape of AI-driven search.

Sam's insights reveal that while traditional SEO metrics like traffic are changing, with some clients seeing up to a 20% decline in traffic from traditional search, new opportunities are emerging.

The key is to understand these changes and adapt our strategies accordingly.

Google remains the undisputed king of search.

Despite the buzz around AI search tools, Google remains the undisputed king of search. Sam points out,

This statistic is crucial for SEO professionals and businesses alike. While it's important to prepare for the future of AI-driven search, we can't ignore the present reality.

Google is still the primary source of organic traffic for most websites, and optimizing for Google's algorithms remains a critical part of any comprehensive SEO strategy.

Adapting SEO Strategies for AI-Driven Search

While Google still dominates, the rise of AI search tools like ChatGPT can't be ignored.

Sam advises,

"It's about building that foundation for LLMs. So not disregarding Google. Still need to work on it because the chances are when they fully roll out Google AI mode in a year or so (it’s already out!), it would likely take a lot of those ranking factors across, but build up a foundation to start ranking on chatGPT, and similar as well so you don't get kind of caught behind."

This dual approach - optimizing for both traditional search engines and AI-powered tools - is likely to become the new norm in SEO.

It requires a deeper understanding of how AI interprets and ranks content, as well as a willingness to experiment with new strategies.

Understanding your Customer before Content Creation is the key.

One aspect of SEO that remains unchanged, regardless of the technology driving search, is the importance of understanding your audience. —> This has never changed!

Sam emphasizes,

"If the content you're putting out just ranks, but it doesn't actually resonate with your dream clients, expensive problems, motivations, jobs to be done, address their common questions, position your product as a painkiller, and show why you and it's almost a waste of time."

This insight tells me that thorough customer research should be top priority in your SEO strategy.

It's not just about ranking for keywords; it's about creating content that truly resonates with your target audience's needs, pain points, and motivations.

This customer-centric approach not only helps with SEO but also improves conversion rates and overall marketing effectiveness.

Technical SEO? Do your users understand your content?

While technical SEO elements remain important, Sam's approach emphasizes the need to balance these with user-focused content. He states,

"So many SEOs talk about the technical things, the step by step processes, but they forget that we're actually selling complex solutions to prospects that have jobs to be done, struggling moments, etcetera."

This balance is crucial in the evolving SEO landscape. Technical optimizations help search engines understand and rank your content, but it's the user-focused elements that keep visitors engaged and encourage conversions.

As AI becomes more sophisticated in understanding user intent, this balance will likely become even more important.

Backlinks and brand mentions still works…

Despite the changes in search technology, some fundamental SEO principles remain crucial. Sam highlights the continued importance of backlinks and brand mentions, especially in the context of AI search.

He notes,

"They evaluated 75,000 brands. And that evaluation, it showed that the ones that are ranking the most consistently on AI overviews were getting consistent brand mentions on external websites."

This insight suggests that while the mechanisms of search are changing, the underlying principles of authority and relevance - as demonstrated through backlinks and brand mentions - continue to play a significant role.

It's a reminder that SEO isn't just about on-page optimization, but also about building a strong presence across the web.

Strategies for Ranking in Both Traditional and AI-Driven Search

Sam provides several strategies for optimizing content for both traditional and AI-driven search:

Read more

"Software is changing again, and I think it's changing quite fundamentally. I think roughly speaking, software has not changed much on such a fundamental level for 70 years, and then it's changed I think about twice quite rapidly in the last few years." - Andrej Karpathy

These words from Andrej Karpathy, former director of AI at Tesla, set the stage for a profound transformation in the world of software development. His recent keynote talk at the Startup School in San Francisco is super inspiring.

As AI engineers, data scientists, and tech professionals, we find ourselves at the forefront of this revolution. But with great power comes great responsibility, and the specter of AI risk looms large over our exciting new frontier.

The Evolution of Software: From 1.0 to 3.0

Software 1.0: The Traditional Paradigm We All Know

For decades, software development meant writing explicit instructions for computers in languages like C++, Python, or Java. Karpathy refers to this as "Software 1.0" – the foundation of our digital world.

Software 2.0: The Rise of Neural Networks

The advent of deep learning ushered in the era of "Software 2.0." Instead of writing explicit code, we began training neural networks, effectively programming through data and optimization algorithms. This shift marked a significant departure from traditional software development practices.

Software 3.0: Programming in Natural Language

Now, we stand at the precipice of "Software 3.0" – a paradigm where we program Large Language Models (LLMs) using natural language prompts. This revolutionary approach democratizes programming, but it also introduces new challenges and potential AI risks that we must carefully navigate.

LLMs: The New Operating Systems

Why LLMs Are More Than Just Another Tool

Karpathy draws a compelling analogy between LLMs and operating systems:

"LLMs don't only have properties of utilities. I think it's also fair to say that they have some properties of fabs1, and the reason for this is that the capex2 required for building LLM is actually quite large."

This perspective shifts our understanding of LLMs from mere tools to fundamental infrastructure. Just as operating systems provide a platform for applications, LLMs are becoming the foundation for a new generation of AI-powered software.

The Utility-Like Nature of LLM Providers

LLM providers like OpenAI, Google (with Gemini), and Anthropic are emerging as utility-like entities. They invest heavily in infrastructure (akin to power plants) and offer metered access to their intelligence via APIs. This utility model introduces new considerations for AI safety and regulation, as we become increasingly dependent on these "intelligence grids."

The Psychology of LLMs: Understanding Our New Digital Colleagues

LLMs as "People Spirits"

Karpathy introduces a fascinating concept: LLMs as "people spirits" – stochastic simulations of human-like intelligence. This anthropomorphic view helps us understand both the strengths and limitations of these systems.

Cognitive Quirks and Limitations

While LLMs possess superhuman capabilities in certain areas, they also exhibit cognitive deficits:

• Hallucinations and confabulation

• Jagged intelligence (excelling in some areas while failing at simple tasks)

• Lack of persistent memory

Understanding these limitations is crucial for mitigating AI risk and designing effective human-AI collaboration systems.

Designing LLM Apps with Partial Autonomy

The Autonomy Slider: Finding the Right Balance

Karpathy introduces the concept of an "autonomy slider" in LLM applications. This allows users to control the level of AI involvement, from simple autocomplete to full-fledged autonomous agents. Striking the right balance is key to maximizing productivity while maintaining human oversight.

We can Model After Cursor for AI-Assisted Coding

The Cursor app exemplifies effective LLM integration in software development:

1. Context management

2. Orchestration of multiple LLM calls

3. Application-specific GUI for easy human auditing

4. Flexible autonomy levels

By studying successful applications like Cursor, we can derive best practices for designing LLM-powered tools that enhance productivity without compromising on AI safety.

Human-AI Collaboration Loops is Super Important

Speeding Up the Verification Process

To truly leverage the power of LLMs, we need to optimize the human-AI collaboration loop. Karpathy emphasizes two key aspects:

1. Speeding up verification through effective GUIs

2. Keeping AI "on a leash" to prevent overwhelming the human collaborator

Lessons from Tesla's Autopilot Development

Karpathy shares insights from his experience with Tesla's Autopilot:

"We did more and more autonomous tasks for the user, and maybe the story that I wanted to tell very briefly is actually the first time I drove a self-driving vehicle was in 2013... This drive was perfect. There were zero interventions, and this was 2013, which is now 12 years ago."

This anecdote illustrates the long journey from impressive demos to real-world deployment, highlighting the need for patience and rigorous testing in AI development.

Democratizing Programming in the AI Era

Everyone Is Now a Programmer

The advent of natural language programming through LLMs has democratized software development. Karpathy coins the term "vibe coding" to describe this phenomenon:

"Suddenly, everyone is a programmer because everyone speaks natural language like English. This is extremely bullish and very interesting to me and also completely unprecedented."

Building a Custom iOS App Without Swift Knowledge

Karpathy shares his experience of creating an iOS app using vibe coding:

"I built this iOS app, and I don't actually know how to program in Swift, but I was really shocked that I was able to build like a super basic app... This was just like a day of work, and this was running on my phone like later that day."

This example demonstrates the transformative potential of LLM-assisted programming, but it also raises questions about job security for traditional developers and the need for new skills in the AI era.

Building for Agents: Creating Future-Ready Digital Infrastructure

Making Our Digital World LLM-Friendly

As LLMs become more prevalent, we need to adapt our digital infrastructure to be more "LLM-friendly." Karpathy suggests several approaches:

Read more

But doing so effectively—and in a way that suits your preferences…

We're bombarded with news from countless sources, often cluttered with ads and irrelevant content. As media theorist Clay Shirky points out,

But what if there was a way to curate you…

Read more

"You could stop a thousand people on the street and ask them what they want in life and no one is going to say, 'I really wish I was more efficient.'" - Cara Smith

Today, founders are racing to create the next groundbreaking product.

But amidst the frenzy of development and feature optimization, many are falling into a dangerous trap: the efficiency obsession.

What surprised me was how often, this focus on productivity gains can be myopic and limiting, moreover, it's potentially fatal for your brand. Here’s what I’ve learned why efficiency alone won't cut it, and how AI founders can craft brands that truly resonate with their audience.

People Buy Feelings, Not Features

When it comes to building a brand that sticks, emotion is your secret weapon. Cara Smith, co-founder of branding agency Smith and Diction, puts it bluntly:

"To me, the main difference between fine copy and good copy is what's the emotion? What's the feeling that's behind it?"

This insight is crucial for AI founders who often get caught up in the technical capabilities of their products.

Consider this:

while your AI might be able to process tasks 10x faster than a human, that fact alone won't make someone fall in love with your brand (Although Garry from YCombinator would say that’s a must have).

People don't wake up in the morning dreaming about being more efficient. They dream about feeling empowered, creative, or connected. Your branding needs to tap into these deeper human desires.

Your Brand Is More Than Just Your Logo, it's How People See You

"Everybody has a brand. Brand is simply how people see you,"

This perspective shift is vital for AI founders who might be tempted to think branding is just about creating a sleek logo or choosing the right color palette.

Your brand encompasses every interaction a person has with your company. It's the tone of your customer service emails, the user experience of your app, and yes, even the efficiency of your AI. But it's also about the emotions you evoke and the stories you tell.

For AI founders, this means considering how your technology makes people feel, not just what it can do.

Crafting a Verbal Identity That Resonates Beyond Efficiency

One of the most powerful tools in your branding arsenal is your verbal identity.

"Verbal identity is once you cross that line between talking to your internal audience or maybe your partners and talking to your external audience, right? So that's what is your website going to say? What's the tone that you use?"

This goes far beyond a catchy tagline. It's about developing a consistent voice and personality that speaks directly to your audience's needs and aspirations.

Take Gamma, an AI-powered presentation tool. Instead of focusing solely on how quickly it can create slides, their branding revolves around the idea of "get your ideas out there."

This speaks to a deeper desire of their target audience – educators and non-designers who have valuable ideas but struggle with the technical aspects of presentation creation.

Developing a strong verbal identity allows you to maintain a consistent voice across all platforms, helping to build trust and recognition with your audience.

For AI companies, this might mean finding ways to humanize complex technology or articulating your ethical stance on AI development.

Moving Beyond Generic AI Aesthetics and Use Design That Tells a Story

"I always just try to make sure that our logos have like a 10% strange thing about them," says Mike Smith, discussing his approach to logo design.

In a sea of generic AI company logos featuring abstract geometric shapes or glowing orbs, this philosophy of intentional uniqueness can set you apart.

Your visual identity should tell a story about who you are and what you stand for. It's not just about looking sleek or futuristic, but what’s the story behind it.

Take the example of Perplexity AI. Their verbal identity focused on the concept of "where knowledge begins," positioning their product not just as an answer machine, but as a gateway to deeper understanding.

As its designer explained:

"Perplexity is now your starting point and then you can click any of those citation links and go anywhere you want to go."

And that’s why their logo is a spinning door visually represents their mission of opening doors to knowledge.

For AI founders, this might mean moving beyond cliché tech imagery and finding visual metaphors that represent the unique value your AI brings to users' lives.

"As long as the logo can kind of tell a story and like convey an emotion or convey a strategy or something like that, that's when we know we're on to something."

Understanding Your Audience: The Key to Authentic Connection

Creating a brand that resonates requires a deep understanding of your audience. This goes beyond basic demographics. You need to get inside their heads, understand their daily lives, their aspirations, and their pain points.

"The more time you just really spend treating your audience like almost like you're in a novel or you're in a movie and you're putting yourself in the shoes of that character, then that allows you to really think, okay, how do I talk to them in a way that's going to matter to them, not necessarily matter to me."

We talked about the observation of focusing too much on efficiency being a common pitfall for AI founders. As Cara puts it,

"I see a ton of ads, especially in the AI space, that are all about efficiency and I just think it's ridiculous,"

Therefore, instead of focusing solely on how your AI can make processes more efficient, ask yourself these critical questions:

Read more

As much as we don’t want to believe that AI is impacting us,

it is.

I’m floored with what Jerry said and in the meantime not so surprised.

This year to next year, the question on our mind should be:

“Is my career AI-ready?”

To answer this, we turn to Jerry Wang, a visionary in the AI-powered hiring space, for h…

Read more

Gone are the days when AI companies could simply charge by the token or API call. The market is maturing, and with it, the expectations of customers are evolving.

As Manny Medina, founder of Paid, puts it:

“if it does work, it is up to the AI agent builder and creator to go back to the same customer and say, 'Let's align on things that are important to you and charge for it.'"

This statement encapsulates the core of what’s happening in pricing for AI products: a move from charging for inputs to charging for outcomes.

But why is this shift happening, and what does it mean for the AI ecosystem?

The Four Pillars of AI Pricing: A New Framework for Value

There are four main approaches to pricing that are currently working in the AI space:

1. Activity-based pricing

2. Workflow-based pricing

3. Outcome-based pricing

4. Agent-based pricing

Each of these approaches represents a step towards more sophisticated, value-aligned pricing models. Let's break them down:

Activity-Based Pricing: The Starting Point

Activity-based pricing is the most straightforward approach. It's essentially a credit consumption model where customers pay for the specific actions or computations performed by the AI. While this model is easy to understand and implement, it's also the most vulnerable to commoditization.

Medina warns:

"If you stay there somebody will come along and say I'll do the same thing for cheaper. Yeah. And then you are in a nightmare scenario in which there is tons of others who look just like you."

Workflow-Based Pricing: Stepping Towards Value

Workflow-based pricing involves charging for a series of connected activities that deliver a specific result.

This model begins to align pricing with the value delivered to the customer. As Medina explains:

"When you string a number of activities and you say this workflow is cost this much, like a document review, right? Because then you can separate documents that are small from documents that are long and complicated because they have different consumption patterns and it feels like you're getting closer to value based pricing as opposed to costbased pricing."

Outcome-Based Pricing: Aligning Incentives

Outcome-based pricing is where things get really interesting. Instead of charging for the work done, companies charge based on the results achieved. This could be in the form of a bonus for reaching certain performance metrics or a fee structure tied directly to customer outcomes.

Medina suggests an innovative approach:

"What I I've been recommending to my customer, it's not out there yet, but I'm going to give it a push, is to charge instead of charging per outcome, get an outcome bonus."

This model creates a powerful alignment between the AI provider and the customer, ensuring that both parties are invested in achieving meaningful results.

Agent-Based Pricing: The Future of AI Workforce

Agent-based pricing is perhaps the most forward-thinking model.

It involves charging for AI agents as if they were human employees, based on the work they perform and the outcomes they achieve.

Medina elaborates:

"You can pay instead of saying a platform fee say like you know I'm going to deploy x many agents the agent is going to do this amount of work that is equivalent of a $90,000 a year uh sr I'm going to charge you 20,000 per agent the agent is going to deliver this much work and you can pay me a bonus for the number of meetings booked."

This approach allows AI companies to tap into HR budgets rather than competing for limited software budgets, potentially opening up larger revenue streams.

Lack of visibility into unit economics is a ticking time bomb for many AI startups. Why?

Read more

This week, we’ll explore another set of techniques that includes retriever, generator, and general pipeline enhancement.

Why Your AI Might Be Suffering from "Middle Child Syndrome"

Remember …

Read more

It’s probably my favorite definition of creativity — simple, profound, and surprisingly relevant today.

At countless AI events I’ve attended, one question always comes up:

“Will AI replace us?”

My answer has always been: “No. AI will make us all creators.”

But recently, after reading into researches on AI’s impact on human creativity, and seeing firsthand how people are actually using these tools, I’ve started to rethink not just what creativity means, but how we can outgrow the older versions of ourselves with AI, instead of fearing being replaced by it.

In a world shaped more and more by artificial intelligence, it’s never been more important, (or more inspiring) to learn how to collaborate with AI, and to use it wisely as we move into the future.

The Churchill Conundrum: From Bathtub Brilliance to AI Assistance

Imagine Winston Churchill, lounging in his bathtub, dictating a national address to his assistant in the next room. He shouts corrections and refinements, pushing beyond his initial thoughts to craft a message that will resonate with the nation. This scene, immortalized in Albert Finney's portrayal, represents a level of creative collaboration that was once reserved for the privileged few.

Fast forward to today, and we find ourselves in a world where, as Jeremy Utley, adjunct professor of creativity and AI at Stanford University, puts it, "the poorest villager in Palo Alto can have what only Winston Churchill used to have." We now have AI assistants that can understand our context, voice, and intent, ready to collaborate with us in our creative endeavors.

How should we treat AI, a Tool or …?

As we embrace this new era of AI-assisted creativity, we face a crucial decision:

This choice can dramatically impact our creative output and productivity.

Why Treating AI as a Tool Limits Your Creative Potential

Research conducted revealed a surprising finding: while AI can make people 25% faster, 12% more productive, and improve work quality by 40%, less than 10% of professionals are actually reaping these benefits.

In many cases, AI made people less creative. Why?

What’s your guess of the results from the above experiment?

Read more

Artificial intelligence is evolving at a breakneck pace. Every week seems to bring breathless announcements of new AI breakthroughs and capabilities. Amid this frenzy, agentic AI - autonomous software agents that can orchestrate complex tasks - has emerged as the next frontier.

But while the potential of agentic AI is immense, the reality for most enterprises is far more sobering. Despite the hype, true agentic AI remains years away for all but the most advanced organisations.

As Dave Vellante of theCUBE Research puts it,

"2025 is not going to be the year of the agent. It'll be the year of agent marketing."

Enterprises are leaning in, but it's still early days for AI adoption

Recent survey data from ETR paints a picture of cautious but growing enterprise AI adoption:

• 80% of enterprises are now paying for AI in some form

• Nearly two-thirds are tapping into AI APIs

• 39% are experimenting with open-source models

• 27% are training proprietary models on-premises

These numbers show enterprises aren't sitting on their hands when it comes to AI. But they're a far cry from the agentic vision being hyped by many vendors.

The AI infrastructure boom is real, but it's a tale of two speeds

Make no mistake - AI is driving massive infrastructure investment. As Dave Vellante notes in his recent analysis:

"The data center super cycle kicked in in earnest in 2024. What was essentially a perpetually roughly $200 billion business...exploded in 2024...to 350 billion. That's a 58% growth rate in a single year, which is going to continue at an accelerated pace for a decade."

But this boom is playing out very differently for cloud providers versus traditional enterprises:

• Cloud titans like AWS, Google, Microsoft and Meta are pouring over $300 billion into AI data centers this year alone

• On-premises enterprise AI spending is growing, but at a much slower pace

This bifurcation means cloud AI adoption is rocketing ahead, while on-premises enterprise AI won't reach "escape velocity" until later this decade.

2025 is the year of agent marketing, not agent reality

Why? Because the vast majority of enterprises still lack the foundational capabilities needed to effectively deploy and manage autonomous AI agents:

"Our research shows that more than 85% of enterprises still need major upgrades in data quality, and the other 15% probably don't know that they do," says Vellante.

The harsh reality is that most companies are still grappling with basic data management issues. As one example shared by the researchers, "one bank found 6,000 table entries that defined what a customer was." If you can't even agree on what constitutes a customer, how can you possibly expect AI agents to make accurate, business-critical decisions?

As Vellante warns,

"agents hallucinate when the ground truth is fuzzy."

In other words, garbage in, garbage out – but now with potentially catastrophic consequences as AI agents act on flawed information.

This doesn't mean AI won't deliver value in the near-term. Co-pilots and other assistive AI will drive real productivity gains. But the grand vision of fully autonomous agents orchestrating complex business processes? That's still years away for most.

There are three critical areas where enterprises are falling short in their readiness for agentic AI:

1. Data quality and lineage remain massive hurdles

AI agents need high-quality, well-structured data to function effectively. But most enterprises are drowning in data silos and inconsistencies.

2. Integration and orchestration are far from seamless

Vendors love to tout "seamless integration" for their AI solutions. The reality is far messier. Decades of patchwork enterprise integration efforts have left a tangled web that autonomous agents can't easily navigate.

Some SaaS vendors are making progress within their own domains. But cross-application integration remains a major challenge. The risk is ending up with siloed AI agents that can't effectively collaborate across business functions.

What’s even more important but very few vendors factor it in their plan is…

Read more

Are you tired of AI giving you half-baked answers? Frustrated with language models that seem to know everything but your specific data?

You're not alone. While Retrieval Augmented Generation (RAG) promised to bridge the gap between AI and your proprietary information, many are finding …

Read more