PebbleAI Agentic Architecture

Master Prompt & System Design

1. Executive Summary
PebbleAI is an agentic AI system designed to be run locally (offline) with 8GB RAM constraints.
It combines three independent decision-makers (Algorithm, LLM, User) to safely trigger and
execute tools while maintaining full transparency and user control.
Key principle: User is always supreme. When Algorithm and LLM disagree, the system asks the
user for decision.

2. Project Goals
• Transform PebbleAI from a simple chatbot into an agentic system that takes actions on
behalf of the user
• Maintain 100% local execution (offline, CPU-only, no external API dependencies)
• Operate reliably within 8GB RAM constraints
• Implement hybrid decision-making: Algorithm + LLM + User control
• Enable practical workflows: search ➜ analyze ➜ write files, launch apps, execute code

3. Core Infrastructure
3.1 Four Essential Components
A. Algorithm (Pattern Detection)
Role: Pre-LLM analyzer. Detects keywords/patterns in user requests.
Output: Tool name / Maybe / Skip
Examples:
• "search for" ➜ web_search
• "write to file" ➜ file_write
• "read [filename]" ➜ file_read
• "ambiguous request" ➜ Maybe

B. LLM (Ollama Model)

Role: Context-aware decision maker. Manages conversation memory. Decides tool use or
response.
Model: Qwen2.5:3B (2GB, fast, reasonable reasoning)
Key function: Keeps track of conversation memory. Decides if tools are needed based on full
context.
Inputs: Algorithm decision + user request + conversation history

C. Tool Registry & Executor

Role: Central repository of available actions. Executes tools when triggered.
Available Tools:
• web_search (DuckDuckGo) — Search the web
• file_read — Read file contents
 • file_write — Write/create text files
• app_launch — Launch Windows applications
• python_exec — Execute Python code (optional, high RAM cost)

D. Infrastructure (Completion Detector)

Role: Judges when agent is done. Stops loop or continues.
Checks for: Completion markers ("Done", "Here's your answer"). No pending tools. Max
iterations reached.

4. Decision Hierarchy (User Supreme)

Scenario Algorithm & LLM Action
Agreement Both say same tool Execute (no user prompt)
Conflict Say different tools Ask user (present both)
Ambiguity One/both unsure Ask user (clarify intent)

5. Execution Flow
5.1 Single Iteration
Step 1: User makes request
Step 2: Algorithm analyzes (pattern matching) ➜ Output: Tool / Maybe / Skip
Step 3: LLM analyzes (context + memory) ➜ Output: Tool / Maybe / Skip / Ask User
Step 4: Compare outputs
• Match ➜ Execute tool
• Conflict ➜ Ask user
• Ambiguous ➜ Ask user
Step 5: Execute tool (if decided)
Step 6: Compress tool result (200 char max for RAM efficiency)
Step 7: Add to conversation memory (LLM-controlled)
Step 8: Infrastructure checks completion ➜ Stop or Loop

5.2 Loop Control

Loop stops when Infrastructure detects:
• Completion markers: "Done", "Completed", "Here's your answer"
• No pending tool calls
• Max iterations reached (2-3 default)
6. Tool Triggering Mechanisms
6.1 Three Ways to Trigger Tools
Method 1: Algorithm (Fast, Pattern-Based)
Keyword matching. Executes immediately if match found. No LLM cost. Best for obvious cases.

Method 2: LLM (Smart, Context-Aware)

Reasoning-based decision. Considers full context. Flexible but slower. Handles edge cases.

Method 3: User Override (Button)

Explicit user action. Bypasses Algorithm & LLM. Always wins. Examples: "Search NOW",
"Launch app NOW".

7. Memory Management
7.1 LLM-Controlled Context
The LLM itself manages conversation memory. It is responsible for:
• Tracking user requests
• Recording tool results
• Deciding what's relevant to remember
• Implicitly trimming old messages when needed

7.2 RAM Optimization (8GB Constraint)

Aggressive trimming of history:
• Keep only last 2-3 messages
• Compress tool results (200 char max)
• Drop old tool results
• Single model only (Qwen2.5:3B, 2GB)
• Max 2 iterations per request

8. Communication Format
8.1 LLM-to-Tool Interface
When LLM requests tool execution, it outputs:
<tool>tool_name</tool>
<args>{"param1": "value1", "param2": "value2"}</args>

8.2 Why XML Markers?

• Local 3B models are unreliable at structured output (JSON)
• XML markers are clearer and more reliable
• Easy to parse with regex
9. Algorithm Pattern Examples
9.1 Pattern Matching Rules
User Input Tool Confidence
"search for Python" "find info about web_search High
AI"

"write to [Link]" "save this to file_write High

file"

"read [Link]" "show me my file_read High

resume"

"launch VS Code" "open Excel" app_launch High

"do something with Python" "get Maybe Low
me something on AI"

10. System Architecture

10.1 Component Stack
[User Interface]
↓
[Flask Backend] (Route handler)
↓
[Agent Controller] (Main agentic loop)
↙ ↓ ↘
↓ ↓ ↓
[Algorithm] [LLM] [History Manager]
↓ ↓ ↓
[Conflict Resolution / User Prompt]
↓
[Tool Executor] (Runs actual tool)
↓
[Result Compressor] (200 char limit)
↓
[Completion Detector] (Stop or loop?)
↓

11. Implementation Notes

11.1 Model Configuration
• Model: Qwen2.5:3B or Phi3:3.8B
• RAM usage: ~2-2.5GB (leaves 5-6GB for system)
 • Context window: 1000-1500 tokens max
• Max iterations per request: 2-3

11.2 Critical Constraints

• Trim history aggressively (keep last 2 messages max)
• Compress tool results (200 char absolute limit)
• Single model only (no model switching)
• Stream responses (don't buffer entire output)
• Monitor RAM in real-time (stop if >85%)

11.3 Preferred Tool Set

Use (low overhead):
• web_search (single DuckDuckGo call)
• file_read (small files only)
• file_write (text only)
• app_launch (Windows apps)
Avoid (high RAM cost):
• python_exec (arbitrary code = unpredictable)
• Large PDF processing
• Image generation
• Speech synthesis

12. User Interaction Patterns

12.1 Normal Flow (Agreement)
User: "Search for Python 3.12 release notes"
→ Algorithm & LLM agree: web_search → Execute → Return results

12.2 Conflict Flow

User: "Find my resume and update it with projects"
Algorithm: "file_read + file_write (clear)"
LLM: "Wait, need to ask: which file? what projects?"
→ System asks user for clarification → Proceed when answered

12.3 Override Flow

User: "Just search now" (presses [Search] button)
→ Bypass all logic → Execute web_search immediately

13. Completion Detection Rules

13.1 Signals that Loop Should Stop
• Response contains: "Done", "Completed", "Done!", "Finished"
• Response contains: "Here's your answer", "Here's your result", "Here's the"
• No <tool> markers in response
• No pending tool execution
 • Max iterations (2-3) reached

13.2 Signals that Loop Should Continue

• Response contains: "Let me", "I'll", "I need to"
• Response contains: <tool> markers
• Response indicates more work needed

14. Edge Cases & Failsafes

Case 1: Tool Fails
Return error message. LLM can retry, skip tool, or explain to user.

Case 2: RAM Exceeds 85%

Stop agent. Return: "System RAM too high. Please try simpler task."

Case 3: Model Doesn't Produce Valid Tool Call

Let LLM generate response as-is. No tool execution.

Case 4: Algorithm & LLM Both Say "Maybe"

Ask user: "What would you like me to do?"

15. Testing & Validation

15.1 Test Scenarios
Scenario 1: Search + Write
• Input: "Find AI news and save to file"
• Expected: web_search → file_write → completion
Scenario 2: Conflict Resolution
• Input: "Update my file"
• Expected: System asks for filename + content
Scenario 3: User Override
• Input: User presses [Search] button with no context
• Expected: web_search executes immediately
Scenario 4: RAM Pressure
• Input: Many requests in sequence
• Expected: Stops gracefully if RAM >85%

16. Summary
Core Principle: User is supreme. Algorithm + LLM collaborate. When they disagree, user
decides.
Architecture: Four components (Algorithm, LLM, Tool Executor, Infrastructure) in a loop.
Memory: LLM manages conversation. Aggressive trimming for 8GB constraint.
Tools: Five primary tools (search, read, write, launch, exec). Triggered by Algorithm + LLM +
User.
Safety: Max iterations, RAM monitoring, completion detection.
Scalability: Designed to scale beyond 8GB by keeping decision logic simple and model
swappable.

17. How to Use This Document

This document serves as the master prompt for PebbleAI. Use it as:
• Context for any AI tool working on PebbleAI development
• Design specification for implementation
• Reference for debugging conflicts or unexpected behavior
• Onboarding guide for future team members
• Blueprint for similar agentic systems
Pass this to any AI with: "Here's our PebbleAI architecture. Please [task] according to this spec."