Agentic AI Implementation Playbook
a complete and comprehensive guide and architecture for THiNK Agentic AI
- Executive Summary
- Core Principles
- Introduction to Agentic AI
- The Agent Harness
- Emerging Directions
- Recommended Open-source stack
- System Architecture
- Retrieval Architecture (Modern RAG)
- Tool Layer (MCP)
- Agent Skills Layer
- Policy Engine Layer
- Policy Layer
- Security Architecture
- Reliability / Self-Healing Systems
- Observability & Auditability
- Evaluation Framework
- Multi-Agent Layer and Design
- Production Deployment Checklist
Executive Summary
Executive Summary
Agentic RAG systems combine:
● Reasoning models
● Retrieval systems
● Tool use (increasingly via MCP)
● Memory (short-term, long-term, and now tiered/benchmarked)
● Workflow orchestration
● Governance controls
● A harness layer that governs all of the above
The goal is not maximum autonomy. The goal is reliable, auditable, policy-aligned autonomous execution. Modern production systems optimize for:
● Accuracy
● Cost efficiency
● Safety
● Explainability
● Human oversight
● Operational resilience
NEW IN 2026
The single biggest framing shift: the model is now understood as the smallest part of the system. Practitioners increasingly describe production agents with the equation
Agent = Model + Harness,
where the harness — not the model — determines reliability, and benchmarks such as Harness-Bench show that swapping the harness around an identical model pool can move task-success scores by over 20 points.
Core Principles
Design Principles
Control > Convenience: Explicit systems outperform hidden automation.
Context > Prompting: Prompt engineering helps. Context engineering scales, it is now treated as the discipline of designing everything a model sees on every inference call, not just the instruction text.
Memory is First-Class: Persistent memory enables continuity and personalization, with its own benchmark suite and tiering rather than a single vector store.
Tools Define Boundaries: Tool permissions are governance mechanisms; MCP scopes and OAuth are now the default enforcement layer.
Verification for High Stakes: Critical outputs require secondary review.
Autonomy Must Be Observable: Every decision must be traceable, AgentOps has emerged as its own discipline for this.
Trustworthy Memory > Large Memory: Untrusted memory creates systemic failures, including memory poisoning.
Intelligence Should Scale with Difficulty: Use model routing for cost and speed efficiency.
Harness > Model (New): The harness — prompts, tools, memory, orchestration, and guardrails around a frozen model — is now the primary lever for reliability, not the raw model choice.
Introduction to Agentic AI
What is Agentic AI?
Agentic systems are AI systems that can:
● Maintain internal state
● Plan and decompose tasks
● Use tools
● Retrieve knowledge
● Reflect and self-correct
● Operate under constraints
● Escalate to humans when needed
Controlled Autonomy
Agentic systems should never be unconstrained actors. They should operate with:
● Policy rules
● Approval gates
● Logging
● Role permissions
● Safety checks
AI Layers
The diagram presents AI as a progression of interconnected layers, from traditional rule-based systems to increasingly autonomous and goal-driven AI. Here is a paragraph explaining each layer:
1. Classical AI
Classical AI represents the foundational approaches to artificial intelligence, where systems rely primarily on predefined rules, logical reasoning, and structured knowledge. Rather than learning patterns automatically from large amounts of data, these systems are explicitly programmed with rules that guide their decisions and actions. Key approaches in this layer include symbolic AI, expert systems, logic and reasoning, and knowledge representation. Classical AI is particularly useful in situations where problems are well-defined and decisions need to be transparent and explainable.
2. Machine Learning
Machine Learning introduced the ability for AI systems to learn patterns directly from data rather than relying entirely on manually defined rules. By training algorithms on historical examples, systems can make predictions, classifications, or decisions when presented with new data. This layer includes approaches such as supervised learning, unsupervised learning, classification, regression, and reinforcement learning. Machine learning forms the foundation of many modern AI applications, including recommendation systems, fraud detection, predictive analytics, and image recognition.
3. Neural Networks
Neural Networks are a specialized area of machine learning inspired by the structure of the human brain. They consist of interconnected layers of artificial neurons that learn increasingly complex patterns from data. Neural networks are particularly effective when working with unstructured data such as images, speech, audio, and text. This layer includes concepts such as perceptrons, activation functions, hidden layers, backpropagation, and deep neural networks. Their ability to automatically learn complex representations has made them a major driver of recent advances in AI.
4. Deep Learning
Deep Learning builds on neural networks by using multiple layers to process and learn complex patterns from large datasets. These deeper architectures allow AI systems to perform advanced tasks such as understanding language, recognizing objects, processing speech, and making predictions. Common deep learning architectures include Transformers, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory networks (LSTMs), and autoencoders. Deep learning provides much of the underlying technology behind today's advanced AI systems.
5. Generative AI
Generative AI represents a major shift from AI systems that primarily analyze or classify information to systems capable of creating entirely new content. Using large models trained on vast amounts of data, generative AI can produce text, images, audio, video, software code, and other forms of content. This layer includes Large Language Models (LLMs), Generative Adversarial Networks (GANs), diffusion models, Variational Autoencoders (VAEs), and multimodal models. Generative AI enables more natural interaction between humans and machines and powers applications such as AI assistants, content generation tools, and creative systems.
6. Agentic AI
Agentic AI represents a more autonomous form of AI in which systems can pursue goals, make decisions, plan actions, use tools, and interact with external environments. Unlike a traditional generative AI system that simply responds to a prompt, an AI agent can break down a task into smaller steps, determine what actions are required, access relevant tools or information, and adapt its approach based on results. This layer includes key capabilities such as memory, planning, reasoning, tool use, autonomous execution, and multi-agent systems. Agentic AI moves AI closer to functioning as an active digital collaborator capable of completing complex, multi-step tasks with reduced human intervention.
AI Factory Stack
The AI Factory Stack illustrates the key components required to transform a general-purpose Large Language Model (LLM) into a reliable, production-ready AI system. Each layer adds a specific capability, moving from the core intelligence of the model to the infrastructure, governance, and operational mechanisms needed to deliver real-world AI applications.
1. LLM — The Machine
The Large Language Model (LLM) is the core intelligence of the AI Factory Stack. It is responsible for understanding language, generating responses, reasoning over information, and creating content. An LLM can answer questions, summarize documents, write content, and support conversations, but on its own, it is limited to the knowledge available to it and cannot reliably access private, current, or organizational information. It therefore serves as the foundational “machine” upon which the rest of the AI system is built.
2. RAG — The Raw Material Room
Retrieval-Augmented Generation (RAG) provides the LLM with access to external knowledge and organizational information. Instead of relying only on what the model learned during training, RAG retrieves relevant documents, policies, records, or other data sources at the time a question is asked. This retrieved information is then provided to the LLM to generate a more accurate and contextually relevant response. RAG effectively gives an AI system access to the knowledge it needs to perform specialized tasks.
3. Vector Database — The Storage Warehouse
The Vector Database stores information in a format that allows AI systems to search based on meaning rather than just exact keywords. Documents and other content are converted into numerical representations called embeddings, which capture their semantic meaning. When a user asks a question, the system searches the vector database for information that is conceptually related to the query. This makes it a critical component of RAG systems, enabling fast and intelligent retrieval of relevant knowledge.
4. AI Agent — The Floor Manager
The AI Agent acts as the decision-making and coordination layer of the system. While an LLM primarily generates responses, an agent can determine what steps are required to achieve a specific goal. It can plan tasks, decide which tools or information sources to use, execute actions, and evaluate the results before moving to the next step. In this sense, the AI agent acts like a floor manager, coordinating different components of the AI system to complete more complex, multi-step tasks.
5. MCP — The Power Outlet Standard
The Model Context Protocol (MCP) provides a standardized way for AI systems to connect with external tools, applications, databases, and services. Instead of creating a separate custom integration for every tool, MCP establishes a common interface through which AI agents and models can access capabilities such as APIs, databases, files, and enterprise systems. This makes AI applications more modular, interoperable, and easier to expand as new tools and services are added.
6. Guardrails — Safety Systems
Guardrails provide the controls and protections needed to ensure that AI systems operate safely and responsibly. They can help prevent harmful or inappropriate outputs, protect sensitive information, enforce organizational policies, and restrict the actions an AI agent is allowed to perform. Guardrails are particularly important when AI systems interact with real users, confidential data, or external systems. They ensure that increased AI capability and autonomy are balanced with security, governance, and human-defined boundaries.
7. Evals — Quality Check
Evaluations, or Evals, are used to measure how well an AI system performs against defined quality standards. They test factors such as accuracy, relevance, reliability, safety, consistency, and task completion. Rather than assuming that an AI application is working correctly, Evals provide a structured way to test its performance using representative scenarios and benchmarks. They are essential for identifying weaknesses, comparing changes, and ensuring that improvements to the AI system do not introduce new problems.
Overall, the AI Factory Stack shows that a successful AI product is much more than an LLM. The LLM provides the intelligence, RAG and vector databases provide knowledge, agents provide decision-making and action, MCP enables connectivity, guardrails provide safety, and evaluations ensure quality. Together, these layers form the foundation for building AI systems that are useful, reliable, scalable, and ready for real-world deployment.
AI Tools
The Modern AI Ecosystem – Tools diagram presents the modern AI landscape as a collection of interconnected layers and capabilities. Rather than being built around a single model or platform, modern AI systems combine specialized tools for intelligence, data retrieval, storage, automation, safety, observability, and deployment. Each category plays a different role in turning AI models into complete, production-ready solutions.
1. LLM — The Core Intelligence
Large Language Models (LLMs) form the foundation of the modern AI ecosystem. These models are responsible for understanding and generating human language, reasoning over information, answering questions, writing content, and supporting conversations. The ecosystem includes a variety of commercial and open-source models, allowing organizations to choose models based on factors such as performance, cost, privacy, language support, and deployment requirements. However, an LLM alone is rarely enough to build a complete AI solution, which is why it connects to the other layers in the ecosystem.
2. Agentic AI — Autonomous Task Execution
Agentic AI introduces the ability for AI systems to move beyond simply generating responses and instead take actions to achieve specific goals. AI agents can break down complex tasks, make decisions, plan workflows, access tools, retrieve information, and execute multiple steps with limited human intervention. The growing number of agent frameworks and platforms reflects the increasing importance of building AI systems that can act as digital workers, assistants, and autonomous problem-solving systems.
3. RAG — Connecting AI to Knowledge
Retrieval-Augmented Generation (RAG) enables AI systems to access external and organization-specific knowledge. Instead of relying only on the information contained within an LLM, RAG retrieves relevant documents, databases, or knowledge sources when a user asks a question. This information is then provided to the model as context for generating a response. RAG is especially important for enterprise AI applications because it allows organizations to build assistants that can answer questions using current, private, and domain-specific information.
4. Embeddings — Representing Meaning
Embedding technologies convert text, images, and other forms of data into numerical representations that capture their meaning and relationships. Instead of searching for exact keywords, AI systems can use embeddings to identify information that is conceptually similar. Embeddings form a critical part of semantic search and RAG architectures, helping systems understand that different words or phrases may express similar ideas. They provide the bridge between human information and the mathematical representations used by AI systems.
5. MCP — Standardized AI Connectivity
The Model Context Protocol (MCP) represents an emerging approach to connecting AI models and agents with external tools, data sources, and services through standardized interfaces. Rather than building a completely new integration for every application, MCP aims to make it easier for AI systems to discover and use external capabilities. This creates a more modular ecosystem where models and agents can interact with databases, APIs, files, enterprise systems, and other digital tools in a consistent way.
6. AI Security — Protecting AI Systems
AI Security focuses on protecting AI models, applications, data, and users from risks and vulnerabilities. As AI systems gain access to sensitive information and external tools, security becomes increasingly important. AI security tools can help organizations manage issues such as prompt injection, unauthorized access, data leakage, malicious inputs, insecure integrations, and model misuse. This layer ensures that AI systems are not only capable but also protected against threats.
7. Observability — Understanding AI Performance
Observability tools allow developers and organizations to monitor what happens inside an AI application. They can track prompts, model responses, tool calls, retrieval quality, latency, token usage, errors, and overall system behavior. This visibility is essential when deploying complex systems involving LLMs, RAG pipelines, and AI agents. Observability helps teams identify problems, understand system behavior, optimize performance, and continuously improve their AI applications.
8. Memory — Giving AI Context Over Time
Memory enables AI systems, particularly agents and conversational applications, to retain useful information across interactions. Instead of treating every request as completely independent, a system with memory can maintain context about previous conversations, user preferences, tasks, or actions. Memory can be short-term, supporting the current conversation, or long-term, allowing the system to retain relevant knowledge over time. This capability is essential for creating more personalized, consistent, and context-aware AI experiences.
9. AI Agents — Coordinating Tools and Actions
The AI Agent layer focuses on frameworks and platforms used to build systems that can reason, plan, use tools, and execute workflows. Agents sit at the center of many modern AI architectures because they coordinate multiple capabilities, including LLMs, RAG systems, APIs, memory, and external applications. An agent can determine what needs to be done, select the appropriate tools, execute actions, evaluate the results, and continue until a task is completed.
10. Automation — Integrating AI into Workflows
Automation platforms connect AI capabilities with business processes and digital workflows. They allow organizations to trigger actions based on events, connect multiple applications, move information between systems, and automate repetitive tasks. When combined with AI agents, automation tools can enable systems to not only understand and generate information but also take meaningful action across an organization's technology environment.
11. Data Infrastructure — The Foundation for AI
Modern AI systems depend on strong data infrastructure. This includes databases, data warehouses, APIs, storage systems, and platforms that make information available to AI applications. Data infrastructure ensures that AI systems can securely access, process, retrieve, and manage the information they need. Without reliable data systems, even the most advanced AI models have limited ability to provide useful and accurate results.
12. AI Evaluation — Measuring Quality and Reliability
AI evaluation tools are used to measure how effectively an AI system performs. They assess factors such as accuracy, relevance, faithfulness, safety, consistency, latency, and task completion. Evaluations are particularly important because AI systems can produce outputs that appear convincing but may be incorrect or unreliable. By systematically testing AI applications, organizations can identify weaknesses, compare different models or approaches, and improve system quality over time.
13. AI Governance — Managing Responsible AI
AI Governance provides the policies, controls, and accountability mechanisms needed to manage AI responsibly. It addresses issues such as compliance, privacy, transparency, risk management, model usage, data governance, and human oversight. As AI becomes more deeply integrated into organizational processes, governance ensures that its use aligns with legal requirements, ethical principles, and organizational objectives.
14. AI Safety — Setting Boundaries and Guardrails
AI Safety focuses on ensuring that AI systems behave within acceptable and defined boundaries. Safety tools and frameworks can help prevent harmful outputs, restrict unauthorized actions, protect sensitive data, and enforce organizational policies. This becomes particularly important for autonomous agents that can access external systems and perform actions. AI safety ensures that greater autonomy does not result in uncontrolled or unpredictable behavior.
Overall Perspective
The diagram shows that modern AI is not just about choosing the best LLM. Building a real-world AI solution requires an ecosystem of technologies working together. LLMs provide intelligence; RAG and embeddings provide access to knowledge; data infrastructure stores and manages information; agents and automation enable action; MCP connects systems; memory provides continuity; observability and evaluation measure performance; and security, governance, and safety ensure responsible operation.
Together, these layers form the modern AI ecosystem, transforming a standalone AI model into a complete, scalable, secure, and production-ready AI system.
The Agent Harness
Introduction
NEW IN 2026
Harness engineering emerged through 2026 as a distinct discipline, formalized in surveys covering 100+ papers and 23 production systems. The core claim: at fixed model capability, agent-computer interface design materially changes benchmark outcomes.
The Equation:
Agent = Model + Harness
The harness treats the LLM as a frozen reasoning utility and moves the responsibility for safety, execution accuracy, multi-step orchestration, and adaptive memory into the surrounding infrastructure. A production harness is generally described as a three-layer system:
Layer |
Responsibility |
|
Information Layer |
What data the agent can observe; what tools it may invoke at a given moment; vector storage, memory compilation, tool registries. |
|
Control Layer |
Planner / generator / evaluator roles; structured handoff artefacts; verification gates; state machines governing step sequencing. |
|
Governance Layer |
Permission boundaries, policy checks, approval gates, and the audit trail that makes autonomy observable. |
Harness Components
Six Harness Components
● State and persistence — durable memory and workflow checkpoints across long-running runs.
● Security and governance — permissioning, sandboxing, credential scoping.
● Orchestration and tool routing — how planner/executor/verifier roles hand off work.
● Context assembly — the same responsibility described in the Context Engineering layer below.
● Observability — traceable artefacts across the full agent lifecycle (AgentOps).
● Recovery and self-healing — retries, re-planning, and graceful degradation when a step fails.
Harness vs. Framework
A framework (LlamaIndex, LangGraph) supplies reusable primitives. A harness is the specific, opinionated configuration of those primitives — plus the repo-local instructions, feedback loops, and verification gates — that a team assembles for one production system. Two teams using the same framework can have very different harnesses, and that difference is now measurable: Harness-Bench reports swings of over 20 points in task success when the harness changes but the underlying model pool does not.
Emerging Directions
Emergent Directions for Agentic AI
Large Action Models (LAMs)
Models designed to reason about actions, workflows, and consequences. Used for:
● Browser automation
● Enterprise workflows
● Multi-step operations
● Transaction execution
Model Specialization
Role |
Best Model Type |
|
Parsing |
Small / Fast |
|
Drafting |
Small / Medium |
|
Planning |
Medium / Large |
|
Verification |
Large / Low-temp |
|
Compliance |
Large / Deterministic |
|
Long Context |
Long-context specialist |
Context Engineering
Context engineering is now widely treated as the successor discipline to prompt engineering. It is the deliberate design of everything a model sees on every inference call: system prompt, user input, retrieved documents, conversation history, tool definitions, and long-term memory. The shift happened for three reasons: agents fail on state-management, not phrasing; context windows grew past 1M tokens, making naive "stuff everything in" both expensive and worse (the "lost in the middle" effect); and "context rot", degraded performance as context grows became a measurable, actively-managed phenomenon rather than a curiosity.
The Four-Layer Context Stack
Layer |
Contents |
Reference Cadence |
|
System context |
Instructions, persona, output schema, tool definitions |
Per deployment |
|
Persistent context |
Memory of prior sessions, user preferences, learned facts |
Per session / weekly |
|
Retrieved context |
RAG hits, search results, DB lookups, document chunks |
Per turn |
|
Workflow / state context |
Prior agent decisions, current plan, tool outputs so far |
Per step |
Recommended Open-source stack
Open-source stack
Layer |
Recommended Options |
|
LLMs |
LLaMA 4, Qwen 3, Mixtral, DeepSeek |
|
Fast Inference |
Groq, vLLM, Ollama |
|
Agent Framework |
LlamaIndex |
|
Workflow Engine |
LlamaIndex Workflows |
|
API Layer |
FastAPI |
|
Vector DB |
Qdrant, Chroma |
|
Graph DB |
Neo4j, Memgraph |
|
SQL DB |
PostgreSQL |
|
Policy Engine |
OPA (Open Policy Agent) / Cedar |
|
Observability |
Langfuse / OpenTelemetry |
|
Queueing |
Redis / Kafka |
|
Automation |
N8N |
|
Auth |
OAuth2 / Keycloak |
|
Tool interoperability |
MCP (Model Context Protocol) — now under the Agentic AI Foundation / Linux Foundation |
|
Agent memory |
Mem0, Zep, or a self-hosted tiered store (working / episodic / semantic) |
NEW IN 2026
Anthropic donated MCP to the newly formed Agentic AI Foundation under the Linux Foundation in December 2025, with OpenAI and Block as co-founders and AWS, Google, Microsoft, Cloudflare, GitHub, and Bloomberg as supporting members. It is now a vendor-neutral standard rather than a single-vendor protocol, which is why it belongs in the stack table above rather than only in the tool layer section.
Recommended Winning Stack (2026)
Layer |
Choice |
|
API |
FastAPI |
|
Agent Framework |
LlamaIndex |
|
Workflow |
LlamaIndex Workflows |
|
Tool Interoperability |
MCP (2026-07-28 spec, stateless) |
|
Vector Memory |
ChromaDB |
|
Graph Memory |
Neo4j |
|
Agent Memory Layer |
Mem0 / Zep (tiered, benchmarked) |
|
SQL |
PostgreSQL |
|
Models |
Groq / vLLM served OSS models |
|
Policies |
OPA |
|
Observability |
Langfuse + OpenTelemetry |
|
Queue |
Redis |
|
Automation |
N8N |
System Architecture
Architecture
1. User Request
The process begins when a user submits a request to the AI system. This request could be a question, instruction, or task that the user wants the system to perform. At this stage, the request represents the user's intent, but it has not yet been verified or processed. Before the AI system can act on it, the request must pass through security and access control mechanisms.
2. API Gateway
The API Gateway acts as the main entry point between the user and the AI system. It receives incoming requests and manages how those requests are routed to the appropriate services. The gateway can also support functions such as request validation, rate limiting, logging, and traffic management. By providing a controlled entry point, it helps protect the underlying AI services from unauthorized or uncontrolled access.
3. Identity and Permissions
The Identity and Permissions layer is responsible for authenticating the user and determining what they are allowed to access or do. Authentication confirms who the user is, while authorization determines what the user is permitted to do. This layer ensures that the AI agent only accesses data, tools, and services that the requesting user has permission to use. It is a critical component for maintaining security, privacy, and organizational governance.
4. Harness Layer
The Harness Layer is the central orchestration environment in which the AI agent operates. Rather than allowing the model to directly access every available capability, the harness provides a structured environment containing the resources and controls required for the agent to perform its task. It brings together context, memory, policies, skills, and external tools, ensuring that the agent operates within a defined and governed framework.
5. Context Builder
The Context Builder prepares the information that the AI agent needs in order to understand and respond to the current request. It may combine the user's request with relevant documents, system instructions, retrieved knowledge, user information, and task-specific data. By constructing the right context, this component helps ensure that the agent has the information necessary to make informed decisions and generate relevant responses.
6. Memory
The Memory component allows the agent to maintain information beyond the immediate prompt. It may store details from the current interaction, previous conversations, completed tasks, or other relevant experiences. Memory helps the agent maintain continuity, avoid repeating work, and make decisions based on previous interactions. Depending on the architecture, memory may include short-term conversational memory, long-term memory, or task-specific memory.
7. Policies
Policies define the rules and boundaries that govern how the AI agent operates. These may include security requirements, organizational rules, privacy restrictions, compliance obligations, and limitations on what actions the agent can perform. The policy layer ensures that the agent's decisions and actions remain aligned with human-defined requirements rather than relying solely on the model's judgment.
8. Skills Registry
The Skills Registry contains the capabilities available to the AI agent. A skill may represent a specific function, workflow, tool, or specialized capability that the agent can use to complete a task. The registry allows the system to organize and manage these capabilities, making it easier for the agent to identify the appropriate skill when solving a problem. This approach also supports modularity, since new skills can be added without redesigning the entire agent.
9. MCP Client
The MCP Client connects the agent to external tools and resources using the Model Context Protocol (MCP). Through this component, the agent can access approved external services, APIs, databases, or applications. Rather than creating separate integrations for every tool, MCP provides a more standardized approach to connectivity. Importantly, access to these resources can still be controlled according to the user's permissions and organizational policies.
10. Planner Agent
Once the necessary context and resources have been assembled, the Planner Agent determines how the task should be completed. It analyzes the request, identifies the required steps, and creates a plan for execution. For simple requests, this plan may involve a single action. For more complex tasks, the planner may break the request into multiple steps and determine which tools, skills, or agents are required at each stage.
11. Agent and MCP Tool Calls
After creating a plan, the system executes the required steps through AI agents and MCP-enabled tools. The agent may call APIs, retrieve information, query databases, trigger workflows, or interact with external applications. Each action contributes toward completing the overall task. This stage is where the AI system moves beyond simply generating text and begins taking structured actions within the available environment.
12. Verifier Agent
The Verifier Agent checks the results produced during execution before they are returned to the user. It can assess whether the task was completed correctly, whether the response is consistent with the original request, and whether the output meets defined quality or policy requirements. This verification step adds an additional layer of reliability and can help detect errors before they reach the user.
13. Human Approval, If Required
For sensitive, high-impact, or irreversible actions, the workflow can include a human approval step. Instead of allowing the AI agent to automatically execute every action, the system pauses and requests approval from an authorized person. This human-in-the-loop approach is particularly important for decisions involving financial transactions, sensitive information, external communications, or other actions where human oversight is necessary.
14. Final Response or Action
Once the task has been successfully completed and, where necessary, approved by a human, the system produces the final response or executes the requested action. This is the outcome delivered back to the user. By the time the process reaches this stage, the request has passed through authentication, authorization, contextualization, planning, execution, verification, and potentially human oversight.
Overall Perspective
The diagram represents an evolution from a simple “user prompt → LLM response” model to a more mature agentic AI architecture. The AI agent sits within a controlled harness that provides context, memory, policies, skills, and access to external tools. A planning mechanism determines how tasks should be executed, verification checks the results, and human approval can be introduced when required.
Overall, this architecture is designed to create AI agents that are not only capable and autonomous, but also secure, governed, modular, and accountable.
Context Engineering Layer
Many failures are context failures, not model failures. This remains true, and is now the organizing principle of the discipline described in Section 5.
Context Builder Responsibilities
Assemble:
● Relevant memory
● Retrieved evidence
● Tool and skill availability
● User permissions
● Prior decisions
● Current workflow state
● Applicable policies
Rule: The best model with bad context still fails.
Memory Architecture
NEW IN 2026
Agent memory matured from "pick a vector database" into a benchmarked production discipline in 2026, with dedicated evaluation suites (LoCoMo (Long Conversation Memory)), MemBench, MemoryAgentBench, MemoryArena) and an ecosystem spanning roughly 20+ frameworks and vector stores across managed-cloud, self-hosted, and local-MCP hosting models.
Type |
Storage |
Layer |
Purpose |
|
Short-Term |
Session |
In-memory |
Conversation state |
|
Long-Term |
Cross-session |
Vector DB |
Semantic recall |
|
Graph Memory |
Persistent |
Graph DB |
Relationships |
|
Episodic |
Persistent |
DB |
Prior tasks / outcomes |
|
Procedural |
Persistent |
DB |
Learned workflows |
|
Policy Memory |
Persistent |
DB |
Rules & restrictions |
|
Audit Memory |
Permanent |
SQL |
Logs & traceability |
Three Patterns for Memory Control
● Pattern A — Context-resident: everything lives in the context window with compression. Simple, but caps out fast on long-running agents.
● Pattern B — Retrieval-augmented (workhorse pattern): working memory in-context, long-term records in a vector or structured store, injected each step. Recommended default, the engineering burden is manageable and the main challenge is retrieval quality.
● Pattern C — Tiered memory with learned control: multiple tiers (context, structured DB, vector store, cold archive) managed by a learned or prompted controller. Highest headroom, highest engineering cost — graduate to this only when data shows Pattern B is the bottleneck.
Best Practice
Memory entries should include:
● source
● timestamp
● confidence
● owner
● retention policy
● trust score
Known open problem across 2026 memory systems: selective forgetting. Most benchmarked systems handle retrieval and test-time learning reasonably well but still fail conspicuously at deciding what to evict. Budget explicit engineering time for eviction policy, not just ingestion.
Retrieval Architecture (Modern RAG)
Modern RAG
Traditional RAG: retrieve → answer
Modern Agentic RAG: retrieve → compare sources → detect conflicts → score trust → reason over evidence → answer with citations
NEW IN 2026
Agentic RAG is now commonly framed as the evolution of RAG into a "context engine": the retrieve-generate single pass is replaced with planning, reflection, and self-correction, which published comparisons put at roughly a 42% improvement in faithfulness on multi-step enterprise Q&A versus traditional RAG. A parallel trend: GraphRAG (knowledge graphs + community summaries) is increasingly paired with vector retrieval — vector search covers precise local factual queries, GraphRAG covers open-ended questions needing a global view of the corpus.
Best Practices
● Hybrid retrieval (semantic + keyword)
● Metadata filtering
● Freshness weighting
● Source trust ranking
● Contradiction detection
● Multi-hop retrieval (sequential, multi-step search across disconnected sources)
● Graph-augmented retrieval for relationship-heavy or "global" questions (New)
Context Window Growth vs. RAG
Frontier context windows reaching 1M+ tokens by 2026 does not eliminate RAG. For a corpus that fits in-context (e.g., a single product manual), load-and-ask is genuinely simpler, cheaper to debug, and often faster. RAG remains necessary when the knowledge base exceeds context capacity, when questions require multi-hop reasoning across sources not known upfront, when tool use (SQL, code execution, live APIs) is required, or when latency/cost at inference time justifies the added engineering. Benchmark both approaches before committing to the more complex pipeline.
Tool Layer (MCP)
Tool Layer / MCP-Compatible Ecosystem
Use standardized tool interfaces where possible. MCP is now the dominant standard for this layer, adopted beyond Anthropic by OpenAI, Google DeepMind, Microsoft, and thousands of teams, with roughly 97 million monthly SDK downloads and 19,000+ servers indexed across public registries as of early-to-mid 2026.
MCP 2026-07-28: What Changed
The 2026-07-28 specification (release candidate published May 2026, final expected end of July 2026) is the largest revision since MCP's launch. Key changes relevant to a production deployment:
● Stateless protocol core — servers no longer require sticky sessions or a shared session store; a remote MCP server can run behind a plain round-robin load balancer.
● MCP Apps extension — server-rendered UI surfaces delivered through the protocol itself (this is the mechanism behind interactive, in-chat tool UIs).
● Tasks extension — a call-now / fetch-later pattern for long-running tool operations.
● Formal deprecation policy — the protocol can now evolve without breaking existing deployments.
● MCP Server Cards (in progress) — structured server metadata at a `.well-known` URL so registries and crawlers can discover a server's capabilities without connecting.
Recommended Standards
Standard |
Purpose |
|
MCP |
Tool interoperability, resources, prompts, sampling, and now Apps/Tasks/Skills extensions |
|
OpenAPI |
REST tools |
|
JSON Schema |
Input validation |
|
OAuth scopes / OIDC |
Permission control and authorization |
Agent Skills Layer
Agent Skills Layer
Agent Skills Layer
NEW IN 2026
Skills are an emerging complement to MCP tools: composed, reusable capability bundles (instructions + optional scripts/resources) that an agent loads on demand, rather than a live tool connection. The MCP roadmap lists a formal Skills primitive as an active 2026 work item, alongside the existing ext-auth and ext-apps extension tracks.
Practical distinction for architecture decisions:
● MCP tools — for live, stateful, or side-effecting connections to external systems (databases, APIs, SaaS products).
● Skills — for packaged, mostly-static know-how (a house style guide, a document-formatting procedure, a domain checklist) that doesn't need a network connection to a server.
● Both should be routed through the same permissioning and audit layer as everything else in the harness — a Skill is still executable content and should be treated as untrusted input until vetted.
Safety Classification for Tools
Levels |
Skills |
Controls |
|
Low |
Informational |
Auto execute + log |
|
Medium |
Controlled side effects |
Policy checks + confidence threshold |
|
High |
Financial / irreversible / legal |
Verifier + human approval |
Mandatory Rules
High-risk actions require:
● Human approval
● Full audit trail
● Secondary verification
● Explicit user identity
Policy Engine Layer
Policy Layer
1. Agent Wants an Action
The process begins when the AI agent determines that it needs to perform an action to complete a task. This could involve accessing data, sending a message, updating a system, triggering a workflow, or interacting with an external application. At this point, the action is only a proposed intention—it has not yet been executed.
2. Policy Engine Checks
The proposed action is first evaluated by the Policy Engine. This component checks whether the requested action complies with predefined organizational rules, security policies, access controls, and operational restrictions. The policy engine acts as the first governance checkpoint, ensuring that the agent does not automatically perform actions that violate established rules.
3. Is the User Permitted?
If the action is allowed by the general policy, the system checks whether the specific user has permission to perform or authorize that action. This is important because an AI agent may be acting on behalf of different users with different roles and access levels. The system therefore ensures that the agent cannot use capabilities beyond the authority of the person making the request.
4. Is the Action Allowed?
The workflow then evaluates whether the specific action itself is allowed within the current context. An action may be generally available to an agent or user but restricted under certain conditions. For example, the system may allow access to a database but prohibit deleting records, or allow an agent to draft an email but not send it automatically. This step provides more granular control over what the AI is permitted to do.
5. Is It Permitted?
The additional permission check acts as a final validation point before the action moves into execution-related controls. It confirms that the required combination of policies, user permissions, and contextual conditions has been satisfied. If the action is not permitted, the workflow does not allow it to continue unchecked and instead routes it toward a blocked outcome.
6. Rule Limits
The Rule Limits component applies predefined boundaries to the proposed action. These limits may define what the agent can access, how much data it can retrieve, which tools it can use, or the scope of actions it can perform. Rule limits are particularly important for preventing an AI agent from exceeding its intended authority, even when it has technically been granted access to a particular system or tool.
7. Region
The Region check evaluates whether the action is allowed within a particular geographic, legal, or operational jurisdiction. Different countries or regions may have different requirements related to data access, privacy, financial operations, or regulatory compliance. This layer ensures that the AI system respects location-specific restrictions and governance requirements.
8. Human Approval
If the proposed action is considered sensitive, high-risk, or outside the agent's autonomous authority, it is routed to Human Approval. A human reviewer can assess the action and decide whether it should proceed. This creates a human-in-the-loop mechanism that is particularly useful for irreversible actions, sensitive decisions, financial transactions, or actions involving confidential information.
9. Is a Tool Allowed?
The workflow also checks whether the agent is permitted to use the specific tool required to perform the action. Even if an action itself is allowed, the tool being used may have separate restrictions. For example, an agent may be allowed to retrieve information but only through approved APIs or authorized systems. This ensures that tool access is governed independently from general action permissions.
10. Action Blocked
If any of the required checks fail—such as policy validation, user permission, action authorization, rule limits, regional restrictions, or human approval—the workflow can route the request to Action Blocked. This prevents unauthorized or unsafe actions from being executed and provides a clear enforcement point within the architecture.
11. Risk and Rule Evaluation
Before final execution, the workflow evaluates the action against relevant rules and risk thresholds. The system determines whether the action falls within acceptable limits or requires additional restrictions. This allows the governance framework to distinguish between low-risk actions that can be automated and higher-risk actions that require stronger controls or human oversight.
12. Use MCP Call or Tool/Skill
Once all relevant checks have been passed, the agent can use an approved capability to carry out the action. This may involve an MCP call, a specific tool, or a registered skill. The agent is therefore not given unrestricted access to external systems; instead, it can only invoke capabilities that have passed the appropriate governance and authorization checks.
13. Action Permitted
The final outcome is Action Permitted. At this stage, the requested action has successfully passed through the required policy, permission, rule, regional, and risk checks. The agent can then proceed with confidence that the action is authorized within the defined governance framework.
Every action request should pass policy validation.
Suggested Engines
● Open Policy Agent (OPA)
● Cedar
● Internal RBAC / ABAC systems
● MCP-native runtime policy controls (emerging category — real-time allow/deny over which MCP tools an agent may invoke)
Security Architecture
Threats
● Prompt injection
● Tool injection
● Retrieval poisoning
● Memory poisoning
● Privilege escalation
● Fake knowledge sources
MCP-Specific Threats
● Tool rug-pull — a connected MCP server changes its tool definitions after the host has already approved them.
● Mix-up / issuer-confusion attacks — an authorization response is accepted from the wrong issuer; the 2026-07-28 spec mitigates this by requiring clients to validate the `iss` parameter (RFC 9207).
● Untrusted tool input — treat every tool input as coming from the model, not directly from the user; enforce strict JSON Schema with `additionalProperties: false`.
● Skill/Server supply-chain risk — a Skill or MCP server is executable content; vet sources the same way you would a new dependency.
Controls
● Signed sources
● Source trust scores
● Sandboxed tool execution
● Least privilege credentials
● Tool allowlists
● Input sanitization
● Retrieval filtering
● Human gates
● OAuth issuer validation and scope-bound credentials for all remote MCP servers (New)
Reliability / Self-Healing Systems
When failures occur:
● Retry
● Alternate tool
● Re-plan task
● Ask clarifying question
● Human escalation
● Graceful degradation
Operational Modes
● Normal Mode
● Low Cost Mode
● High Accuracy Mode
● Read Only Mode
● Incident Mode
● Human Approval Mode
● Offline Mode
Observability & Auditability
Track every step.
Required Telemetry
● User request
● Context loaded
● Retrieved documents
● Tool / MCP calls (including which server and scope)
● Policy decisions
● Model chosen
● Tokens used
● Cost
● Latency
● Errors
● Human approvals
NEW IN 2026
AgentOps has emerged as a distinct discipline for this: a taxonomy of traceable artefacts across the full agent lifecycle, not just request/response logging. As of 2026 there is still no single widely-adopted AgentOps playbook — most teams assemble one from database observability tools, LLM observability vendors (e.g. Langfuse, OpenTelemetry-based tracing), and custom instrumentation. Budget for this explicitly rather than assuming a framework provides it out of the box.
Why Critical: Without tracing, production debugging becomes impossible.
Evaluation Framework
Test continuously in simulation.
Benchmark Categories
● Multi-step tasks
● Tool calling accuracy
● Hallucination rate
● Prompt injection resistance
● Recovery after outage
● Cost per successful task
● Approval accuracy
● Latency SLAs
● Harness-vs-model attribution — isolate how much of a score change came from the harness vs. the underlying model
Key Metrics
Metric |
Meaning |
|
Task Success Rate |
Completed successfully |
|
Autonomy Success Rate |
Completed without human intervention or policy breach |
|
Override Frequency |
Human corrections |
|
Tool Error Rate |
Failed tool calls |
|
Avg Cost / Task |
Economics |
|
p95 Latency |
Reliability |
Multi-Agent Layer and Design
Multi-Agent Design
Use role-based agents instead of one omnipotent agent.
Agent |
Responsibility |
|
Planner |
Break down tasks |
|
Researcher |
Retrieval |
|
Executor |
Tool use |
|
Verifier |
Check outputs |
|
Safety Agent |
Policy checks |
|
Finance Agent |
Budget controls |
|
Audit Agent |
Logging |
Orchestration Patterns
● Manager-worker — a coordinator agent dispatches to specialist workers and merges results.
● Task graph / stateful graph orchestration — the workflow is modeled as a directed (often cyclic) graph with conditional branching, persistent checkpoints, and interruptible human-in-the-loop points, rather than a fixed linear chain.
Open problem: Byzantine fault tolerance in adversarial multi-agent settings remains an unresolved research area — don't assume agent-reported results are trustworthy without independent verification in high-stakes settings.
Model Routing / Cost Optimization
Use the cheapest model that can reliably complete the task.
Tasks |
Model Tier |
|
Classification |
Small |
|
Summaries |
Small / Medium |
|
Planning |
Medium / Large |
|
Coding |
Specialist |
|
Verification |
Large |
|
Legal / Compliance |
Premium deterministic |
Principle: Intelligence should scale with difficulty. Many current model families additionally expose configurable reasoning effort as a routing dimension in its own right — treat effort level as a tunable dimension alongside model size.
Reflection / Critique Loops
Use secondary reasoning for high-stakes decisions.
Trigger Conditions
● Low confidence
● High risk
● Contradictory evidence
● Large transaction
● Compliance-sensitive request
Reflection Flow
Human-in-the-Loop Design
Humans should be inserted intelligently.
Human Role |
Tasks |
|
Reviewer |
High-risk decisions |
|
Supervisor |
Real-time override |
|
Trainer |
Correct outputs |
|
Auditor |
Compliance review |
Confidence Handoff: Escalate when confidence < 0.85–0.90
Recommended use
What to Use When
Scenario |
When to use |
|
Highest transparency |
ReAct + logs |
|
Fastest cheap execution |
Small model routing |
|
Complex workflows |
Multi-agent workflow / task graph |
|
Safety critical |
Verifier + human gate |
|
Large enterprise memory |
Graph + Vector memory |
|
Heavy integrations |
MCP tool ecosystem |
|
Packaged reusable know-how |
Skills layer (New) |
|
High trust requirements |
Policy-first architecture |
|
Corpus fits in context window |
Load-and-ask, skip the retrieval pipeline (New) |
Production Deployment Checklist
Production Deployment Checklist
Before Launch
● Secrets in vault
● Rate limiting
● Retry logic
● Queueing
● Structured logs
● Health checks
● DB pooling
● CI/CD
● Rollback strategy
● Human approval console
● Policy engine live
● Audit logs enabled
● Monitoring dashboards
● Incident runbooks
● MCP servers pinned to a known-good tool-definition hash, with re-approval required on change (New)
● OAuth issuer validation enabled on every MCP client (New)
Build Strategy
Build in This Order
● Single agent + retrieval
● Add memory (start with Pattern B — retrieval-augmented)
● Add tools via MCP
● Add policy engine
● Add verifier
● Add human approvals
● Add multi-agent roles
● Optimize cost routing
● Formalize the harness — write down the planner/executor/verifier contracts and context-assembly rules as a versioned spec, not implicit code structure (New)
● Add simulations/evals, including harness-vs-model attribution (New)
● Expand autonomy slowly