What is an AI agent? From my understanding, an AI agent is an integrated system that can perceive its environment, make decisions, and take actions to achieve specific goals. It often has these abilities:

Perception: The ability to sense and interpret the environment through various means.
Use LLM: The ability to utilize large language models (LLMs) to understand and generate human-like text, enabling it to communicate effectively and process information.
Planning: The ability to create a sequence of actions to achieve a specific goal.
Using tools: The ability to utilize external tools or resources to enhance its capabilities and achieve its objectives more efficiently.
Memory: The ability to store and recall information, allowing it to learn from past experiences and improve its performance over time.

Agent is often the final form of an AI system, which can be used to solve real-world problems. There are some drawbacks of agent system:

This agent system depends on the capabilities of the underlying LLM.
It is also depends on how the workflow of the agent is designed, such as how the agent interacts with the tools, how the agent plans its actions, how the agent uses its memory, etc.
A agent often has mutiple LLM calls, which can make the system slow.
The capablity of transforming the task to a new task is also important, if the agent can not transform the task to a new task, it may not be able to solve the problem.
It depends quite a lot on the quality of the prompts, if the prompts are not good enough, the performance of the agent will be limited.

I am a contributor at Camel AI and it is a community that focuses on the discovery the scaling laws of AI agents. At Camel AI, we believe that with the number of agent increasing, the performance of the system will also increase. We are interested in understanding how the performance of AI agents scales with the number of agents and how to optimize their interactions to achieve better results.

ReAct is a classic schema for AI agents, which stands for Reasoning and Acting. It is a framework that describes how an AI agent can reason about its environment and take actions to achieve its goals. The ReAct schema consists of the following steps:

flowchart TD
    Q[User Question / Task] --> R1[LLM Reasoning Step]
    R1 --> A1[Action Step: Tool / API / DB Call]
    A1 --> O1[Observation / Result]
    O1 --> R1[LLM Reasoning Step]
    R1 --> F[Final Answer]

From here, we can see tat the key components for an agent are the tools that we provided to the LLM. If the tools are not good enough, the performance of the agent will be limited.

Prompting for agents

When we design prompts for agents, we need to consider the following aspects:

Role: Define the role of the agent, such as a helpful assistant, a knowledgeable expert, or a creative problem solver. This helps the agent understand its purpose and how to approach the task.
Task description: Clearly describe the task that the agent needs to perform.
Constraints: Specify any limitations or constraints that the agent should consider while performing the task, such as time limits, resource constraints, or ethical guidelines.
Available tools: List the tools that the agent can use to perform the task, such as APIs, databases, or external resources. This helps the agent understand what resources it has at its disposal and how to utilize them effectively.
Performance evaluation: Define how the performance of the agent will be evaluated, such as through specific metrics, user feedback, or comparison to human performance and what is the output format of the answer.

Agent Schema

Baby AGI

flowchart TD
    PQ[Priority Queue]
    TL((Task List))
    NT[New Task]
    EA[Execution Agent]
    LM((Long-term Memory))

    TL <-->|Re-sort| PQ
    NT -->|Add task| TL
    TL -->|Get task| EA
    EA -->|Execution result| NT
    LM -->|Get context| NT
    EA <-->|Memory R/W| LM

HuggingGPT

In huggingGPT, the agent is designed to leverage the capabilities of various Hugging Face models to perform a wide range of tasks.

But this can of method can make the pipeline too long and the performance is not good enough.

LLAMAIndex

LLAMAIndex is a framework that allows you to build AI agents that can interact with various data sources and tools. The key is to improve the memory of the agent.

Llama index get data and create nodes. Each node has a unique id and the content of the data. Then we can use LLM to query the nodes and get the answer.

There are several way of indexing the data:

List Index: A simple list of nodes, where each node represents a piece of data. When querying, it is the key word matching that is used to find the relevant nodes.
Vector Index: Each node is represented as a vector in a high-dimensional space. When querying, the query is also converted into a vector, and the similarity between the query vector and the node vectors is calculated to find the relevant nodes. LlamaIndex can support local vector database and remote vector database like Milvus.
Tree Index: The nodes are organized in a tree structure, where each node can have child nodes. When querying, the tree is traversed to find the relevant nodes based on the query.
keyword Table Index: A keyword table is created, where each keyword is associated with a list of nodes that are relevant to that keyword. When querying, the keywords in the query are used to look up the relevant nodes in the keyword table.
Knoledge Graph Index: The nodes are organized in a graph structure, where each node can have relationships with other nodes. When querying, the graph is traversed to find the relevant nodes based on the query and the relationships between the nodes. The format is (Subject， Verb， Object)

Prompt Patterns

Chain-of-Thought (CoT) Prompting

In a Chain-of-Thought (CoT) prompt, when giving instructions to the model, you also provide a decomposition of the task execution process or examples of intermediate reasoning steps.

This helps the model reason through the task step by step instead of producing an answer directly.

Self-Review / Self-Critique

Before producing the final result, the model can be prompted to perform a self-review to check whether a better solution exists. Another approach is to review the result after it is generated and force the model to critique its own output.

For example, a guideline used in AutoGPT: “Constructively self-criticize your big-picture behavior constantly.”

This encourages the model to continuously evaluate and improve its reasoning.

Divide-and-Conquer Task Decomposition

When writing prompts, practitioners often observe that the more specific the context and goal are, the better the model performs. Therefore, it is often better to break a complex task into smaller subtasks and let the model solve them sequentially. Instead of asking the model to solve the entire problem in a single step, using external tools or agent frameworks follows the same idea and can be seen as a natural extension of Chain-of-Thought reasoning.

Plan-Then-Execute Pattern

A common pattern is: Planning stage, Execution stage. This pattern is used in many agent systems such as: BabyAGI, HuggingGPT, Generative Agents. The planning stage can also be extended so that the model:

asks clarification questions
refines the task objective
proposes several possible solutions

Then a human reviewer can confirm or provide feedback before execution, reducing the risk of deviating from the original goal.

Memory Systems

Agent systems often include memory mechanisms, such as: short-term memory,long-term memory. These support storing information, processing knowledge, retrieving relevant context. Memory systems are used in most agent projects and can enable a form of continuous learning or adaptive behavior in models.

MCP

MCP is a protocol for agent to communicate with tools. It can let the agent to use tools in your local system or a MCP server. Here is an example of using MCP to let agent fetch data on aws service:

flowchart LR
    User[User Input: Natural Language Query] -->|Prompt| BedrockLLM[Bedrock LLM]
    BedrockLLM -->|Function Call JSON| Orchestrator[Orchestration Layer]

    Orchestrator --> MCP[Custom MCP Function / Lambda / API]
    MCP -->|Fetch Data| DB[(Database)]
    MCP -->|Save Raw Data| S3[(S3 Bucket)]
    MCP -->|Return Metadata Only| Orchestrator
    Orchestrator -->|Metadata Summary| BedrockLLM
    BedrockLLM -->|Final Response| User

    subgraph Security & Best Practices
        S3 -->|Encrypted + IAM| DB
        MCP -->|IAM Restricted| S3
        Orchestrator -->|IAM Restricted| MCP
    end

Prompting for agents#

Agent Schema#

Baby AGI#

HuggingGPT#

LLAMAIndex#

Prompt Patterns#

Chain-of-Thought (CoT) Prompting#

Self-Review / Self-Critique#

Divide-and-Conquer Task Decomposition#

Plan-Then-Execute Pattern#

Memory Systems#

MCP#