Agent Memory

The Problem:

By default, LLMs are stateless - they have no memory between API calls. Each conversation starts completely fresh, as if the AI has amnesia. This is like a customer support agent who forgets you the moment you hang up and call back.

Memory transforms a stateless AI into a personalized assistant that knows your preferences, remembers past conversations, and builds knowledge over time.

Real-World Example - Your Favorite Chai Wala:

Think about the chai wala outside your office. After a few visits:

• He remembers you like cutting chai with less sugar
• He knows you come at 10 AM and 4 PM
• He remembers you asked about green tea last week and tells you he got it now
• He knows your colleague prefers black coffee and asks if he is coming today

This is exactly what agent memory does - it turns a generic AI into one that knows you and adapts to you.

Types of Agent Memory:

How It Works:

Working memory is simply the list of messages in the current conversation that gets sent to the LLM with each API call. It includes user messages, assistant responses, tool calls, and tool results.

The challenge: LLMs have a finite context window (4K to 200K tokens). Long conversations eventually exceed this limit. You need strategies to manage this.

Context Window Management Strategies:

• Sliding Window: Keep only the last N messages. Simple but loses early context. Like forgetting what was discussed at the start of a long meeting.
• Summarization: Periodically summarize older messages into a compact summary. Keep the summary + recent messages. Like meeting notes - you do not remember every word but you have the key points.
• Token Budget: Set a max token budget for history. Truncate oldest messages when approaching the limit.
• Smart Selection: Use an LLM to decide which past messages are most relevant to the current question and include only those.

Best Practice - Hybrid Approach:

Context sent to LLM:

1. System Prompt (fixed)
2. Long-term memory facts (from DB)
   "User prefers vegetarian food, lives in Bangalore"
3. Summary of older conversation
   "Earlier we discussed flight options to Goa for March..."
4. Last 10 messages (full detail)
5. Current user message

Total: Fits within context window while preserving key information

The Core Challenge:

When a conversation ends, all working memory is lost. Long-term memory solves this by persisting important information to a database so it can be retrieved in future conversations.

What to Store in Long-term Memory:

• User Preferences: "Prefers window seats, vegetarian, likes technical explanations"
• Learned Facts: "Works at TCS as a senior developer, team of 8 people"
• Past Decisions: "Chose React over Angular for the new project (Feb 2026)"
• Interaction Patterns: "Usually asks about code review on Mondays, deployment help on Fridays"

Storage Approaches:

Memory Extraction - How the Agent Decides What to Remember:

After each conversation turn, run a "memory extraction" prompt:

"Based on this conversation, extract any new facts about the user
that should be remembered for future conversations.

Conversation: [last few messages]

Extract in JSON format:
{
  "preferences": [...],
  "facts": [...],
  "action_items": [...]
}"

Example output:
{
  "preferences": ["Prefers dark mode", "Likes concise explanations"],
  "facts": ["Working on e-commerce project with Next.js"],
  "action_items": ["Follow up on deployment issue next week"]
}

What is Episodic Memory?

Episodic memory stores complete past experiences - not just facts, but the full context of what happened, what the agent did, and what the outcome was. This lets the agent learn from past successes and failures.

Think of it like a doctor who remembers: "Last time a patient had these symptoms, I prescribed X and it worked. But for patient Y with similar symptoms, X did not work because of their allergy, so I used Z instead."

Example - Customer Support Agent:

Episodic Memory Store:

Episode 1: (Success)
  Context: User complaint about slow Swiggy delivery
  Action: Checked order status, found restaurant delay
  Resolution: Offered 20% discount coupon
  Outcome: User satisfied (5-star rating)
  Learning: Restaurant delays -> proactive discount works well

Episode 2: (Failure)
  Context: User complaint about wrong item delivered
  Action: Offered refund
  Resolution: User wanted replacement, not refund
  Outcome: User frustrated (1-star rating)
  Learning: Wrong items -> always ask "refund or replacement?" first

Now when a new complaint comes in, the agent searches
episodic memory for similar situations and applies
the learned strategy.

How to Implement Episodic Memory:

• Store episodes: After each completed task, save the context, actions taken, and outcome
• Embed episodes: Convert to vector embeddings for semantic search
• Retrieve similar episodes: When a new task comes in, search for similar past episodes
• Apply learnings: Include relevant past episodes in the prompt as examples
• Score outcomes: Track success/failure to weight episodes (prefer successful strategies)

Production Memory Architecture:

[Agent receives user message]
        |
        v
[Memory Retrieval Phase]
  |-- Fetch user profile (Redis/Postgres)
  |-- Search semantic memory (vector DB)
  |     Query: embed(user_message) -> top 5 relevant memories
  |-- Search episodic memory (vector DB)
  |     Query: similar past situations -> top 3 episodes
  |-- Get conversation summary (Redis)
        |
        v
[Compose Context]
  System prompt + user profile + relevant memories
  + episodes + conversation summary + recent messages
        |
        v
[LLM generates response]
        |
        v
[Memory Storage Phase (async, after response)]
  |-- Extract new facts -> store in long-term memory
  |-- Update user profile if new preferences detected
  |-- If task completed -> store as episode
  |-- Update conversation summary

Frameworks with Built-in Memory:

• Mem0 (formerly MemGPT): Purpose-built memory layer for AI agents. Handles extraction, storage, and retrieval automatically. Very popular.
• LangGraph Persistence: Built-in checkpointing and state persistence for LangGraph agents.
• Zep: Long-term memory service for AI assistants. Provides automatic fact extraction and temporal awareness.
• Letta: Stateful AI agents with self-editing memory. The agent decides what to remember and forget.

Memory Gotchas:

• Stale Memories: User changed preferences but old memory persists. Solution: timestamp memories and prefer recent ones.
• Contradictions: New info contradicts stored memory. Solution: detect conflicts and update (or ask user).
• Privacy: Storing personal data has legal implications (GDPR, DPDPA). Solution: clear consent, data deletion capability, encryption.
• Memory Bloat: Storing too much irrelevant info degrades retrieval quality. Solution: periodic cleanup, relevance scoring.

Q: What are the different types of memory in AI agents?

Q: How do you handle context window limits for long conversations?

Q: Vector DB vs Key-Value store for agent memory?

Q: How does episodic memory help agents improve over time?