Teaching the Shepherd to Remember
A Graph-RAG memory system that gives AI agents long-term recall, semantic search, and self-compacting knowledge.
Agents forget. Every conversation starts from zero unless you build memory into the system. Poimen Memory is a Rust-based Graph-RAG service that ingests conversations, extracts entities and relationships via LLM, stores them in a temporal knowledge graph backed by pgvector, and serves them back through three-tier retrieval — signature match at 50ms, graph-boosted hybrid search, and Obsidian fallback. The cache self-compacts: stale embeddings are evicted, frequently-accessed chunks are promoted, and the graph auto-reconciles on every write.
Architecture Overview
The memory system runs as a Rust microservice inside the Kubernetes poimen namespace. An actix-web API server handles HTTP requests, authenticates via Authentik OIDC JWT, and routes to either the search path (direct pgvector query) or the ingest path (async pipeline through an internal queue → LLM extraction → graph persistence). All data lives in a CloudNativePG cluster with HNSW indexes for vector similarity.
🏗️ System Architecture
Ingest Pipeline
Conversations arrive as episodes — raw message sequences. The pipeline splits into two parallel LLM extraction paths: entity extraction (people, tools, concepts with type + summary) and fact extraction (relationships between entity pairs as directed edges). A reflection pass filters hallucinated entities before persistence. The result is a temporal knowledge graph where every edge carries t_valid and t_invalid timestamps — knowledge that knows when it was true.
🔄 Ingest Data Flow
Ingest Request Lifecycle
The ingest path is fully async. The agent sends a POST /memory/ingest and gets a 202 Accepted immediately — no blocking on LLM latency. A background worker polls jobs from the queue, runs two sequential LLM calls (entity extraction → fact extraction), cleans JSON responses of thinking tags and fences, then persists entities, edges, and embeddings to pgvector. The agent can query the graph within seconds of ingestion completing.
⚡ Request Sequence
Three-Tier Retrieval
Tier 1: Signature Match
50ms exact entity lookup by name hash. Instant recall for known entities — no embedding computation needed.
Tier 2: Graph-Boosted Hybrid
HNSW cosine similarity + BFS graph traversal. RRF fusion ranks results across vector matches and graph neighbors. Context-aware retrieval that follows relationships.
Tier 3: Obsidian Fallback
When graph search yields low confidence, falls back to wiki-link indexed Obsidian vault. Bidirectional link traversal surfaces related notes the graph hasn't captured yet.