Building Agent Memory with Vendor Lock-in Resistance (Part 2)

In my previous post, How I Build Agent Memory Free from Vendor Lock-in, I proudly showed off my Two-Tier Hybrid RAG + NAS Markdown memory scheme for my AI assistant, Nouva. When I first deployed it, it felt like a dream. Simple, portable, and not overkill.

But as we all know in software engineering, no solution is "done and perfect forever" on the first try. After running this setup for a few weeks, I bumped into three frustrating issues:

1. Index Bloating: The flat MEMORY_INDEX.md file grew way too fast. Because it accumulated daily topic lists from the beginning of time, it got heavier by the day. As a result, the AnythingLLM RAG started struggling, and semantic search accuracy degraded due to excessive noise.
2. Vector Dilution: Semantic scores from the vector database were often too close to differentiate between dates. The AI easily confused a serious server architecture discussion from last week with a casual chat on a different day, just because of some random overlapping keywords.
3. Infinite Graph Traversal: When trying to follow threaded chats (continue_of / related_dates), without a boundary, the script could pull dozens of markdown files from the NAS at once. It was painfully slow.

So yesterday, I decided to pull off a major refactor of Nouva's memory system. The solution? Two-Tier Hierarchical Summary & Hybrid Scoring.

---

Architecture Comparison: Before vs. After

To make it easier to visualize, here is a comparison of Nouva's memory system flow before and after this refactoring:

1. Previous Architecture

In the initial architecture, semantic search directly targeted the flat MEMORY_INDEX.md which grew indefinitely, then fetched raw transcripts from the NAS without any re-ranking.

2. New Architecture

In the new architecture, we introduce a Hybrid Scoring layer (Semantic + Importance + Recency) to filter candidate dates before fetching raw transcripts from the NAS. The synchronization process is also enhanced with a Reconciliation Pass and Daily Summary Generation to keep everything tidy and indexed.

Let's break down the key updates one by one.

---

1. Fighting Bloat with Daily Summary & YAML Frontmatter

Previously, we relied on raw daily notes as the sole memory source in RAG. Now, before those daily notes are archived to the NAS, the auto-sync.py script calls an LLM to generate a new file: memory/summaries/YYYY-MM-DD.summary.md.

This summary file is super clean and features a structured YAML frontmatter:

---
schema_version: 1
date: 2026-07-06
people: [Gading, Rina, Freedy]
projects: [Nouverse Core, Homelab]
tags: [Kubernetes, Docker, RAG]
importance: 7
mood: excited
related_dates: [2026-07-01]
uncategorized:
  technologies: []
  libraries: [Pydantic]
  projects: []
---
### Today's Summary
- Completed the refactor of Nouva's agent memory with the hybrid scoring scheme.
- Discussed K3s cluster worker nodes setup with Freedy.

• Controlled Vocabulary: The projects, technologies, and libraries fields must match a pre-defined list in memory_config.json. If a new term appears (like the Pydantic library which isn't registered yet), the LLM places it under uncategorized to keep the main vocabulary clean.

  You might wonder, "Why hardcode the controlled vocabulary and scoring weights in a static memory_config.json file?" The answer is simple: because this memory system is purely for my personal use. At a personal scale, a static configuration that is re-read per invocation is more than enough, highly secure, and avoids adding the unnecessary complexity of a dynamic database layer or a hot-reloaded service.

• Idempotency & Reconciliation: I added a reconcile_missing_summaries() pass. If the LLM proxy timeouts or crashes mid-sync, the next cron job automatically scans for missing summaries and regenerates them. No more silent data loss.

---

2. Hybrid Scoring: Balancing Semantics, Importance, and Recency

Relying solely on RAG semantic scores means old, highly important architectural decisions get buried under newer, less important daily chats. Conversely, fresh casual banter often bubbles up to the top.

To fix this, I implemented a custom Hybrid Scoring formula in query-memory.py:

$\text{Final Score} = (w_{\text{semantic}} \times S_{\text{semantic}}) + (w_{\text{importance}} \times S_{\text{importance}}) + (w_{\text{recency}} \times e^{-c \times \text{days}})$

The weights are tuned in memory_config.json:

• Semantic (50%): The raw similarity score from the AnythingLLM vector database.
• Importance (30%): A scale of 1-10 from the YAML metadata (architectural migrations get an 8-10, casual chat gets a 1-3).
• Recency Decay (20%): Exponential time decay. Newer notes naturally get a boost.

With this math, a critical system design discussion from 3 months ago (high importance) won't lose to a skincare reminder from last night (low importance).

---

3. Keep it Simple: Replacing BFS Traversal with Programmatic 1-Hop Expansion

Initially, when designing this system, I wrote a complex graph traversal algorithm using Breadth-First Search (BFS) with a strict depth limit (max_depth = 2) to automatically crawl and pull threaded daily notes.

However, after testing it in production, I realized this recursive BFS traversal was slow and highly prone to date hallucination by the LLM (the LLM frequently hallucinated past dates that didn't actually have logs).

Ultimately, I refactored this approach to make it much simpler and deterministic:

1. Programmatic Topic Detection (Sync-time): During the daily sync process, our Python script automatically scans the new summary's metadata and matches it against keywords in the local MEMORY_INDEX.md file. The top 2 most relevant dates are injected into the YAML metadata as related_dates. This has zero LLM token cost and guarantees 100% valid dates.
2. 1-Hop Expansion (Retrieval-time): When the AI agent queries memory (query-memory.py), we no longer run a recursive BFS traversal. We simply take the candidate dates from the Semantic RAG and perform a 1-hop expansion to their related_dates, decaying their semantic score by 30% (score * 0.7).

With this approach, we still get the benefits of document relationships (topic clusters), but the code remains lightweight, fast, and free from complex traversal bugs. Keep it simple!

---

4. Obsidian Vault Compatibility (A Beautiful Side Effect)

Because our file naming convention (YYYY-MM-DD.md) and the new summary files use standard YAML headers, I got a wonderful bonus: the memory/ folder can be mounted directly as an Obsidian vault!

I just added a quick instruction to the summary generator prompt to append wikilinks at the bottom of the markdown body:

---
# YAML Frontmatter...
---
### Today's Summary
- [Point 1]

---
**🔗 Links:** [[Gading]] · [[Nouverse Core]]

When opened in Obsidian on my laptop, I instantly get Graph View, Backlinks, and a Calendar Heatmap out-of-the-box, without writing a single line of visualization code. The RAG benefits from clean metadata, and I get a gorgeous UI to browse Nouva's memory history.

However, to keep the graph clean and prevent it from turning into a chaotic spiderweb, I applied a few extra rules:

1. No Related Dates Wikilinks: The relationships between dates (related_dates) are intentionally not rendered as visual wikilinks in the markdown body. They are kept only in the YAML frontmatter to be read programmatically by the AI. This keeps the graph free from confusing criss-crossing lines between dates.
2. Timeline Spine: Each daily note automatically links to the previous day's note (e.g., « [[2026-07-05]] | Timeline Spine). This forms a straight chronological line (a spine) from left to right in the graph.
3. Parent Link: Raw session transcripts automatically link to their parent daily note. This forces chat sessions to orbit neatly around their corresponding date node instead of floating around as orphans.

Here is a preview of Nouva's memory graph view generated automatically in Obsidian:

---

Backfill & Migrating Historical Data

The final piece of the puzzle is historical data already archived on the NAS. To align it, I wrote a one-time migration script (backfill-nas-memories.py) that:

1. Scans and reads old daily notes from the NAS.
2. Generates the new .summary.md files via the LLM incrementally (with a time.sleep(2) delay to avoid hammering the LLM).
3. Saves the summary files in the local memory/summaries/ directory (their footprint is tiny, so they stay local).
4. Syncs all new summaries to AnythingLLM.

So, do I finally need an expensive, complex external memory stack? The answer remains no. I can still "hack" my way through using local markdown files, a few Python scripts, and simple hybrid scoring math to fit my real needs without wasting LLM tokens or depending on third-party vendors.

This refactoring proves that with a well-structured markdown folder, YAML headers, a bit of hybrid scoring math, and standard markdown tools like Obsidian, you can build a smart, lightning-fast, and 100% self-owned AI Agent memory system.

Related Blogs

This memory architecture didn't happen overnight. It is the evolution of several experiments and failures before it:

1. Building GraphRAG for Autonomous Agents with Neo4J and Graphitti (The initial phase of trying GraphRAG)
2. The GraphRAG Trap: Why I Uninstalled Neo4j for My Personal Assistant (The decision to uninstall due to overkill)
3. How I Build Agent Memory Free from Vendor Lock-in (Initial solution using flat RAG + NAS)