The Vector Database I Almost Built
Every AI lab is selling you a vector memory layer. Most of them solve a problem you do not have. They add one you did not want.
A few months ago I sat down to add a vector store to our memory system. I had the case ready. Embeddings are cheap now. Semantic search beats grep. Every serious AI stack has a vector layer. Pinecone. Chroma. Weaviate. Qdrant. Take your pick. The pattern is everywhere.
I almost did it.
Then I thought about it for ten more minutes and did not.
Why I stopped
Our memory is a tree of flat files. Markdown notes. JSON logs. Plain text transcripts. Sorted by topic in a folder called MEMORY. Every agent on every host can grep across it in milliseconds. Every tool we have built reads from it and writes to it the same way. Git tracks each change. Backups are just rsync. If something breaks, I can open a file and read it.
The thing that nags me about a vector store is that it would become a second source of truth. Two stores. Two things that have to stay in sync. The flat files would still be the truth. But the vector index would be the thing the agents query first. The moment the index falls out of sync, you get an agent stating notes that do not exist any more. Or missing notes that do.
I have been burned by that pattern. Twice. Both times the symptom was the same. An agent quoted something with full faith. The quote was real two weeks ago. Before the source moved or got rewritten. The index did not know.
What I'd do instead
The right way to add semantic search is not to swap out the file system. It is to layer on top of it.
A SQLite store with full text search built in. Maybe a small vector add-on if I really want embeddings. Built as a view of the flat files. Rebuilt on its own when files change. Always able to be rebuilt from scratch. If the index ever lies to me, I blow it away and rebuild. The truth stays where it always was. In the files.
That is the shape I would trust. Not because it is clever. Because it makes one source of truth and one speed-up. Not two stores that argue.
When I'll actually build it
When grep starts hurting.
Right now there are about two hundred active files in our memory tree. Grep is instant. Agents find what they need by name or by topic. The system works.
At two thousand files it will be slower. At ten thousand it will be a real choke point. That is where the index earns its keep. Until then, building it now would be solving a problem I do not have. With a pattern that adds the fail modes I most want to dodge.
Engineers do this thing where we build setup we do not yet need. The pattern is everywhere and it feels right. That is also how you end up with three caches. Two queues. A service mesh. And a bug nobody can find.
The wider point
Every AI lab right now is selling you the same thing. A vector memory layer. A graph of your data. A retrieval tool that "remembers everything." Most of them are good ideas that do not fit your case. Most of them are good ideas that add a sync problem you did not have.
If your data fits in grep, you do not need a vector store yet. You need to do the work of writing things down clearly. Naming them well. Putting them where you can find them. That part does not get any easier when you add embeddings on top.
I will add the index when grep stops being enough. I will not add it because Andreessen Horowitz said I should.
References:
- SQLite FTS5 full text search
- sqlite-vss vector extension
- Simon Willison on local vector search
- Why I don't use embeddings for everything (Hamel Husain)