Vector Databases for AI Agents

Vector databases are the long-term memory of AI agents. Without them, LLMs rely solely on what’s in their training data — which is often outdated, incomplete, or domain-specific. Vector databases enable agents to search semantically, remember earlier conversations, and retrieve domain-specific knowledge through RAG (Retrieval-Augmented Generation).

If you’re building an AI agent that does more than “just chat,” you’ll eventually need a vector database. This article explains the core concepts, compares the leading solutions, and includes complete working examples.

TL;DR — Vector Databases in 90 Seconds

What Is a Vector Database?

Embeddings — The Foundation

Before understanding vector databases, you need to understand embeddings. An embedding is a numerical representation of text — an array of hundreds or thousands of numbers that capture the “meaning” of that text.

from openai import OpenAI
client = OpenAI()

# Text → vector (1536 dimensions with text-embedding-3-small)
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Python is a programming language."
)

vector = response.data[0].embedding
print(f"Dimensions: {len(vector)}")  # e.g. 1536
print(f"First 5 values: {vector[:5]}")
# [0.012, -0.034, 0.056, 0.078, -0.091, ...]

The key principle: Texts with similar meaning have similar vectors. “Python is a programming language” and “Python is a scripting language” have vectors that lie close together. “Python is a snake” has a vector that’s much further away.

Similarity Search — How Vector Databases Find Things

Vector databases find similar vectors using distance metrics:

• Cosine Similarity: Measures the angle between vectors. Popular for text embeddings. A value of 1 means identical, 0 means unrelated.
• Euclidean Distance (L2): Measures the straight-line distance between vectors. Smaller distance means more similar.
• Dot Product: Fast, but less meaningful without normalization.

# Cosine similarity computed manually
import numpy as np

def cosine_similarity(v1, v2):
    return np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))

sim = cosine_similarity(vector_a, vector_b)
# 0.95 = very similar, 0.50 = moderately similar, 0.10 = barely similar

The Complete Workflow

STORING:
Text → embedding model → vector [0.12, -0.34, 0.56, ...] 
    → vector database (with metadata: source, date, category)

SEARCHING:
Query "What is Python?" → embedding model → vector 
    → similarity search in DB → top-K most similar vectors 
    → return corresponding texts

Why Do AI Agents Need Vector Databases?

1. Long-Term Memory

LLMs have a limited context window (e.g. 128K tokens for GPT-4o). Conversations longer than that need to be stored externally. Vector databases let you semantically find relevant earlier conversations and load them into context.

2. RAG — Retrieval-Augmented Generation

Instead of relying on the LLM’s training data, you search for relevant documents in the vector database and pass them as context:

User question → vector database search → top 5 relevant documents 
    → LLM prompt: "Answer the question based on these documents: ..."

This is the standard approach for domain-specific AI agents (e.g. customer support, internal knowledge bases, code documentation).

3. Semantic Search

Traditional search needs exact keywords. Vector databases find semantic similarity: “How do I write a loop?” also finds “Loop implementation in Python”.

4. Scalability

With millions of documents, keyword search becomes slow and imprecise. Vector databases use Approximate Nearest Neighbor (ANN) algorithms that can search billions of vectors in milliseconds.

The Leading Vector Databases — In Detail

Qdrant — High-Performance Rust Implementation

Qdrant is a vector database written in Rust, optimized for speed and production use. It supports filtering, payload metadata, and various distance metrics.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

# Connection (local or cloud)
client = QdrantClient(host="localhost", port=6333)
# Or cloud: QdrantClient(url="https://cluster-url", api_key="...")

# Create a collection (like a table in SQL)
client.create_collection(
    "agent_memory",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
    # Optional: production optimization
    optimizers_config=OptimizersConfigDiff(
        indexing_threshold=0,  # Index immediately
    )
)

# Insert vectors with metadata
client.upsert(
    "agent_memory",
    points=[
        PointStruct(
            id=1,
            vector=[0.12, -0.34, 0.56],  # In practice: 1536-dimensional
            payload={
                "text": "Python is a programming language.",
                "category": "programming",
                "source": "wikipedia",
                "date": "2026-07"
            }
        ),
        PointStruct(
            id=2,
            vector=[0.15, -0.31, 0.52],
            payload={
                "text": "Java is an object-oriented language.",
                "category": "programming",
                "source": "oracle-docs",
                "date": "2026-07"
            }
        )
    ]
)

# Semantic search with filtering
results = client.search(
    "agent_memory",
    query_vector=[0.11, -0.32, 0.55],
    query_filter=Filter(
        must=[
            FieldCondition(key="category", match=MatchValue(value="programming"))
        ]
    ),
    limit=5
)

for result in results:
    print(f"Score: {result.score:.4f} - {result.payload['text']}")

Strengths: Very fast (Rust), production-ready, filtering support, self-hosted or cloud options, open source. Weaknesses: Smaller community than Pinecone, no built-in embeddings (you need an external model). Best for: Production systems where performance and self-hosting matter.

Weaviate — GraphQL API with Built-In Embeddings

Weaviate is a vector database written in Go that comes with built-in embedding models. You don’t need to generate embeddings externally — Weaviate handles it automatically.

import weaviate

# Local connection
client = weaviate.connect_to_local()

# Create a collection with automatic embeddings
articles = client.collections.create(
    name="Article",
    vectorizer_config=weaviate.Configure.Vectorizer.text2vec_openai(),
    # Weaviate generates embeddings automatically!
)

# Add an object (no manual vectors required)
articles.data.insert({
    "title": "AI Programming 2026",
    "content": "Full article text...",
    "category": "programming"
})
# Weaviate generates the vector automatically

# Semantic search (near_text, not near_vector!)
results = articles.query.near_text(
    query="How does AI programming work?",
    limit=5,
    filters=Filter.by_property("category").equal("programming")
)

for obj in results.objects:
    print(f"{obj.properties['title']}: {obj.properties['content'][:100]}")

Strengths: Built-in embeddings (no external model needed), GraphQL API, hybrid search (vector + keyword), multi-modal support (text, images, audio). Drawbacks: Complex setup, higher resource requirements, limited flexibility in choosing embedding models. Best for: Projects requiring hybrid search where you’d rather not manage embeddings yourself.

Chroma — Simplest Integration for Python

Chroma (formerly ChromaDB) is the easiest vector database for Python. Written in Python itself, it needs no separate server — everything runs in-process.

import chromadb

# Persistent or in-memory
client = chromadb.PersistentClient(path="./chroma_data")
# Or in-memory: client = chromadb.Client()

# Create a collection
collection = client.create_collection(
    name="agent_memory",
    metadata={"description": "Long-term memory for AI agent"}
)

# Add documents (Chroma generates embeddings automatically with default model)
collection.add(
    documents=["Python is a programming language."],
    metadatas=[{"source": "wiki", "category": "programming"}],
    ids=["1"]
)

# Or with your own embeddings:
# collection.add(embeddings=[[0.12, ...]], documents=[...], ids=[...])

# Semantic search
results = collection.query(
    query_texts=["What is Python?"],
    n_results=5,
    where={"category": "programming"}  # Metadata filter
)

for doc, score, meta in zip(
    results["documents"][0],
    results["distances"][0],
    results["metadatas"][0]
):
    print(f"Score: {score:.4f} - {doc} (Source: {meta['source']})")

Strengths: Easiest integration (3 lines of code), free, no server required, automatic embeddings with default model. Drawbacks: Not designed for large scale (millions+ vectors), fewer features than Qdrant or Weaviate. Best for: Prototypes, development, small projects, local testing.

Pinecone — Managed Cloud Vector Database

Pinecone is a fully managed cloud vector database. No infrastructure, no maintenance, no scaling headaches — but it comes at a cost.

from pinecone import Pinecone

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("agent-memory")

# Upsert
index.upsert(
    vectors=[
        {
            "id": "1",
            "values": [0.12, -0.34, 0.56],  # 1536-dimensional
            "metadata": {"text": "Python is a programming language.", "category": "programming"}
        }
    ]
)

# Query with filtering
results = index.query(
    vector=[0.11, -0.32, 0.55],
    top_k=5,
    include_metadata=True,
    filter={"category": {"$eq": "programming"}}
)

for match in results["matches"]:
    print(f"Score: {match['score']:.4f} - {match['metadata']['text']}")

Strengths: Fully managed, scalable (billions of vectors), no infrastructure overhead, serverless option. Drawbacks: Costs (starting around $70/month), vendor lock-in, data stored in the cloud. Best for: Enterprise deployments when you want to avoid infrastructure management and have the budget.

Comparison

Feature	Qdrant	Weaviate	Chroma	Pinecone
Hosting	Self/Cloud	Self/Cloud	Self/Local	Cloud only
Language	Rust	Go	Python	Go
Embeddings	External	Built-in	External/Default	External
Hybrid Search	Yes	Yes	No	Yes
Scalability	High	High	Medium	Very high
Cost	OSS/Cloud	OSS/Cloud	Free	From ~$70/month
Filtering	Yes (payload)	Yes (GraphQL)	Yes (where)	Yes (metadata)
Multi-Modal	No	Yes (text/image/audio)	No	No
Best For	Production	Complex/Hybrid	Prototype	Enterprise

My recommendations:

• Development/Prototype: Chroma — 3 lines of code, free, no server
• Production (Self-Hosted): Qdrant — fast, Rust-based, open source, well documented
• Production (Managed): Pinecone — zero infrastructure if budget allows
• Hybrid Search Required: Weaviate — the only one with true vector + keyword hybrid search

RAG Pipeline with Vector Databases

RAG (Retrieval-Augmented Generation) is the most common use case for vector databases in AI agents. Here’s a complete pipeline:

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_qdrant import QdrantVectorStore
from langchain.text_splitter import RecursiveCharacterTextSplitter

# 1. Load and chunk documents
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,        # 500 tokens per chunk
    chunk_overlap=100,     # 100 token overlap between chunks
    separators=["\n\n", "\n", ". ", " "]
)
chunks = text_splitter.split_text(document_text)
# chunks = ["First section...", "Second section...", ...]

# 2. Generate embeddings and store in Qdrant
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = QdrantVectorStore.from_texts(
    chunks,
    embeddings,
    url="http://localhost:6333",
    collection_name="knowledge_base"
)

# 3. RAG query: Find relevant documents
llm = ChatOpenAI(model="gpt-4o", temperature=0)
retriever = vector_store.as_retriever(search_kwargs={"k": 5})

# 4. Generate answer using retrieved documents as context
from langchain.chains import RetrievalQA
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True  # Shows which documents were used
)

result = qa_chain.invoke({"query": "What is AI programming?"})
print(f"Answer: {result['result']}")
print(f"Sources: {len(result['source_documents'])} documents used")

What happens here:

1. Chunking: Document text is split into 500-token pieces (with 100-token overlap to preserve context)
2. Embedding + Storage: Each chunk becomes a vector stored in Qdrant
3. Retrieval: The user’s question is converted to a vector, and Qdrant finds the 5 most similar chunks
4. Generation: The LLM receives the question plus the 5 retrieved chunks as context and generates an answer

RAG in Multi-Agent Systems

In a multi-agent system, each agent can have its own vector database:

# Research Agent: Searches web scraping data
research_db = QdrantVectorStore(collection_name="web_research", ...)

# Code Agent: Searches code documentation
code_db = QdrantVectorStore(collection_name="code_docs", ...)

# Support Agent: Searches knowledge base articles
support_db = QdrantVectorStore(collection_name="kb_articles", ...)

Best Practices

1. Optimize chunk size

Chunk size is the most critical parameter for RAG quality:

• Too small (100 tokens): Loses context — the chunk alone doesn’t make sense
• Too large (2000 tokens): Imprecise search — the vector represents too many topics
• Recommendation: 500-1000 tokens with 100-200 token overlap

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " "]  # Natural breakpoints
)

2. Use metadata

Metadata enables filtering — critical for large databases:

# Store with metadata
collection.add(
    documents=["Article about Python"],
    metadatas=[{
        "category": "programming",
        "date": "2026-07",
        "author": "IRC",
        "source": "wiki",
        "language": "en"
    }],
    ids=["1"]
)

# Filtered search: Only English programming articles
results = collection.query(
    query_texts=["Python"],
    where={"$and": [
        {"category": {"$eq": "programming"}},
        {"language": {"$eq": "en"}}
    ]},
    n_results=5
)

3. Choose an embedding model

Model	Dimensions	Cost	Use case
text-embedding-3-small	1536	$0.02/M tokens	Standard, fast
text-embedding-3-large	3072	$0.13/M tokens	Higher quality
sentence-transformers (local)	384-768	Free	Development, privacy

Important: Switching embedding models requires re-embedding all vectors. Different models produce incompatible vectors.

4. Monitor performance

• Query latency: Should stay under 100ms for good UX
• Recall quality: Test against known query-document pairs
• Storage consumption: Vectors require memory (1536 dimensions × 4 bytes ≈ 6KB per vector)
• Index statistics: Qdrant and Pinecone both expose index metrics

5. Re-embedding strategy

When updating your embedding model:

1. Create a new collection
2. Re-embed all documents and store in the new collection
3. Run tests against the new collection
4. Switch your alias from the old collection to the new one
5. Delete the old collection

Key concepts for review

• Vector database: Stores embeddings (numerical vectors) for semantic search
• Embeddings: Numerical representation of text, generated by embedding models (e.g., OpenAI text-embedding-3-small, 1536 dimensions)
• Similarity search: Cosine similarity, Euclidean distance, dot product
• RAG: Retrieval-Augmented Generation — the LLM receives relevant documents from the vector database as context
• Chunking: Split text into pieces (500-1000 tokens, 100-200 overlap)
• Tools: Qdrant (Rust, production), Weaviate (Go, hybrid search, built-in embeddings), Chroma (Python, prototyping), Pinecone (cloud, enterprise)
• Best practices: Optimize chunk size, use metadata for filtering, choose embedding models deliberately, monitor, re-embed on model changes

FAQ

Which vector database should I start with? Chroma for prototyping — three lines of code, free, no server needed. Qdrant for production — fast, open source, self-hosted or cloud options.

Do I need a vector database for every AI agent? No. Only if your agent searches large knowledge bases, needs long-term memory, or uses RAG for domain-specific answers. Simple chatbots work fine with the LLM’s context window.

Can I run vector databases locally? Yes. Qdrant (Docker), Weaviate (Docker), and Chroma (Python process) all run locally. Pinecone is cloud-only.

How many vectors fit in a vector database? Qdrant and Pinecone: Billions. Weaviate: Millions. Chroma: Hundreds of thousands (then it slows down). The limit is usually storage, not the software.

What does running a vector database cost?

• Chroma: Free (self-hosted)
• Qdrant: Free (self-hosted) or cloud from ~$25/month
• Weaviate: Free (self-hosted) or cloud from ~$25/month
• Pinecone: From ~$70/month (cloud only)
• Embedding costs: text-embedding-3-small costs $0.02 per 1M tokens

Can I use vector databases with local embedding models? Yes. With sentence-transformers (e.g., all-MiniLM-L6-v2) you can embed locally for free. Vectors are smaller (384 dimensions) and quality is slightly lower than OpenAI, but sufficient for many use cases. Ideal for privacy-sensitive data.

Recommended reading