Storing raw data vs. storing only vectors

Vectors only: lean VDB, external content

Storing only vectors keeps the VDB small and fast: you do a nearest-neighbor search, get back IDs, then look up the actual content in another store (e.g. PostgreSQL, object store). That separates search from storage and lets you change or redact raw data without touching the index, but adds a second round-trip and requires keeping IDs in sync.

Storing metadata (e.g. category, date) alongside the vector in the VDB allows metadata filtering at query time so you only search within a subset; many VDBs support this with payloads or columns. Trade-off: vectors-only minimizes VDB size and can improve privacy (no raw text in the VDB), but you need a reliable external store and ID mapping. When to use: when you want a clear separation between search and content, or when content is large or frequently updated.

Storing raw data in the VDB

Storing full raw data (e.g. the original text) in the VDB as a payload is convenient for returning snippets or titles directly from the search response and simplifies the architecture, at the cost of higher storage and potentially larger responses. See storing payloads alongside vectors for implementation details.

Practical tip: keep payloads lean if you need low latency; put large content in object storage and store only a reference in metadata. Many VDBs let you attach a payload (e.g. JSON) with each point—you can store title, snippet, doc_id, and optionally full text, subject to size limits. If you need to redact or update stored text, update the payload (many VDBs support update by ID); if you store only vectors and text elsewhere, update the external store—the vector stays valid unless you change the underlying content and re-embed.

Common pattern for RAG

For RAG and semantic search, a common pattern is: store vectors + minimal metadata (e.g. doc_id, chunk_id) in the VDB, and keep full documents in a separate store keyed by ID so you can fetch and pass only the needed chunks to the LLM. That keeps the VDB focused on search while the document store holds the source content.

For compliance, storing only vectors reduces exposure of raw content in the VDB; see privacy and reconstruction. You still need to protect the external store and control who can resolve IDs to content. Pipeline: index pipeline writes vectors (and optionally payload) to VDB and may write full docs to another store; query pipeline runs ANN, gets IDs (and optional payload), then fetches full content if needed.