What is the “Latent Space”?

Definition and intuition

When an embedding model (e.g. a transformer) maps text or images to vectors, it’s projecting them into a latent space. The model is trained so that items that “belong together”—same topic, similar meaning, related image—end up close, while unrelated items are farther apart. The dimensions of this space aren’t interpretable like “age” or “price”; they’re learned abstractions. Distance in this space is then a proxy for semantic or structural similarity, which is what we use for nearest-neighbor and semantic retrieval. A vector database effectively indexes this latent space so you can query “points near this vector” at scale.

The geometry of the latent space is determined entirely by the model’s training objective (e.g. contrastive loss, triplet loss). Good embedding models produce spaces where meaningful clusters and gradients emerge, making similarity search effective without any hand-tuned features.

Why “latent”

“Latent” means hidden or inferred: the dimensions aren’t directly observed in the raw data; the model learns them during training. So you can’t label axis 1 as “topic” and axis 2 as “sentiment”—the space is a continuous representation where geometry encodes similarity. That’s why we rely on distance metrics (cosine, L2) rather than interpreting coordinates. The same idea appears in other ML settings (e.g. VAEs, GANs) where “latent space” is the learned representation space.

In vector search we care about relative distances and neighborhoods, not individual coordinates. That’s why metrics like cosine similarity and L2 distance are central: they summarize “how close” two points are in this unknown geometry.

Dimensionality and the curse of dimensionality

The number of dimensions is the latent space’s dimensionality (e.g. 768, 1536). High dimension can make search harder (curse of dimensionality): in high dimensions, most points are roughly equidistant, and naive partitioning doesn’t create tight regions. That’s why vector DBs use specialized indexes (HNSW, IVF) rather than brute-force scan. So in short: latent space = the learned space of embeddings where “close” means “similar”; it’s the geometry that makes vector search meaningful.

Dimensionality is fixed by the model. You can’t mix vectors of different dimensions in the same index; they don’t live in the same space. Reducing dimension (e.g. via PCA) is sometimes used for visualization or to save memory, but it usually changes distances and can hurt recall.

One space per model

Each embedding model defines its own latent space. Vectors from different models are not comparable: you can’t meaningfully compute distance between a vector from model A and a vector from model B. So in a collection, all vectors must come from the same model (and same normalization). If you change models, you need to re-embed and re-index.

Multimodal models (e.g. CLIP) define a single latent space for multiple modalities—text and images share the same space so you can search images with a text query. That’s still one model, one space; the constraint “same model for all vectors in a collection” remains.