How are embeddings generated for images (CNNs/ViTs)?

CNN-based image embeddings

In CNNs, the image passes through convolutional and pooling layers that capture local patterns and hierarchy; the final layer before the classifier (e.g. the last fully connected layer or a global average pooling output) is used as the embedding. That vector is a feature vector summarizing the image content. Models like ResNet, EfficientNet, or older VGG are often used with their classification head removed and the penultimate layer taken as the embedding.

The result is fixed-length and dense, suitable for cosine or L2 similarity in a vector database. CNN embeddings are well understood, efficient on CPU/GPU, and widely supported. Trade-off: they can lag behind ViTs on fine-grained or abstract visual similarity because they were originally trained for classification; for best quality on modern benchmarks, ViT or hybrid backbones are often preferred. When to use CNNs: when you need low latency, smaller models, or compatibility with older pipelines.

Vision Transformers (ViTs)

In ViTs, the image is split into patches, each patch is linearly embedded, and transformer layers process the sequence of patch tokens (plus a [CLS] token). The [CLS] token output or the mean of patch outputs gives the image vector. ViTs often outperform CNNs for representation quality and are the backbone of many modern image encoders.

Both CNN and ViT yield a single vector per image that can be normalized and stored in a collection for nearest neighbor search. ViTs typically need more compute and data for training but offer better transfer and robustness to distribution shift. Practical tip: use the same preprocessing (resize, crop, normalization) at index and query time so that distances are meaningful.

Joint text–image and use cases

For joint text–image search (e.g. “find images like this description”), multi-modal models like CLIP embed both text and images into the same space so one index can serve both. Image-only embeddings power visual search, duplicate or near-duplicate detection, and recommendation (“images like this”). Once generated, they are indexed like any other vector in a VDB and queried via the same lifecycle.

When to use image-only vs. multi-modal: use image-only encoders when queries are always images (e.g. “find similar product photos”). Use CLIP-style multi-modal when you need to query with text, with images, or both. Pipeline tip: batch image encoding for bulk ingestion and consider dimensionality and index type (e.g. HNSW vs. IVF) based on scale and latency targets; see choosing the right embedding model and impact of embedding model dimensionality for VDB performance.