How are embeddings generated for text (Transformers)?

Pipeline: text to vector

The pipeline is: raw text → tokenization → transformer forward pass → pooling over token outputs → one embedding vector. The tokenizer splits text into subword or word tokens and feeds them to the transformer; the model outputs a representation per token. To get a single vector per sentence or passage, we pool: often the [CLS] token output (in BERT-style models) or mean/max pooling over all token representations. The result is a fixed-length dense vector, typically normalized to unit length for cosine similarity or dot-product search in the vector database.

Each step affects downstream quality and performance. Tokenization determines vocabulary coverage and handling of rare or out-of-vocabulary terms. The choice of pooling (CLS vs. mean vs. max) can favor different aspects of the text—CLS is trained to summarize the sequence in BERT, while mean pooling often gives more stable sentence-level representations. Normalization is required when your vector database uses cosine or inner-product similarity so that scores are comparable across documents of different lengths or styles.

Model families and training

Models like BERT, Sentence-BERT (SBERT), and modern embedding models (e.g. OpenAI text-embedding-3, Cohere embed, E5, BGE) are trained so that semantically similar sentences have similar vectors. Training is often contrastive or triplet-based: pull together matching pairs, push apart non-matching. That creates the latent space where “close” means “similar in meaning,” which is what makes semantic search and vector query useful.

Trade-offs between model families include size (speed vs. quality), dimensionality (storage and index build cost), max sequence length (how much text per vector), and language or domain coverage. Encoder-only models (BERT-style) are standard for symmetric tasks (e.g. passage retrieval); some systems use separate query and document encoders. When to use which: small models for low latency and high throughput, larger models when recall and nuance matter more than cost.

Choosing the right model affects dimensionality and quality; you can fine-tune for your domain. For ingestion at scale, batching embedding calls is common. Fine-tuning on in-domain pairs (query–passage or passage–passage) usually improves retrieval metrics; ensure your evaluation set reflects real queries to avoid overfitting.

Using text embeddings in a VDB

Storing these vectors in a VDB lets you query by meaning rather than keywords—the core of RAG and semantic retrieval. The same model must be used for indexing and for embedding the query so that query and corpus live in the same space. See handling model version drift when you change or update the model.

Practical tips: run the same preprocessing (and tokenizer) at index and query time; align max input length with your chunking strategy so that chunks fit within the model’s context; monitor for distribution shift if your content or query mix changes. For multi-tenant or multi-domain setups, either use one model with broad training or maintain separate collections per model/domain so that distances remain comparable within each space.