Hybrid search is an effective innovation that involves multiple search algorithms. To deliver more accurate results, it combines the best of traditional keyword search and modern semantic search. This practice allows the search engine to deliver the results based on exact keyword match as well as contextual understanding.
In particular, hybrid search is valuable for applications that are based on Retrieval-Augmented Generation (RAG). Hybrid search enables RAG-based systems like AI agents to understand a wide range of natural language enquiries and deliver results that improve customer experience and business performance.
In this blog, we will dive deep into how hybrid search is leveraging semantics and the keyword search model to deliver the best search results
Introduction to Enhanced Information Retrieval
Retrieval-Augmented Generation is a groundbreaking paradigm that extends the capabilities of Large Language Models (LLMs) by tapping into external sources of knowledge. Instead of relying solely on training data, RAG systems dynamically extract appropriate information from knowledge bases, greatly improving response accuracy and factuality.
The secret to successful RAG implementation is advanced retrieval mechanisms. The mechanisms are primarily based on two basic strategies: lexical matching methods and semantic vector space models. Although every approach has its own merits, the limitations of individual methods have catalyzed the emergence of hybrid approaches, which leverage the synergistic benefits of the two methods.
Understanding Hybrid Search Architecture
Hybrid search represents an innovative fusion of lexical retrieval (sparse vectors) and semantic search (dense vectors) methodologies. This sophisticated approach addresses the inherent weaknesses of individual techniques by creating a unified scoring mechanism that evaluates documents from multiple perspectives.
What is Lexical retrieval(sparse vectors)?
Sparse vectors represent documents and queries as high-dimensional vectors where most elements are zero, with non-zero values corresponding to specific terms or features present in the text.
Characteristics of Sparse Vectors:
- High Dimensionality: Vector size equals vocabulary size (typically 10K-1M dimensions).
- Sparsity: Only a small fraction of dimensions have non-zero values.
- Interpretability: Each dimension corresponds to a specific term or n-gram.
Exact Matching: Excels at capturing precise lexical overlap.
Common Sparse Vectors models:
1. TF-IDF (Term Frequency-Inverse Document Frequency)
Architecture:
TF-IDF(t,d) = TF(t,d) × IDF(t)
Where:
– TF(t,d) = (count of term t in document d) / (total terms in d)
– IDF(t) = log(N / df(t))
– N = total documents, df(t) = documents containing term t
Model Structure:
- Input Layer: Raw text tokenization
- Feature Extraction: Term frequency calculation
- Weighting Scheme: IDF normalization
- Output: Sparse vector with vocabulary-sized dimensions
2. BM25 (Best Matching 25)
Architecture:
BM25(q,d) = Σ IDF(qi) × [f(qi,d) × (k1 + 1)] / [f(qi,d) + k1 × (1 – b + b × |d|/avgdl)]
Model Components:
- Saturation Function: Prevents term frequency overflow
- Length Normalization: Adjusts for document length bias
- Parameter Tuning: k1 (1.2-2.0) and b (0.75) for optimization
- Collection Statistics: Incorporates corpus-wide term distributions
3. SPLADE (Sparse Lexical and Expansion)
Modern Neural Sparse Architecture:
Input Text → BERT Encoder → MLM Head → ReLU → Log-Saturation → Sparse Vector
Key Features:
- Neural Backbone: Leverages BERT’s contextual understanding
- Expansion Mechanism: Generates additional relevant terms
- Learned Sparsity: Neural networks determine important dimensions
- Interpretable Output: Maintains term-level interpretability
Dense Vectors: Semantic Understanding Through Embeddings
Dense vectors represent documents and queries as fixed-size, low-dimensional vectors where every element typically contains non-zero values, capturing semantic relationships and contextual meaning.
Characteristics of Dense Vectors:
- Lower Dimensionality: Typically 128-1024 dimensions
- Dense Representation: All dimensions contain meaningful values
- Semantic Capture: Encodes conceptual relationships and context
- Continuous Space: Enables smooth similarity measurements
Common Dense Vector models:
1. Word2Vec Family
– Skip-gram Architecture:
Input Word → Embedding Layer → Hidden Layer → Softmax Output (Context Words)
– CBOW (Continuous Bag of Words) Architecture:
Context Words → Embedding Layer → Average → Hidden Layer → Softmax (Target Word)
Model Specifications:
- Embedding Dimension: 100-300 dimensions
- Context Window: 5-10 surrounding words
- Training Objective: Predict context from words or vice versa
- Limitations: Word-level representations, no contextual variation
2. Sentence-BERT (SBERT)
Siamese Network Architecture:
Input Text → BERT Encoder → Pooling Layer → Normalization → Dense Vector
Pooling Strategies:
- CLS Token: Using [CLS] token representation
- Mean Pooling: Average of all token embeddings
- Max Pooling: Maximum values across token dimensions
Popular SBERT Models:
- all-MiniLM-L6-v2: 384 dimensions, fast inference
- all-mpnet-base-v2: 768 dimensions, high quality
- all-distilroberta-v1: 768 dimensions, balanced performance
3. E5 (EmbEddings from bidirEctional Encoder rEpresentations)
Multi-task Training Architecture:
Text Input → Encoder (DeBERTa/RoBERTa) → Pooling → L2 Normalization → Output
Training Methodology:
- Contrastive Learning: Positive/negative pair optimization
- Multi-task Objectives: Various downstream tasks
- Large-scale Training: Billions of text pairs
- Cross-lingual Capability: Multilingual understanding
E5 Model Variants:
- E5-small: 384 dimensions, efficient processing
- E5-base: 768 dimensions, standard performance
- E5-large: 1024 dimensions, highest quality
4. BGE (Beijing Academy of Artificial Intelligence General Embedding)
Optimized Retrieval Architecture:
Input → Encoder (BERT/RoBERTa) → Representation → Contrastive Training → Dense Vector
Key Innovations:
- Retrieval-focused Training: Optimized for search tasks
- Hard Negative Mining: Advanced negative sampling
- Cross-encoder Distillation: Knowledge transfer from reranking models
- Multiple Languages: Comprehensive multilingual support
BGE Model Family:
- BGE-small-en: 384 dimensions, English-focused
- BGE-base-en: 768 dimensions, balanced English model
- BGE-large-en: 1024 dimensions, premium English performance
- BGE-M3: Multilingual, multi-functionality model
5. OpenAI ada-002 and text-embedding-3
Transformer-based Architecture (Proprietary):
Input Text → Multi-layer Transformer → Attention Mechanisms → Dense Representation
Model Characteristics:
- text-embedding-ada-002: 1536 dimensions
- text-embedding-3-small: 1536 dimensions (configurable)
- text-embedding-3-large: 3072 dimensions (configurable)
- Advanced Training: Large-scale, diverse training data
The Rationale Behind Hybrid Approaches
Lexical Retrieval Strengths and Limitations:
- Excels at exact term matching and handles precise queries effectively.
- Struggles with semantic variations, synonyms, and contextual understanding.
- Provides high precision but may sacrifice recall.
Semantic Search Capabilities and Challenges:
- Captures conceptual relationships and contextual meaning.
- May retrieve contextually related but topically irrelevant content.
- Offers superior recall but can compromise precision.
Mathematical Foundation of Hybrid Scoring
The hybrid retrieval system employs a weighted combination formula:
Final Score = β × Lexical Score + (1 – β) × Semantic Score
Where β represents the tunable weighting parameter that balances lexical precision against semantic understanding.
Example Code of Hybrid Search using Qdrant Vectorstore:
class QdrantStore:
def __init__(
self,
collection_name,
url=QDRANT_URL,
api_key = QDRANTAPIKEY,
delete=False
):
self.collection_name = collection_name
self.embeddings = OpenAIEmbeddingsWrapper()
self.client = QdrantClient(url=url,api_key = api_key)
# Check existing collections
existing_collections = [c.name for c in self.client.get_collections().collections]
# Delete collection if flagged
if self.collection_name in existing_collections and delete:
self.client.delete_collection(collection_name=self.collection_name)
# Create collection if missing
if self.collection_name not in existing_collections:
self.client.create_collection(
collection_name=self.collection_name,
vectors_config={
"text-dense": models.VectorParams(
size=384,
distance=models.Distance.COSINE,
hnsw_config=models.HnswConfigDiff(
m=16,
ef_construct=100,
on_disk=False
),
on_disk=False
)
},
sparse_vectors_config={
"text-sparse": models.SparseVectorParams(
index=models.SparseIndexParams(on_disk=True)
)
},
optimizers_config=models.OptimizersConfigDiff(
default_segment_number=2
),
on_disk_payload=True
)
# LangChain wrapper
self.vectorstore = QdrantVectorStore(
client=self.client,
collection_name=self.collection_name,
embedding=self.embeddings,
vector_name="text-dense"
)
def hybrid_search(self, query_vector, sparse_vector, total_results: int = 5, filters=None):
"""
Perform hybrid (dense + sparse) search with RRF fusion.
"""
prefetch_n = total_results
prefetch = [
models.Prefetch(
query=models.SparseVector(**sparse_vector),
using="text-sparse",
limit=prefetch_n,
filter=filters,
),
models.Prefetch(
query=query_vector,
using="text-dense",
limit=prefetch_n,
filter=filters,
),
]
results = self.client.query_points(
collection_name=self.collection_name,
prefetch=prefetch,
query=models.FusionQuery(fusion=models.Fusion.RRF),
with_payload=True,
limit=total_results,
search_params=models.SearchParams(hnsw_ef=128),
)
print(f"Results: {results}")
return results
Conclusion and Future Directions
Hybrid search is an established solution for today’s information retrieval problems. Intelligent fusion of lexical accuracy with semantic comprehension provides these systems with better performance across wide-ranging query types and document sets.
Trendsetters:
- Neural Information Retrieval: Learned sparse representation integration
- Adaptive Weighting: Query analysis-based dynamic adjustment of alpha
- Multi-modal Extensions: Integration of visual and audio media
The ongoing development of hybrid search practices holds out the potential for even more advanced methods of information seeking, making them integral parts of future AI systems.
Ready to level up your customer experience and business performance? Introduce AI agents in your business operations to deliver results that not only enhance customer experience but also provide advanced analytics for streamlining business processes. Partner with Xcelore, AI agent development company for AI agents that leverage hybrid search algorithms.
FAQs
-
1. What is hybrid search AI?
Hybrid search is a technique that combines keyword-based search (exact matches of words) with semantic search (understanding the meaning behind words). By combining both, AI systems can provide precise and contextually relevant results.
-
2. What is the difference between hybrid search and semantic search?
Hybrid search involves providing results based on both keyword match and context based, while semantic search focuses on understanding meaning and context rather than just exact words. Semantic search is a part of hybrid search.
-
3. What are the advantages of hybrid search engines?
The hybrid search covers exact keyword matching and contextually related terms, which enables search engines to provide accurate results. For example, if a user searches for a vague query, the hybrid search will show results based on both keyword match and understanding the intent.
