kebab/tasks/p3/p3-4-hybrid-fusion.md

phase: P3 component: kebab-search (hybrid) task_id: p3-4 title: "Hybrid Retriever (RRF) over lexical + vector" status: completed depends_on: [p2-2, p3-3] unblocks: [p4-3] contract_source: ../../docs/superpowers/specs/2026-04-27-kebab-final-form-design.md contract_sections: [§3.7 RetrievalDetail, §0 Q3, §1.6 search --explain, §6.4 [search] rrf settings]

p3-4 — Hybrid Retriever (RRF)

Goal

Compose LexicalRetriever (p2-2) and a vector retriever wrapper around LanceVectorStore (p3-3) into a single Retriever that dispatches by SearchMode. For Hybrid, fuse via Reciprocal Rank Fusion (RRF) and populate full RetrievalDetail per SearchHit.

Why now / why this size

Single mediator. Keeps the lexical and vector retrievers focused; only this task knows how to fuse. RAG (p4-3) consumes hybrid output without caring about the underlying retrievers.

Allowed dependencies

• kebab-core
• kebab-config
• kebab-store-sqlite (for LexicalRetriever)
• kebab-store-vector (for LanceVectorStore)
• kebab-embed (trait only — for query embedding via Embedder)
• tracing
• thiserror

Forbidden dependencies

• kebab-source-fs, kebab-parse-md, kebab-normalize, kebab-chunk, kebab-llm*, kebab-rag, kebab-tui, kebab-desktop. (kebab-embed-local is a runtime-injected dyn Embedder; this crate must not depend on the concrete adapter directly.)

Inputs

input	type	source
LexicalRetriever	trait object	constructed elsewhere
LanceVectorStore	trait object	constructed elsewhere
Box<dyn Embedder>	for query embedding	runtime-injected
kebab-config::Config.search	default_k, hybrid_fusion, rrf_k	runtime
SearchQuery	kebab_core::SearchQuery	kebab-app::search

Outputs

output	type	downstream
Vec<SearchHit> (with full RetrievalDetail)	kebab_core::SearchHit	kebab-cli printer, kebab-rag packer

Public surface (signatures only — no new types)

pub struct HybridRetriever {
    lexical: std::sync::Arc<dyn kebab_core::Retriever>,
    vector:  std::sync::Arc<dyn kebab_core::Retriever>,   // wrapper over LanceVectorStore + Embedder
    fusion:  FusionPolicy,
    k:       usize,
}

pub enum FusionPolicy { Rrf { k_rrf: u32 } }

impl HybridRetriever {
    pub fn new(
        config: &kebab_config::Config,
        lexical: std::sync::Arc<dyn kebab_core::Retriever>,
        vector:  std::sync::Arc<dyn kebab_core::Retriever>,
    ) -> Self;
}

impl kebab_core::Retriever for HybridRetriever {
    fn search(&self, query: &kebab_core::SearchQuery) -> anyhow::Result<Vec<kebab_core::SearchHit>>;
    fn index_version(&self) -> kebab_core::IndexVersion;
}

/// Wrapper that turns a VectorStore + Embedder into a Retriever.
pub struct VectorRetriever {
    store:   std::sync::Arc<dyn kebab_core::VectorStore>,
    embed:   std::sync::Arc<dyn kebab_core::Embedder>,
    /* heading_path/snippet enrichment hits SQLite via kebab-store-sqlite read accessor */
}
impl VectorRetriever {
    pub fn new(store: std::sync::Arc<dyn kebab_core::VectorStore>, embed: std::sync::Arc<dyn kebab_core::Embedder>, sqlite: std::sync::Arc<kebab_store_sqlite::SqliteStore>) -> Self;
}
impl kebab_core::Retriever for VectorRetriever { /* per §7.2 */ }

Behavior contract

• SearchMode::Lexical dispatches solely to lexical. RetrievalDetail.method = Lexical, vector_* fields are None.
• SearchMode::Vector dispatches solely to vector. RetrievalDetail.method = Vector, lexical_* fields are None.
• SearchMode::Hybrid:
  • run lexical.search(query) and vector.search(query) in sequence (fan-out is fine; not required).
  • fuse with RRF: raw(c) = Σ_{m ∈ {lex, vec}} 1 / (k_rrf + rank_m(c)) where k_rrf from config (default 60). rank_m is 1-based; chunks not appearing in retriever m contribute 0.
  • normalize fusion_score to [0, 1] (post-merge fix, 2026-05): divide by num_retrievers / (k_rrf + 1) so the top-1-everywhere case maps to 1.0 and single-retriever chunks cap around 0.5. Without this, raw RRF tops out at ≈ 0.033 and is incomparable with the [0, 1] lexical / vector fusion_score (and incompatible with the config.rag.score_gate default 0.05 — every hybrid query refused). RRF's rank ordering is preserved (we divide every score by the same positive constant). See HOTFIXES.md.
  • sort by fused score DESC, take top query.k.
  • populate every SearchHit.retrieval: method = Hybrid, lexical_score / lexical_rank / vector_score / vector_rank from each retriever's hit (or None if absent), fusion_score = normalized fused score.
  • if a chunk appears in only one retriever, its RetrievalDetail still gets populated with Some(...) from that side and None for the other.
  • tie-break by lexical_rank ascending, then chunk_id ascending (deterministic).
• VectorRetriever:
  • embeds the query via embed.embed(&[EmbeddingInput { text: query.text, kind: Query }]).
  • calls VectorStore::search(query_vec, query.k * 2, query.filters) (over-fetch for filter losses), trims to k.
  • hydrates doc_path / heading_path / section_label / chunker_version / embedding_model from SQLite by joining on chunk_id.
  • builds Citation from chunk's first source span (same logic as p2-2).
• index_version() returns the lexical index version when in pure lexical mode, else the vector index version, else "hybrid:<lex_iv>+<vec_iv>".

Storage / wire effects

• Reads only. No mutations.
• Output JSON conforms to search_hit.v1.

Test plan

kind	description	fixture / data
unit	pure lexical mode delegates 1:1 to lexical.search	mock retrievers
unit	pure vector mode delegates 1:1 to vector.search	mock retrievers
unit	hybrid: chunk only in lexical receives vector_*: None, but still has a fused score	mock retrievers
unit	RRF formula matches expected with k_rrf=60	inline math test
unit	tie-break deterministic (same fused score → stable order)	inline
unit	hybrid recall ≥ max(lexical recall, vector recall) on a tiny corpus where each mode finds disjoint hits	tmp DB + Lance + MockEmbedder
determinism	identical query twice → byte-identical Vec<SearchHit>	tmp DB
snapshot	hybrid output JSON stable	fixtures/search/hybrid/run-1.json

All tests under cargo test -p kebab-search hybrid.

Definition of Done

• cargo check -p kebab-search passes
• cargo test -p kebab-search hybrid passes
• No imports outside Allowed dependencies
• PR links design §3.7, §6.4 search, §0 Q3

Out of scope

• Reranker (P+).
• Multimodal retrieval (image/audio) — P6+.
• Score calibration across modes (RRF makes scores rank-comparable; absolute calibration is P+).

Risks / notes

• Mismatched index_version between lexical and vector should be flagged at construction so users notice stale indexes.
• Over-fetching at the vector retriever (2 * k) is conservative; if filters reject everything, the hybrid k may shrink. Document this in CLI --explain.
• RRF is rank-based, so absolute lexical bm25 normalization (p2-2) doesn't affect fused order; still keep normalization for --explain readability.