altair823
3cd5117a7e
First VectorStore implementation. Per-model Lance tables under config.storage.vector_dir, two-phase upsert (SQLite-pending → Lance MergeInsert → SQLite-committed) with crash-safe retry, search via cosine distance with the spec's score-shift (preserves negative similarity ranking signal that clamping would crush). V003 migration: - Adds status (CHECK constraint pending|committed|tombstone, default pending) and vector_committed columns to embedding_records. - BEFORE DELETE trigger on chunks flips dependent rows to tombstone. Currently overshadowed by V001's ON DELETE CASCADE FK; trigger UPDATE runs first then row vanishes via CASCADE. Spec-faithful tombstone preservation requires recreating embedding_records to drop the CASCADE — deferred to a P+ migration since no production rows exist yet (P3-3 is the first writer). V003 SQL comment explains. LanceVectorStore: - ensure_table is idempotent: opens existing or creates with the Arrow schema (chunk_id, doc_id, embedding FixedSizeList<Float32, dim>, model_id, embedding_version, text, heading_path, created_at). - IndexId computed via id_for_index with collection="chunk_embeddings", index_kind="flat", params_hash = blake3(descriptor JSON). Schema bumps automatically rotate the IndexId. - upsert: phase-1 INSERT OR REPLACE INTO embedding_records (status= 'pending') in a single SQLite tx; phase-2 Lance MergeInsert keyed on chunk_id (idempotent re-run); phase-3 UPDATE status='committed', vector_committed=1. If phase-2 fails the rows stay 'pending' and the next upsert call retries idempotently. - search joins embedding_records WHERE status='committed' so partial- write rows never surface. Cosine distance from Lance ∈ [0, 2] → similarity = 1 - distance ∈ [-1, 1] → score = (similarity + 1)/2 ∈ [0, 1]. NaN coerced to 0 with tracing::warn. Filter by SearchFilters via SqliteStore::filter_chunks (added in this commit). - Sync trait + async LanceDB bridged by an embedded current-thread tokio runtime. Doc-comment on the struct flags the "do NOT call from inside another tokio runtime" panic (block_on cannot nest). kb-app's job scheduler is sync today. kb-store-sqlite additions: - pub fn put_embedding_records_pending(&[EmbeddingRecordRow]) — phase-1 INSERT OR REPLACE (status='pending', vector_committed=0). - pub fn mark_embedding_records_committed(&[EmbeddingId]) — phase-3 single UPDATE … WHERE embedding_id IN (?, ?, …) via params_from_iter, guarded by WHERE status='pending' so tombstones don't get clobbered. - pub fn filter_chunks(&[ChunkId], &SearchFilters) → Vec<ChunkId> consolidates the JOIN against documents/document_tags/ embedding_records + path_glob via globset. Lets kb-store-vector honor SearchFilters without depending on rusqlite or globset directly. (kb-search's filter logic is structurally different — interleaved with the FTS5 SELECT — so it stays as-is for now; consolidation is a P+ refactor.) - 4 new unit tests cover the phase-1 round-trip, empty batch, replay reset of pending rows, and the WHERE-status-pending guard. Tests: - 9 lib unit tests in kb-store-vector covering paths/sanitization, arrow_batch dim validation + descriptor hash, bm25-style cosine score shift math. - 4 new kb-store-sqlite unit tests on filter_chunks (committed-only, tags/lang/trust/path_glob, order preservation, empty input). - 4 new kb-store-sqlite unit tests on the embedding_records helpers. - 8 integration tests in upsert_search.rs and 1 snapshot test marked #[ignore = "requires AVX-capable hardware (LanceDB)"]. They invoke require_avx_or_panic() at the top of each body so a missing-AVX --ignored run fails loudly instead of silently passing. This dev host (qemu64 model) lacks AVX so these were NOT exercised end-to- end here — first CI lane on AVX hardware will validate them. - Snapshot fixture tests/fixtures/vector/run-1.json is a placeholder with an _comment marker. Snapshot test panics until the placeholder is replaced via KB_UPDATE_SNAPSHOTS=1 on AVX hardware. - Workspace 241 passed, 19 ignored, 0 failed; cargo clippy --workspace --all-targets -- -D warnings clean. Allowed deps respected (kb-core, kb-config, kb-store-sqlite, lancedb, arrow + arrow-array + arrow-schema, serde, serde_json, tracing, thiserror) plus forced waivers — anyhow (trait return type), tokio + futures (LanceDB async-only API), blake3 (params_hash). rusqlite and globset are NOT direct deps of kb-store-vector — confirmed via cargo metadata --no-deps. rusqlite stays in [dev-dependencies] for the test fixture seeder only. Out of scope: IVF/PQ index tuning (P+), image vectors (P6), kb-app embed_index orchestration (P3-4 facade). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
56 lines
2.5 KiB
TOML
56 lines
2.5 KiB
TOML
[package]
|
|
name = "kb-store-vector"
|
|
version = { workspace = true }
|
|
edition = { workspace = true }
|
|
rust-version = { workspace = true }
|
|
license = { workspace = true }
|
|
repository = { workspace = true }
|
|
description = "LanceDB-backed VectorStore for kb (§5.6 embedding_records, §6.3 lancedb tables, §7.2 VectorStore)"
|
|
|
|
[dependencies]
|
|
kb-core = { path = "../kb-core" }
|
|
kb-config = { path = "../kb-config" }
|
|
# kb-store-sqlite is allowed for the embedding_records writers only
|
|
# (P3-3 spec: "Allowed dep `kb-store-sqlite` for writing/reading rows in
|
|
# embedding_records"). The Two-phase upsert flow uses
|
|
# `put_embedding_records_pending` + `mark_embedding_records_committed`.
|
|
kb-store-sqlite = { path = "../kb-store-sqlite" }
|
|
|
|
# LanceDB embedded vector store. `default-features=false` opts out of
|
|
# the cloud object-store integrations (aws / gcs / azure / dynamodb /
|
|
# oss); kb is always-local for v1, so dragging in those SDKs would just
|
|
# inflate the build.
|
|
lancedb = { workspace = true }
|
|
arrow = { workspace = true }
|
|
arrow-array = { workspace = true }
|
|
arrow-schema = { workspace = true }
|
|
# Embedded async runtime. The VectorStore trait is sync (§7.2) but
|
|
# LanceDB's Rust API is async-only; we own a current-thread
|
|
# tokio::Runtime and `block_on` per trait method. current-thread saves
|
|
# the two worker threads a multi-thread runtime would spawn — kb-app
|
|
# already serializes vector ops behind its own job scheduler so the
|
|
# extra parallelism wouldn't be exploited.
|
|
tokio = { workspace = true }
|
|
# `try_collect` for streaming Lance query results into a Vec<RecordBatch>.
|
|
futures = { workspace = true }
|
|
|
|
serde = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
tracing = { workspace = true }
|
|
thiserror = { workspace = true }
|
|
anyhow = { workspace = true }
|
|
blake3 = { workspace = true }
|
|
time = { workspace = true }
|
|
|
|
[dev-dependencies]
|
|
tempfile = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
# Integration tests seed `documents` / `chunks` fixtures by raw SQL
|
|
# (no kb-parse-md / kb-normalize / kb-chunk dep) so they can construct
|
|
# adversarial filter / dim-mismatch states. rusqlite is a `[dev-]`
|
|
# dep only — the runtime crate uses kb-store-sqlite's typed surface
|
|
# (`filter_chunks`, `put_embedding_records_pending`, …) and does not
|
|
# touch rusqlite directly (P3-3 spec: kb-store-vector must not list
|
|
# rusqlite/globset as direct deps).
|
|
rusqlite = { workspace = true }
|