altair823
3cd5117a7e
First VectorStore implementation. Per-model Lance tables under config.storage.vector_dir, two-phase upsert (SQLite-pending → Lance MergeInsert → SQLite-committed) with crash-safe retry, search via cosine distance with the spec's score-shift (preserves negative similarity ranking signal that clamping would crush). V003 migration: - Adds status (CHECK constraint pending|committed|tombstone, default pending) and vector_committed columns to embedding_records. - BEFORE DELETE trigger on chunks flips dependent rows to tombstone. Currently overshadowed by V001's ON DELETE CASCADE FK; trigger UPDATE runs first then row vanishes via CASCADE. Spec-faithful tombstone preservation requires recreating embedding_records to drop the CASCADE — deferred to a P+ migration since no production rows exist yet (P3-3 is the first writer). V003 SQL comment explains. LanceVectorStore: - ensure_table is idempotent: opens existing or creates with the Arrow schema (chunk_id, doc_id, embedding FixedSizeList<Float32, dim>, model_id, embedding_version, text, heading_path, created_at). - IndexId computed via id_for_index with collection="chunk_embeddings", index_kind="flat", params_hash = blake3(descriptor JSON). Schema bumps automatically rotate the IndexId. - upsert: phase-1 INSERT OR REPLACE INTO embedding_records (status= 'pending') in a single SQLite tx; phase-2 Lance MergeInsert keyed on chunk_id (idempotent re-run); phase-3 UPDATE status='committed', vector_committed=1. If phase-2 fails the rows stay 'pending' and the next upsert call retries idempotently. - search joins embedding_records WHERE status='committed' so partial- write rows never surface. Cosine distance from Lance ∈ [0, 2] → similarity = 1 - distance ∈ [-1, 1] → score = (similarity + 1)/2 ∈ [0, 1]. NaN coerced to 0 with tracing::warn. Filter by SearchFilters via SqliteStore::filter_chunks (added in this commit). - Sync trait + async LanceDB bridged by an embedded current-thread tokio runtime. Doc-comment on the struct flags the "do NOT call from inside another tokio runtime" panic (block_on cannot nest). kb-app's job scheduler is sync today. kb-store-sqlite additions: - pub fn put_embedding_records_pending(&[EmbeddingRecordRow]) — phase-1 INSERT OR REPLACE (status='pending', vector_committed=0). - pub fn mark_embedding_records_committed(&[EmbeddingId]) — phase-3 single UPDATE … WHERE embedding_id IN (?, ?, …) via params_from_iter, guarded by WHERE status='pending' so tombstones don't get clobbered. - pub fn filter_chunks(&[ChunkId], &SearchFilters) → Vec<ChunkId> consolidates the JOIN against documents/document_tags/ embedding_records + path_glob via globset. Lets kb-store-vector honor SearchFilters without depending on rusqlite or globset directly. (kb-search's filter logic is structurally different — interleaved with the FTS5 SELECT — so it stays as-is for now; consolidation is a P+ refactor.) - 4 new unit tests cover the phase-1 round-trip, empty batch, replay reset of pending rows, and the WHERE-status-pending guard. Tests: - 9 lib unit tests in kb-store-vector covering paths/sanitization, arrow_batch dim validation + descriptor hash, bm25-style cosine score shift math. - 4 new kb-store-sqlite unit tests on filter_chunks (committed-only, tags/lang/trust/path_glob, order preservation, empty input). - 4 new kb-store-sqlite unit tests on the embedding_records helpers. - 8 integration tests in upsert_search.rs and 1 snapshot test marked #[ignore = "requires AVX-capable hardware (LanceDB)"]. They invoke require_avx_or_panic() at the top of each body so a missing-AVX --ignored run fails loudly instead of silently passing. This dev host (qemu64 model) lacks AVX so these were NOT exercised end-to- end here — first CI lane on AVX hardware will validate them. - Snapshot fixture tests/fixtures/vector/run-1.json is a placeholder with an _comment marker. Snapshot test panics until the placeholder is replaced via KB_UPDATE_SNAPSHOTS=1 on AVX hardware. - Workspace 241 passed, 19 ignored, 0 failed; cargo clippy --workspace --all-targets -- -D warnings clean. Allowed deps respected (kb-core, kb-config, kb-store-sqlite, lancedb, arrow + arrow-array + arrow-schema, serde, serde_json, tracing, thiserror) plus forced waivers — anyhow (trait return type), tokio + futures (LanceDB async-only API), blake3 (params_hash). rusqlite and globset are NOT direct deps of kb-store-vector — confirmed via cargo metadata --no-deps. rusqlite stays in [dev-dependencies] for the test fixture seeder only. Out of scope: IVF/PQ index tuning (P+), image vectors (P6), kb-app embed_index orchestration (P3-4 facade). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
47 lines
2.4 KiB
SQL
47 lines
2.4 KiB
SQL
-- V003__embedding_status.sql — additive embedding lifecycle markers (§5.6).
|
|
--
|
|
-- P3-3 introduces a two-phase write to `embedding_records` paired with
|
|
-- a Lance MergeInsert. Phase 1 inserts the row at `status='pending'`;
|
|
-- phase 2 issues the Lance write; phase 3 flips the row to
|
|
-- `status='committed'`. `search` joins back through this table with
|
|
-- `WHERE status='committed'` so partial-write Lance rows never surface
|
|
-- to callers, and a crashed phase 2 retry simply re-runs against the
|
|
-- still-pending row (Lance MergeInsert dedupes on `chunk_id`).
|
|
--
|
|
-- The third state, `tombstone`, is reserved for the deletion pipeline:
|
|
-- when a chunk row goes away, the matching Lance row should also be
|
|
-- garbage-collected, but the GC scheduler is out of P3-3 scope. The
|
|
-- BEFORE DELETE trigger below stages the marker so a future GC has a
|
|
-- well-defined claim; see the comment block on the trigger for why
|
|
-- it currently coexists with V001's `ON DELETE CASCADE` FK rather than
|
|
-- replacing it.
|
|
|
|
ALTER TABLE embedding_records ADD COLUMN status TEXT NOT NULL DEFAULT 'pending'
|
|
CHECK (status IN ('pending','committed','tombstone'));
|
|
|
|
ALTER TABLE embedding_records ADD COLUMN vector_committed INTEGER NOT NULL DEFAULT 0;
|
|
|
|
CREATE INDEX idx_embed_status ON embedding_records(status);
|
|
|
|
-- Tombstone trigger.
|
|
--
|
|
-- Intent: when a `chunks` row is about to be deleted, mark its
|
|
-- dependent `embedding_records` rows as `status='tombstone'` so a later
|
|
-- GC pass can drop the matching Lance rows in lockstep.
|
|
--
|
|
-- Caveat (carried into a future migration): V001 declared the FK as
|
|
-- `chunk_id REFERENCES chunks(chunk_id) ON DELETE CASCADE`. SQLite's
|
|
-- documented order is "BEFORE-DELETE trigger fires first, then CASCADE
|
|
-- runs", so this UPDATE will land a `tombstone` value that is
|
|
-- immediately followed by the CASCADE removing the row. The trigger is
|
|
-- therefore best-effort under the current FK; the only path that
|
|
-- actually preserves the tombstone is to drop the CASCADE (table
|
|
-- recreation, since SQLite has no DROP CONSTRAINT) — that is queued
|
|
-- for a P+ migration once the GC scheduler exists and we have actual
|
|
-- production rows to migrate. Keeping the trigger here documents the
|
|
-- design intent and gives the deletion-pipeline observer a stable hook
|
|
-- to wire into.
|
|
CREATE TRIGGER chunks_bd_tombstone_embeddings BEFORE DELETE ON chunks BEGIN
|
|
UPDATE embedding_records SET status='tombstone' WHERE chunk_id = old.chunk_id;
|
|
END;
|