fix(kebab-store-vector): close P7-3 vector orphan caveat — delete_by_chunk_ids

P7-3 의 storage UNIQUE bug fix 가 SQLite 측 (documents → blocks /
chunks / embedding_records) 만 sweep 했음. LanceDB 의 vector 는 별도
store 라 옛 chunk_id 를 가진 row 가 디스크에 잔존. 검색에는 영향 없지만
디스크는 무한 누적. HOTFIXES `2026-05-02 P7-3` caveat 의 "P+ task" 약속을
같은 후속 PR 안에서 닫음.

변경:
- `VectorStore::delete_by_chunk_ids(&[ChunkId])` trait method 추가 (default
  no-op 제공 — 테스트 fake / 기존 impl 이 그대로 컴파일).
- `LanceVectorStore::delete_by_chunk_ids` 가 connection 의 모든
  `chunk_embeddings_*` 테이블을 순회 + `Table::delete("chunk_id IN (...)")`
  를 batch=200 단위로 실행. 다중 모델 워크스페이스 (마이그레이션 중간 등)
  에서도 안전.
- `SqliteStore::stale_chunk_ids_at(workspace_path, new_asset_id)` 가
  read-only SELECT 로 옛 chunk_id 들 반환. CASCADE 가 흐르기 *전* 에
  caller 가 호출.
- `kebab-app::purge_vector_orphans_for_workspace_path` 가 위 두 단계를
  orchestrate. 세 ingest path (markdown / image / pdf) 의
  `put_asset_with_bytes` 호출 직전에 한 줄로 호출.

Smoke 검증 (release binary, fastembed enabled):
- whitepaper.pdf 첫 ingest → chunk_ids = {f616…, 4e0f…}, vector store 에
  그 두 ID 의 row 존재.
- byte 변경 후 re-ingest → 새 doc_id (3741…) + 새 chunk_ids
  (ed0c…, e13c…). vector search "REWRITTEN chapter two" → 새 chunk_ids 만
  hit. 옛 query "Edited page two body" 시도해도 옛 chunk_ids 는 vector
  store 에 더 이상 없음 (의미적으로 가장 가까운 새 chunks 가 hit).

HOTFIXES `2026-05-02 P7-3` 의 \"vector store cleanup\" 항목이 \"deferred\" →
\"closed by follow-up PR\" 로 갱신. SMOKE.md 의 알려진 동작 (\"옛 vector
잔존\") 도 \"두 store 정합\" 으로 갱신.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-02 12:32:29 +00:00
parent 91ae624a92
commit 0c8821f857
6 changed files with 183 additions and 9 deletions

View File

@@ -615,6 +615,7 @@ fn ingest_one_asset(
// (per-document tx semantics per design §5.8); composing them is
// the kb-app job. A failure mid-way leaves the DB in a state the
// next ingest run can re-converge (UPSERT + DELETE-then-INSERT).
purge_vector_orphans_for_workspace_path(app, asset, vector_store)?;
app.sqlite
.put_asset_with_bytes(asset, &bytes)
.context("DocumentStore::put_asset_with_bytes")?;
@@ -841,6 +842,7 @@ fn ingest_one_image_asset(
.context("kb-chunk::MdHeadingV1Chunker::chunk (image)")?;
// 5. Persist + embed — identical sequence to markdown.
purge_vector_orphans_for_workspace_path(app, asset, vector_store)?;
app.sqlite
.put_asset_with_bytes(asset, &bytes)
.context("DocumentStore::put_asset_with_bytes (image)")?;
@@ -949,6 +951,46 @@ fn record_image_analysis_failure(
warning_notes.push(note);
}
/// HOTFIXES 2026-05-02 P7-3 follow-up: when a tracked file's bytes
/// change, `purge_orphan_at_workspace_path` (in `kebab-store-sqlite`)
/// sweeps the SQLite chain (documents → blocks / chunks / embedding_records)
/// but the LanceDB rows keyed on the now-deleted `chunk_id`s live in a
/// separate store. This helper fetches the stale `chunk_id`s from
/// SQLite **before** they get cascade-deleted, then deletes the
/// matching vectors from every Lance table.
///
/// Called by every per-medium ingest helper at the same point —
/// immediately before `put_asset_with_bytes` runs, so the SELECT
/// still sees the old chunk_ids and the DELETE happens before the
/// new rows land. Empty workspace_path / no embedder → no-op.
fn purge_vector_orphans_for_workspace_path(
app: &App,
asset: &RawAsset,
vector_store: Option<&Arc<kebab_store_vector::LanceVectorStore>>,
) -> anyhow::Result<()> {
let Some(vec_store) = vector_store else {
return Ok(());
};
let stale = app
.sqlite
.stale_chunk_ids_at(&asset.workspace_path.0, &asset.asset_id.0)
.context("SqliteStore::stale_chunk_ids_at")?;
if stale.is_empty() {
return Ok(());
}
use kebab_core::VectorStore as _;
vec_store
.delete_by_chunk_ids(&stale)
.context("VectorStore::delete_by_chunk_ids (orphan vector cleanup)")?;
tracing::debug!(
target: "kebab-app",
path = %asset.workspace_path.0,
count = stale.len(),
"purged orphan vectors for edited asset"
);
Ok(())
}
/// P7-3: process one `MediaType::Pdf` asset end-to-end.
///
/// - Reads bytes from disk.
@@ -1024,6 +1066,7 @@ fn ingest_one_pdf_asset(
.chunk(&canonical, chunk_policy)
.context("kb-chunk::PdfPageV1Chunker::chunk")?;
purge_vector_orphans_for_workspace_path(app, asset, vector_store)?;
app.sqlite
.put_asset_with_bytes(asset, &bytes)
.context("DocumentStore::put_asset_with_bytes (pdf)")?;