PR-2 of fb-41 multi-hop RAG. Decompose + retrieve + synthesize 3-stage
pipeline가 `opts.multi_hop=true` 일 때 dispatch. Dynamic decide loop
는 PR-3.
- `AskOpts.multi_hop: bool` 필드 추가 + `impl Default for AskOpts`
도입 (HOTFIXES 2026-05-07 의 known limitation 해소). 9 explicit
init site 모두 `multi_hop: false` 추가 — Default 도입으로 향후
`..Default::default()` 점진 migrate 가능.
- `RagPipeline::ask` 의 entry 에 dispatcher 한 줄
(`if opts.multi_hop { return self.ask_multi_hop(...) }`).
- `RagPipeline::ask_multi_hop` 신규 method. 1) decompose LLM call
→ JSON array of strings parse, 2) 각 sub-query 로 retrieve +
chunk_id dedup pool, 3) score gate / no-chunks 가드, 4)
pack_context (single-pass 와 helper 공유), 5) synthesize LLM
call w/ MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT, 6) citation extract
+ Answer build. `prompt_template_version` = "rag-multi-hop-v1"
로 stamp — eval `compare` 가 single-pass vs multi-hop 분리.
- Prompt const 신규: MULTI_HOP_DECOMPOSE_SYSTEM_PROMPT +
MULTI_HOP_DECOMPOSE_USER_TEMPLATE + MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT
+ PROMPT_TEMPLATE_VERSION_MULTI_HOP + MULTI_HOP_MAX_SUB_QUERIES_DEFAULT.
- `kebab_core::RefusalReason::MultiHopDecomposeFailed` variant 신규.
Cascade: kebab-store-sqlite `refusal_reason_label` + kebab-tui `ask
refusal render` exhaustive match 갱신.
- `parse_decompose_response` + `strip_markdown_json_fence` helper —
markdown code fence (```json / ```) strip + JSON array of strings
parse + trim + drop empty + cap at MULTI_HOP_MAX_SUB_QUERIES_DEFAULT.
None 반환 시 caller 가 `MultiHopDecomposeFailed` refusal.
Tests (55 passing total, 8 신규):
- 6 unit (parse_decompose_response 의 bare array / fence variants /
garbage / cap / trim 회귀 핀).
- 2 integration: `ask_multi_hop_dispatches_and_decompose_garbage_refuses`
(decompose garbage → MultiHopDecomposeFailed + 정확히 1 LLM call) +
`ask_with_multi_hop_false_keeps_single_pass_path` (회귀 핀, 기존
caller 자동 backwards-compat).
Happy-path multi-hop (decompose 성공 → synthesize) 의 integration
test 는 ScriptedLm helper 가 PR-3 의 decide loop 와 함께 도입될
때 같이 추가. 현 `MockLanguageModel` 는 canned single response 라
2-LLM-call sequence 핀 불가.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
assets.workspace_path is INTENTIONALLY 'last-registered path' for twin
files (identical content at different paths share one asset row PK'd by
blake3 content hash). PR #146 made try_skip_unchanged document-centric;
PR #149 made reset --orphans-only document-centric; this PR removes the
last caller of get_asset_by_workspace_path (fetch.rs:193 in fetch_span,
which used it to reject PDF/audio media — for twins this could read the
wrong asset's media_type and pick the wrong branch).
Replaced with the natural 2-step lookup: get_document_by_workspace_path
(PR #146) → doc.source_asset_id → get_asset (NEW trait method, asset_id
is PRIMARY KEY so flip-flop-immune by construction).
Then removed get_asset_by_workspace_path trait method + SqliteStore impl
— 0 callers after the refactor.
UPSERT doc-comment refreshed in store.rs to make the 'last-registered'
semantics explicit so future readers don't try to 'fix' the flip-flop.
Dogfood follow-up (PR #142 1B + multi-root corpus).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files deleted from disk (rm a.md) were leaving stale documents + chunks +
embeddings in the store, surfacing as ghost citations in search/ask.
Existing purge_orphan_at_workspace_path only handled content-changed
stale (WHERE workspace_path=? AND asset_id != ?) — file deletion has no
new asset_id.
Fix: post-walker-scan sweep. Compute (stored_paths - scanned_paths),
for each candidate check filesystem existence — only purge when the
file is TRULY missing. Scope-narrowing case (file on disk but outside
include glob) is explicitly NOT purged to protect users from accidental
data loss via config edits.
Adds:
- DocumentStore::all_workspace_paths trait method + SqliteStore impl
- purge_deleted_workspace_path in store-sqlite (returns chunk_ids for
vector delete; deletes doc CASCADE + asset row + copied storage file)
- sweep_deleted_files in kebab-app::ingest path; called once per ingest
before the per-asset loop
- IngestReport.purged_deleted_files counter (additive, serde default)
- CLI ingest summary mentions purge count when > 0
- 2 integration tests: file_deletion_auto_purge + include_scope_narrowing_does_NOT_purge
dogfood discovery (PR #142 1B + multi-root: kebab-docs + httpx + zod
+ lodash). Per user decision: only filesystem deletion auto-purges;
scope narrowing requires explicit kebab reset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Identical-content files at different workspace paths share one assets row
(assets.asset_id = blake3 content hash, PRIMARY KEY). The UPSERT
`ON CONFLICT(asset_id) DO UPDATE SET workspace_path = excluded` made
twin files overwrite each other's workspace_path on every ingest, so
`get_asset_by_workspace_path(path1)` returned the OTHER twin's row (or
None) — break idempotent unchanged-detection for both files.
Fix: switch try_skip_unchanged to document-centric lookup. `documents.
workspace_path` is already UNIQUE (V001) and `id_for_doc(path, ...)`
includes path, so each twin has its own stable document row. Compare
`doc.source_asset_id` with the new asset's checksum instead of going
through the assets table.
Dogfood (multi-root: kebab-docs + httpx + zod + lodash) showed 27 of
726 docs marked Updated on every idempotent re-ingest — all 27 are
twin-file victims (empty `__init__.py` ×3, AGENTS.md ↔ CLAUDE.md
same content, duplicate logo PDFs/JPGs).
After: re-ingest reports 0 new / 0 updated / 726 unchanged.
No schema migration needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TDD: red → green cycle confirmed. New `Code(String)` variant serializes
as `{"code":"rust"}` via serde `rename_all = "lowercase"`. All exhaustive
`match` sites updated (`media_label`, `ingest_one_asset` catch-all →
explicit or-pattern). Design §3.5 enum listing synced. Also fix
`/target` symlink gitignore pattern so integration-test binary lookup
via workspace-relative path works with CARGO_TARGET_DIR redirect.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four optional, serde-skipped-when-None fields added to `Metadata` for
code ingest context. All 11 downstream construction sites patched with
`repo: None, git_branch: None, git_commit: None, code_lang: None`.
Full workspace check (`--tests`) and per-crate test suite pass clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds five new u32 counters (skipped_gitignore, skipped_kebabignore,
skipped_builtin_blacklist, skipped_generated, skipped_size_exceeded)
and a SkipExamples struct (≤5 sample paths per category) to
IngestReport. All new fields are #[serde(default)] for backward-compat
deserialization. Downstream literal construction sites patched with
zeros/empty; snapshot re-baked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire two new optional fields onto SearchHit (skip_serializing_if = None)
and two Vec<String> filter fields onto SearchFilters (serde default).
Add RetrievalDetail::Default impl (manual, uses SearchMode::Hybrid as
sentinel). Patch all downstream SearchHit / SearchFilters literal
constructors with repo: None / code_lang: None / vec![] as appropriate.
Also covers Citation::Code arm in kebab-eval metrics match.
- doc TraceFusionInput.fusion_score semantics (single-mode vs hybrid)
- comment why total_ms vs stage sum can drift (millis truncation)
- TODO marker on TUI trace popup filter passthrough
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 additive optional fields. #[serde(default)] preserves
backwards compat for older JSON without the new keys.
MEDIA_KINDS const exposes canonical "markdown"/"pdf"/"image"/
"audio"/"other" labels for downstream alias normalization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Domain types for `kebab fetch` 3 modes (chunk / doc / span). All
types Serialize so wire layers hand them through serde_json
directly. FetchKind is snake_case-renamed to match the wire
discriminator literal in fetch_result.v1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RetrievalDone after retrieve+stale-stamp, Token per LM chunk
(SendError → break, FinishReason::Cancelled, RefusalReason::
LlmStreamAborted), Final on success. answers row still persists
on cancel for audit. Adds FinishReason::Cancelled, re-exports
StreamEvent from kebab_rag, migrates two pre-fb-33 sink tests
in tests/pipeline.rs to the new StreamEvent type (the
"dropped receiver does not abort" test inverts to record cancel).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Domain field additions for p9-fb-32. Wire serialization is
automatic via serde rfc3339. Other crates fail to compile until
they populate the new fields — fixed in subsequent tasks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `DocumentStore::get_asset_by_workspace_path` trait method to
`kebab-core` and implement it on `SqliteStore` via a private
`asset_from_row` helper. Used by the incremental-ingest skip path to
compare a freshly-computed blake3 checksum against the persisted row
without a full round-trip through `put_asset_with_bytes`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spec PR #59 의 §3.8 multi-turn behaviour 구현. RAG facade 가 prior
turns 받아 prompt 에 prepend, retrieval query expansion 적용,
Answer 에 conversation_id / turn_index 채움.
신규 (kebab-core):
- Answer 에 conversation_id (Option<String>) / turn_index (Option<u32>)
field 추가. serde skip_serializing_if 로 single-shot 의 wire
output 변경 0 (기존 외부 wrapper 영향 없음).
- Turn struct (question + answer + citations + created_at).
- RefusalReason::LlmStreamAborted variant.
신규 (kebab-rag):
- AskOpts 에 history (Vec<Turn>) / conversation_id / turn_index 3 field.
- AskOpts::single_shot(mode) helper.
- RagPipeline::ask_with_history(query, history, conversation_id,
turn_index, opts) — combined opts 로 ask 호출.
- expand_query_with_history: history.last() 의 answer 첫 200 자
concat 해 SearchQuery.text 확장 (spec §3.8 의 \"cheap concat\";
LLM-based standalone-question rewriting 은 P+).
- serialize_history + remaining_history_budget_chars: spec 의 priority
enforcement — system+question 필수, retrieved chunks 가 차지한
뒤 남은 char budget 안에서 newest 우선, oldest drop.
- ask 본문: history 가 비어있지 않으면 [이전 대화] 블록을 user
prompt 위에 prepend. Answer 생성 site 3 곳 (정상 / NoChunks /
ScoreGate refuse) 모두 conversation_id / turn_index 채움.
신규 (kebab-store-sqlite):
- refusal_reason_label 가 LlmStreamAborted → 'llm_stream_aborted'.
기존 caller 변경 (single-shot 동작 동일):
- kebab-cli main.rs Cmd::Ask: AskOpts 에 history=Vec::new(),
conversation_id=None, turn_index=None 명시 (CLI multi-turn 은
p9-fb-18 의 --session/--repl 가 채움).
- kebab-tui src/ask.rs spawn site 동일 (multi-turn UI 는 p9-fb-16).
- kebab-eval runner.rs golden eval 동일 (single-shot per query).
- kebab-app tests/ask_smoke.rs / kebab-tui tests/ask.rs / kebab-rag
tests/pipeline.rs / kebab-eval metrics.rs Answer literal 갱신.
Test:
- 9 신규 lib unit (expand_query 4 / serialize_history 3 / remaining_budget 2).
- 기존 12 PASS 회귀 0.
Plan 갱신:
- p9-fb-15 status planned → in_progress. 머지 후 한 줄 commit
으로 completed flip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P7-3 의 storage UNIQUE bug fix 가 SQLite 측 (documents → blocks /
chunks / embedding_records) 만 sweep 했음. LanceDB 의 vector 는 별도
store 라 옛 chunk_id 를 가진 row 가 디스크에 잔존. 검색에는 영향 없지만
디스크는 무한 누적. HOTFIXES `2026-05-02 P7-3` caveat 의 "P+ task" 약속을
같은 후속 PR 안에서 닫음.
변경:
- `VectorStore::delete_by_chunk_ids(&[ChunkId])` trait method 추가 (default
no-op 제공 — 테스트 fake / 기존 impl 이 그대로 컴파일).
- `LanceVectorStore::delete_by_chunk_ids` 가 connection 의 모든
`chunk_embeddings_*` 테이블을 순회 + `Table::delete("chunk_id IN (...)")`
를 batch=200 단위로 실행. 다중 모델 워크스페이스 (마이그레이션 중간 등)
에서도 안전.
- `SqliteStore::stale_chunk_ids_at(workspace_path, new_asset_id)` 가
read-only SELECT 로 옛 chunk_id 들 반환. CASCADE 가 흐르기 *전* 에
caller 가 호출.
- `kebab-app::purge_vector_orphans_for_workspace_path` 가 위 두 단계를
orchestrate. 세 ingest path (markdown / image / pdf) 의
`put_asset_with_bytes` 호출 직전에 한 줄로 호출.
Smoke 검증 (release binary, fastembed enabled):
- whitepaper.pdf 첫 ingest → chunk_ids = {f616…, 4e0f…}, vector store 에
그 두 ID 의 row 존재.
- byte 변경 후 re-ingest → 새 doc_id (3741…) + 새 chunk_ids
(ed0c…, e13c…). vector search "REWRITTEN chapter two" → 새 chunk_ids 만
hit. 옛 query "Edited page two body" 시도해도 옛 chunk_ids 는 vector
store 에 더 이상 없음 (의미적으로 가장 가까운 새 chunks 가 hit).
HOTFIXES `2026-05-02 P7-3` 의 \"vector store cleanup\" 항목이 \"deferred\" →
\"closed by follow-up PR\" 로 갱신. SMOKE.md 의 알려진 동작 (\"옛 vector
잔존\") 도 \"두 store 정합\" 으로 갱신.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
