crates/kebab-app/tests/ingest_log_smoke.rs 신규:
* ingest_log_smoke (AC-9): tempdir + 1 md + 1 scanned PDF →
ingest → assert log file exists + 각 line valid JSON +
각 kind ∈ {ocr,parse_error,skip,error,summary} + last
line kind=summary + scanned>0.
* ingest_log_disabled_emits_no_file (AC-6): enabled=false 일
때 log_dir 안 ingest-*.ndjson 파일 0개 verify.
fixture: ../kebab-parse-pdf/tests/fixtures/scanned_page1.pdf
재사용 (OCR disabled — Ollama 없이 smoke 실행).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
schema_report_reflects_freshly_ingested_kb 가 `!streaming_ask` 를 assert 했으나
Bug #9 fix (760eee8) 로 streaming_ask 가 true 로 정정됨. assertion 을 반전.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
이전: schema.v1.models 가 parser_version / chunker_version 단일 값만 보고 →
multi-medium corpus (md + pdf + code Rust/Python + dockerfile + k8s + manifest)
의 version cascade audit 누락 risk.
이후: additive minor — Models struct 에 active_parsers + active_chunkers Vec<String>
추가. backward compat: 기존 단일 field 보존 (markdown default), 신규 array 는
optional (#[serde(default)] + JSON schema required 미포함).
source:
- kebab_store_sqlite::fetch_distinct_parser_versions() 가
documents.parser_version DISTINCT + ORDER BY 반환.
- fetch_distinct_chunker_versions() 가 chunks.chunker_version 동일 pattern.
- collect_models 가 매 schema 호출마다 재계산 (cache 없음 — R-3 자동 해결).
wire schema additive only — 메이저 bump 불필요. v0.20.1 minor 로 충분.
integrations/claude-code/kebab/SKILL.md 동기 갱신.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR-2 of fb-41 multi-hop RAG. Decompose + retrieve + synthesize 3-stage
pipeline가 `opts.multi_hop=true` 일 때 dispatch. Dynamic decide loop
는 PR-3.
- `AskOpts.multi_hop: bool` 필드 추가 + `impl Default for AskOpts`
도입 (HOTFIXES 2026-05-07 의 known limitation 해소). 9 explicit
init site 모두 `multi_hop: false` 추가 — Default 도입으로 향후
`..Default::default()` 점진 migrate 가능.
- `RagPipeline::ask` 의 entry 에 dispatcher 한 줄
(`if opts.multi_hop { return self.ask_multi_hop(...) }`).
- `RagPipeline::ask_multi_hop` 신규 method. 1) decompose LLM call
→ JSON array of strings parse, 2) 각 sub-query 로 retrieve +
chunk_id dedup pool, 3) score gate / no-chunks 가드, 4)
pack_context (single-pass 와 helper 공유), 5) synthesize LLM
call w/ MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT, 6) citation extract
+ Answer build. `prompt_template_version` = "rag-multi-hop-v1"
로 stamp — eval `compare` 가 single-pass vs multi-hop 분리.
- Prompt const 신규: MULTI_HOP_DECOMPOSE_SYSTEM_PROMPT +
MULTI_HOP_DECOMPOSE_USER_TEMPLATE + MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT
+ PROMPT_TEMPLATE_VERSION_MULTI_HOP + MULTI_HOP_MAX_SUB_QUERIES_DEFAULT.
- `kebab_core::RefusalReason::MultiHopDecomposeFailed` variant 신규.
Cascade: kebab-store-sqlite `refusal_reason_label` + kebab-tui `ask
refusal render` exhaustive match 갱신.
- `parse_decompose_response` + `strip_markdown_json_fence` helper —
markdown code fence (```json / ```) strip + JSON array of strings
parse + trim + drop empty + cap at MULTI_HOP_MAX_SUB_QUERIES_DEFAULT.
None 반환 시 caller 가 `MultiHopDecomposeFailed` refusal.
Tests (55 passing total, 8 신규):
- 6 unit (parse_decompose_response 의 bare array / fence variants /
garbage / cap / trim 회귀 핀).
- 2 integration: `ask_multi_hop_dispatches_and_decompose_garbage_refuses`
(decompose garbage → MultiHopDecomposeFailed + 정확히 1 LLM call) +
`ask_with_multi_hop_false_keeps_single_pass_path` (회귀 핀, 기존
caller 자동 backwards-compat).
Happy-path multi-hop (decompose 성공 → synthesize) 의 integration
test 는 ScriptedLm helper 가 PR-3 의 decide loop 와 함께 도입될
때 같이 추가. 현 `MockLanguageModel` 는 canned single response 라
2-LLM-call sequence 핀 불가.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
closure of HOTFIXES 2026-05-21. C typedef-wrapped anonymous
struct/enum/union 이 typedef alias 이름으로 symbol unit 방출.
- crates/kebab-parse-code/src/c.rs: type_definition 분기 추가.
inner anonymous struct_specifier / enum_specifier / union_specifier
탐지 → declarator field 의 type_identifier 재귀 추출 → synthetic
unit (typedef alias). named inner aggregate / plain alias 는
기존대로 glue. PARSER_VERSION code-c-v1 → code-c-v2.
recover_typedef_alias + extract_typedef_alias_name helper 추가.
- crates/kebab-store-sqlite/src/store.rs: 두 helper 신규
(parser_version bump cascade 용 doc-id 기반 orphan purge).
- stale_chunk_ids_for_workspace_path_except_doc_id(workspace_path,
keep_doc_id) — sister of stale_chunk_ids_at, doc_id 기반.
- purge_document_at_workspace_path_except_doc_id(workspace_path,
keep_doc_id) — CASCADE document/chunks 제거, assets 보존.
keep_doc_id="" 가 "모든 doc 제거" 사용.
- crates/kebab-app/src/lib.rs: try_skip_unchanged 의 parser_mismatch
분기에서 purge_workspace_path_for_parser_bump 호출. helper 가
app.vector() 로 lazy 접근 + delete_by_chunk_ids + SQLite document
row 제거. Ok(None) 반환 전 cleanup 끝나서 caller 의 새 INSERT 시
idx_docs_workspace_path UNIQUE 충돌 회피.
- tests:
- c.rs unit tests 4 신규 — typedef_struct_emits_unit /
typedef_enum_emits_unit / typedef_union_emits_unit /
typedef_to_existing_type_stays_glue (negative).
- tier1_c_ingest_searchable: parser_version assertion code-c-v1 →
code-c-v2.
- 회귀: bytes-edit 경로 (asset_id 변경) 의 기존 purge_orphan_at_workspace_path
+ purge_vector_orphans_for_workspace_path 는 그대로 — 신규 분기와
공존, 기존 test 모두 PASS.
미해결 (Risks): nested typedef (typedef struct { struct {...} inner; }
Outer;) 의 inner 익명 struct 는 여전히 glue — v2 의 1차 범위는
top-level typedef alias 만.
cargo test --workspace --no-fail-fast -j 1 + clippy 통과.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
trigram tokenizer 가 snippet 단위 + 단어 경계 + BM25 raw score 분포를
모두 바꿔서 unicode61 assumption 기반의 3 test 가 regression.
- wire_search_response::search_json_truncates_with_max_tokens +
search_plain_emits_truncated_hint_to_stderr: 단일 doc + 작은
max_tokens 로는 snippet 이 짧아서 budget loop 가 trip 안 함.
다중 doc fixture (5 doc) + budget 30 token 으로 hit-pop 경로
통해 truncated=true 보장.
- fetch_integration::fetch_chunk_with_context_returns_neighbors:
fixture body 의 2-char tokens (A1/A3 등) 가 trigram 비호환으로
0-hit. apples/banana/cherry/durian/elder 5-char unique words
로 갱신, query 도 cherry 로 deterministic pin.
- eval/runner::runner_per_query_snapshot_matches_fixture: trigram
token stream 으로 BM25 raw score 변동. UPDATE_SNAPSHOTS=1 로
regenerate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P10 dogfooding found that a k8s manifest with 2+ documents (e.g.
Deployment + Service in one file) fails to ingest:
UNIQUE constraint failed: chunks.chunk_id
Root cause: tier2_shared::push_chunks_with_oversize's non-oversize branch
hardcoded split_key = None. K8sManifestResourceV1Chunker calls it once per
resource; with split_key None every resource from the same document gets
the same id_hash (= base_policy_hash) → identical chunk_id. p10-3's
code_text_paragraph_v1 had the same bug (fixed in df3c5b8) but it calls
build_chunk_no_symbol directly — the push_chunks_with_oversize path was
never fixed.
Fix: push_chunks_with_oversize gains a base_split_key parameter for the
non-oversize single-chunk case. k8s chunker passes Some(resource.line_start)
so each resource gets a distinct chunk_id; dockerfile / manifest pass None
(1 chunk per file — no sibling collision, chunk_id stays stable).
Regression coverage: k8s_multi_doc_emits_one_chunk_per_resource now asserts
chunk_id distinctness; new integration test
tier2_k8s_multi_resource_yaml_ingests_without_collision ingests a real
2-document YAML end-to-end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PR #155 (p10-3) merged WITHOUT the reviewer's required Option B1 fix —
the implementer reported a commit SHA (2a39513) that never made it to main.
Result: every reingest of a Tier 3-fallback file (non-k8s YAML, invalid
YAML, AST extractor failure) re-runs full extract + chunk + embed because
the parser/chunker version comparison can never match (stored is
code-text-paragraph-v1 / none-v1, but caller uses Tier 1/2 dispatch
values).
This commit:
1. Adds the 7th param `fallback_chunker_version: Option<&ChunkerVersion>`
to try_skip_unchanged + the stored_is_tier3_fallback detection branch
(skip parser/chunker equality, keep embedder check).
2. Threads `None` through non-code call sites (md / image / pdf).
3. Code call site computes tier3_fallback_cv covering all Tier 1/2 langs
that can fall back: rust / python / ts / js / go / java / kotlin /
yaml / dockerfile / toml / json / xml / groovy / go-mod / c / cpp
(p10-1D additions).
4. Adds tier3_yaml_fallback_reingest_is_unchanged + tier3_shell_reingest_is_unchanged
regression tests (the originally-promised PR #155 regression coverage
that also never made it to main).
Smoke tests: 14 + 2 = 16 PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new tests verify end-to-end Tier 3 wiring:
- tier3_shell_ingest_searchable: .sh file → --code-lang shell search →
Citation::Code { symbol: None, lang: "shell" }, chunker_version
"code-text-paragraph-v1".
- tier3_yaml_fallback_picks_up_non_k8s_yaml: docker-compose-shaped yaml
(no apiVersion/kind) triggers k8s chunker's Ok(vec![]) result, fallback
retries with Tier 3 → Citation::Code { symbol: None, lang: "yaml" } and
chunker_version "code-text-paragraph-v1".
Also fixes a bug in CodeTextParagraphV1Chunker (Task B): short paragraphs
(≤80 lines) were emitted with split_key=None, causing all paragraphs from the
same document to share the same chunk_id (UNIQUE constraint violation at
put_chunks). Fix: always use para.line_start as split_key so every paragraph
gets a distinct id regardless of size.
Brings code_ingest_smoke to 14 tests (Tier 1: 9, Tier 2: 3, Tier 3: 2).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces Go bail! arms with GoAstExtractor + CodeGoAstV1Chunker. Adds
go_file_ingests_and_searches_as_code_citation integration test — asserts
citation.lang=go, symbol=chunk.ParseDoc, code_lang=go.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
assets.workspace_path is INTENTIONALLY 'last-registered path' for twin
files (identical content at different paths share one asset row PK'd by
blake3 content hash). PR #146 made try_skip_unchanged document-centric;
PR #149 made reset --orphans-only document-centric; this PR removes the
last caller of get_asset_by_workspace_path (fetch.rs:193 in fetch_span,
which used it to reject PDF/audio media — for twins this could read the
wrong asset's media_type and pick the wrong branch).
Replaced with the natural 2-step lookup: get_document_by_workspace_path
(PR #146) → doc.source_asset_id → get_asset (NEW trait method, asset_id
is PRIMARY KEY so flip-flop-immune by construction).
Then removed get_asset_by_workspace_path trait method + SqliteStore impl
— 0 callers after the refactor.
UPSERT doc-comment refreshed in store.rs to make the 'last-registered'
semantics explicit so future readers don't try to 'fix' the flip-flop.
Dogfood follow-up (PR #142 1B + multi-root corpus).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #148 auto-purges only filesystem-missing files (conservative — leaves
on-disk-but-out-of-scope docs alone for data safety). This is the explicit
complement: when the user has narrowed include / widened exclude / removed
a sub-directory from the workspace and WANTS the stored docs reconciled,
they invoke 'kebab reset --orphans-only'.
Confirm prompt with orphan count + sample paths; --yes required in
non-TTY. SQLite purge via existing purge_deleted_workspace_path (PR #148)
+ vector store delete_by_chunk_ids when configured. No fs existence
check — orphans-only is the explicit 'I know what I'm doing' variant.
dogfood follow-up to PR #148 (file deletion auto-purge).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files deleted from disk (rm a.md) were leaving stale documents + chunks +
embeddings in the store, surfacing as ghost citations in search/ask.
Existing purge_orphan_at_workspace_path only handled content-changed
stale (WHERE workspace_path=? AND asset_id != ?) — file deletion has no
new asset_id.
Fix: post-walker-scan sweep. Compute (stored_paths - scanned_paths),
for each candidate check filesystem existence — only purge when the
file is TRULY missing. Scope-narrowing case (file on disk but outside
include glob) is explicitly NOT purged to protect users from accidental
data loss via config edits.
Adds:
- DocumentStore::all_workspace_paths trait method + SqliteStore impl
- purge_deleted_workspace_path in store-sqlite (returns chunk_ids for
vector delete; deletes doc CASCADE + asset row + copied storage file)
- sweep_deleted_files in kebab-app::ingest path; called once per ingest
before the per-asset loop
- IngestReport.purged_deleted_files counter (additive, serde default)
- CLI ingest summary mentions purge count when > 0
- 2 integration tests: file_deletion_auto_purge + include_scope_narrowing_does_NOT_purge
dogfood discovery (PR #142 1B + multi-root: kebab-docs + httpx + zod
+ lodash). Per user decision: only filesystem deletion auto-purges;
scope narrowing requires explicit kebab reset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Identical-content files at different workspace paths share one assets row
(assets.asset_id = blake3 content hash, PRIMARY KEY). The UPSERT
`ON CONFLICT(asset_id) DO UPDATE SET workspace_path = excluded` made
twin files overwrite each other's workspace_path on every ingest, so
`get_asset_by_workspace_path(path1)` returned the OTHER twin's row (or
None) — break idempotent unchanged-detection for both files.
Fix: switch try_skip_unchanged to document-centric lookup. `documents.
workspace_path` is already UNIQUE (V001) and `id_for_doc(path, ...)`
includes path, so each twin has its own stable document row. Compare
`doc.source_asset_id` with the new asset's checksum instead of going
through the assets table.
Dogfood (multi-root: kebab-docs + httpx + zod + lodash) showed 27 of
726 docs marked Updated on every idempotent re-ingest — all 27 are
twin-file victims (empty `__init__.py` ×3, AGENTS.md ↔ CLAUDE.md
same content, duplicate logo PDFs/JPGs).
After: re-ingest reports 0 new / 0 updated / 726 unchanged.
No schema migration needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task H implementer chose code-typescript-v1 but plan + design §3.3 use the
short form (chunker is code-ts-ast-v1 / code-js-ast-v1). Aligning parser
versions to match: rust=code-rust-v1 / python=code-python-v1 / ts=code-ts-v1
/ js=code-js-v1 (Task K). Fixes 2 sites: const PARSER_VERSION + integration
test assertion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `MediaType::Code("rust")` dispatch arm in `ingest_one_asset`,
`ingest_one_code_asset` fn (faithful mirror of `ingest_one_pdf_asset`),
and `backfill_code_lang` post-processing in `App::search_uncached`.
Integration test `code_ingest_smoke.rs` verifies full pipeline:
ingest `.rs` → Citation::Code hit with lang/symbol/line_start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends CountSummary with media_breakdown, lang_breakdown, stale_doc_count
fields populated via stats_ext::breakdowns(). Adds count_summary_with_threshold
for callers that need real stale counts. Mirrors all new fields onto the
wire-bound Stats struct in kebab-app::schema with #[serde(default)] for
backwards-compat. Also fixes search_budget_integration.rs for the trace field
added to SearchOpts in Task 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- wire schema: relax effective_end.minimum 1 → 0 + expand
description to cover line-clamp + out-of-range sentinel
(panic-fix R1 emits Some(0) when line_start=1 and range is
beyond doc end — schema must accept it)
- tests: tighten first-chunk-target boundary test to assert ≤ 2
total neighbors (3-chunk doc, N=2). Strict "first chunk →
context_before empty" not assertable until chunks.ordinal
column lands (R1 #9 architectural caveat)
- store: trim contradiction in list_chunk_ids_for_doc warning
comment — drop "good enough for sequentially chunked
markdown" phrase that conflicts with "hash sort dominates"
paragraph above
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Line-based slice over fmt_canonical_to_markdown output.
PDF / audio source_type → span_not_supported StructuredError.
Out-of-range line_end clamps to total; effective_end reflects
post-budget trim. invalid_input on zero / inverted bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Walks CanonicalDocument blocks, serializes to markdown, applies
chars/4 budget when opts.max_tokens is set. doc_not_found
preserved through StructuredError.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chunk mode + +-N context. doc / span modes return placeholder
errors (filled by subsequent tasks). fmt_canonical_to_markdown
helper introduced now since doc mode (Task 4) consumes it.
Errors are typed StructuredError so classify preserves
chunk_not_found / doc_not_found through the wire layer.
Adds SqliteStore::list_chunk_ids_for_doc so the facade can derive
+-N neighbors without leaking direct rusqlite usage into kebab-app.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous round-1 fix dropped the speculative cursor branch on
the truncated path, leaving a contradiction with the docs:
- snippet-only shrunk → cursor emitted (returned == k_effective)
- k-popped → cursor null (returned < k_effective)
But docs promised the opposite.
R2 resolution: emit cursor whenever more hits may be reachable
(either retriever filled the page OR budget popped hits — the
popped ones remain fetchable from offset+returned). Drop the
artificial "widen vs paginate" copy; truncated and next_cursor
are now independent signals — caller may do either or both.
Updates: app.rs::search_with_opts logic + SearchResponse doc +
schema description + SKILL.md two bullets + max_tokens=0 test
asserts cursor IS emitted on k-pop case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- error_wire: StructuredError wrapper preserves ErrorV1 through
anyhow → classify pipeline. Adds downcast short-circuit so
cursor::decode's typed code = "stale_cursor" reaches the wire
instead of being string-formatted to code = "generic".
- app: search_with_opts now wraps cursor::decode error in
StructuredError instead of anyhow! string format.
- test: error_wire pins both negative (bare anyhow → not
stale_cursor) AND positive (StructuredError → stale_cursor)
invariants. CLI integration test runs end-to-end and asserts
error.v1.code on stderr.
- app: next_cursor only emitted on full-page (k-pop) path; drop
speculative emit on snippet-only truncation that would point at
a different page than the agent expected.
- cursor: differentiate malformed-base64 / malformed-payload /
revision-mismatch error messages; all keep code = stale_cursor.
- test: cursor_rejected fixture uses .expect() to fail loud on
cursor non-emission instead of silent skip.
- test: max_tokens=0 → 1-hit floor + truncated=true.
- docs: SKILL.md + schema description distinguish snippet-shrink
(widen) vs k-pop (paginate) truncated cases. HOTFIXES notes
--no-cache semantic shift (cached path + clear vs uncached path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Budget loop: snippet shorten → k pop → ≥1 hit floor. Cursor
encode/decode threads corpus_revision; mismatch surfaces as
stale_cursor anyhow error. App::search retained as thin wrapper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opaque base64(JSON{offset, corpus_revision}). Mismatch or
malformed input returns ErrorV1 with code = stale_cursor.
base64 promoted to workspace dep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compute_stale: strict > boundary, threshold=0 disables, future
timestamps treated as fresh (clock skew safety). App::search
re-stamps on cache hit so config threshold changes take effect
without flushing the cache.
Also unblocks the workspace build by plugging placeholder
indexed_at/stale into the two AnswerCitation construction
sites in kebab-rag/pipeline.rs (the score-gate refusal path
forwards from SearchHit; the LLM-citation path uses
UNIX_EPOCH/false until Task 7 wires the real values through
pack_context).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- ingest_file_with_config: lowercase normalize ext (caller-side) +
early error on unsupported extension (`.docx` etc. now Err with
helpful message instead of silent skipped_by_extension counter).
New test ingest_file_errors_on_unsupported_extension.
- ingest_stdin_with_config: doc comment explaining intentional
double-call of ensure helpers (idempotent + ~ms negligible).
- external::inject_frontmatter: simplify precheck via single
trim_start binding + add CR-only line ending edge case.
- external::inject_frontmatter: doc note on yaml_quote escape
contract (agent-supplied titles with special chars are safe).
Round 1 review summary: #111 (comment)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three scenarios — copies external md + reports new=1, idempotent on
second call (unchanged=1), errors on missing path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plug coverage hole flagged in code review — test 1 was asserting only
doc_count + last_ingest_at, leaving count_summary's chunk_count and
asset_count queries un-pinned.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two scenarios: freshly-ingested 2-doc KB (stats reflect counts +
last_ingest_at populated) and empty-but-initialized KB (counts zero,
last_ingest_at None). The empty case runs ingest_with_config over an
empty workspace dir to seed kebab.sqlite before calling schema_with_config,
since open_existing (used internally) returns NotIndexed if the DB is absent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
