Compare commits

...

119 Commits

Author SHA1 Message Date
c1e82cca92 Merge pull request 'refactor(app): extract dispatch polymorphism — App.extract_for(...) + 11 Extractor registry' (#187) from refactor/extractor-dispatch-unification into main
Reviewed-on: #187
2026-05-26 21:07:20 +00:00
2c05dbd0dd refactor(app): extract dispatch polymorphism — App.extract_for(...) + 11 Extractor registry
kebab-app 의 hardcoded extract dispatch (`ImageExtractor` + `PdfTextExtractor` + 9 AST `*Extractor` 의 `::new().extract(…)` callsite 11곳 + 9 AST arm match) 를 `App::extract_for(&MediaType, &ExtractContext, &[u8])` 단일 polymorphic call 로 통합. trait 변경 0, parser source 변경 0, wire schema 변경 0 (success path).

핵심 변경:
- App struct 에 `pub(crate) extractors: Vec<Box<dyn Extractor + Send + Sync>>` field + `pub(crate) fn extract_for(...)` helper method.
- App::open_with_config 의 registry init = 11 Extractor (image + pdf + 9 AST).
- ImagePipeline struct 의 `extractor: &'a ImageExtractor` field 제거 + lib.rs:356 local + lib.rs:1235 alias 삭제 (atomic block).
- 9 AST arm (lib.rs:2012-2047 의 12 arm = 11 explicit + 1 wildcard) → 4 arm (9 AST grouped + 7 manifest + 1 shell + 1 other-bail).
- in-crate unit test (app.rs 의 `mod tests_extractor_dispatch`) 3 class: registry length 11 / mutually-exclusive supports() grid (16 sample MediaType) / extract_for error path (Audio).

scope = AST 9-arm + image + pdf extract callsite only. MarkdownExtractor / Tier 2/3 / outer 4-arm / inner 4 match / Chunker dispatch 모두 future-defer (별 PR — spec §11).

Wire schema (success path) 변경 0 — ingest_report.v1 / search_response.v1 / answer.v1 byte-identical (4-medium SMOKE 비교 검증). error.v1.message 의 internal context string wording 변경 (예: `kb-parse-image::ImageExtractor::extract` → `kb-app::extract_for (image)`) 은 spec §5.5 risk acceptance — `error.v1.code` + `error.v1.schema_version` 보존, user-visible surface 외. Cargo workspace.version bump 0.

Refs:
- docs/superpowers/specs/2026-05-26-extractor-dispatch-unification-spec.md (2 round APPROVE)
- docs/superpowers/plans/2026-05-26-extractor-dispatch-unification-plan.md (3 round APPROVE)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-26 17:43:44 +00:00
96766406aa Merge pull request 'refactor(parse-md): absorb kebab-normalize + kebab-parse-types — 24 → 22 crates + §3.7b 재작성' (#186) from refactor/normalize-absorption into main
Reviewed-on: #186
2026-05-26 15:37:21 +00:00
710945c4b0 refactor(parse-md): absorb kebab-normalize + kebab-parse-types — 24 → 22 crates + §3.7b 재작성
design §3.7b 의 thin layer (ParsedBlock 류) 가 4 parser 중 1개 (markdown) 만 lift 를
경유하는 현실 — fan-in/fan-out 모두 1 → layer 의미 잃음. kebab-normalize (1097 LOC)
+ kebab-parse-types (98 LOC) 둘을 kebab-parse-md 로 흡수.

설계: docs/superpowers/specs/2026-05-26-normalize-absorption-spec.md
플랜: docs/superpowers/plans/2026-05-26-normalize-absorption-plan.md
HOTFIXES: tasks/HOTFIXES.md 의 2026-05-26 entry (design deviation)

- 5 사용 type + 3 forward-declared struct → kebab-parse-md::types module 의 pub explicit re-export.
- build_canonical_document + derive_title + warning_agent → kebab-parse-md::normalize module.
- 4 hard-coded agent literal (lib.rs:122/128/134/153) + warning_agent body return + tracing target literal 모두 보존 — stage label 일관성.
- kebab-app callsite (lib.rs:51 use + :1119 context string) + Cargo.toml 의 2 dep (regular + dead) 제거.
- kebab-chunk + kebab-store-sqlite 의 [dev-dependencies] kebab-normalize → 제거 (kebab-parse-md 로 갈음). 통합 test source 의 use shift.
- test file 이동 (kebab-normalize/tests/normalize_snapshot.rs → kebab-parse-md/tests/).
- workspace Cargo.toml: Hunk (a) members 2 entry 삭제 + Hunk (b) version 0.18.0 → 0.19.0 (frozen contract 변경).
- design §3.7b 4-단락 재작성 (원래 intent 보존 + 현재 상태 + 보존된 surface + future re-extraction trigger).
- design §8 graph 갱신 (3 edge 제거 + 2 forbidden bullet 의미 갱신 + commentary).
- ARCHITECTURE.md crate graph + directory tree mechanical 갱신.
- tasks/INDEX.md L169 closure mention + "Future work / deferred" 섹션 신설 (image/pdf normalize integration entry).
- tasks/HOTFIXES.md 신규 entry (4-block — design deviation Symptom).
- HANDOFF.md cross-link 한 줄.
- 3 dead struct (ParsedImageRegion / ParsedPdfPage / ParsedAudioSegment) 는 보존 — v0.20+ image/pdf normalize integration 의 future surface (spec §11).

Wire / surface impact: 0건. CLI / TUI / MCP / --json 출력 / config / XDG path /
parser_version 모두 unchanged. wire-invisible provenance.events[].agent + tracing target
literal "kb-normalize" 도 보존 — old DB row 와 new DB row 의 audit log 일관성.

Verification: cargo test --workspace --no-fail-fast -j 1 → 1313 passed / 0 failed (172 result blocks).
cargo clippy --workspace --all-targets -j 1 -- -D warnings → 0 warning (5m 46s).
cargo metadata --no-deps --format-version 1 | jq '.workspace_members | length' = 22.
cargo tree -p kebab-app --depth 2 | grep -E "kebab_(parse_types|normalize)" = 0 줄.
2026-05-26 15:00:59 +00:00
d4395a306b Merge pull request 'refactor(source-fs): drop kebab-parse-code dep — 9 tree-sitter grammars drag 제거' (#185) from refactor/source-fs-dep-lightening into main
Reviewed-on: #185
2026-05-26 12:31:29 +00:00
bd48baa19a refactor(source-fs): drop kebab-parse-code dep — 9 tree-sitter grammars drag 제거
kebab-source-fs 가 kebab-parse-code 의 9 tree-sitter grammars 를 drag 했던 무거운 의존성 제거. 4 surface (code_lang_for_path / is_generated_file / is_oversized / BUILTIN_BLACKLIST) 만 사용하지만 dep 그래프에서 9 grammar 전체 link → kebab-source-fs::code_meta 로 이전 + kebab-parse-code 측 cleanup.

핵심 변경:
- kebab-source-fs::code_meta 신설: 4 surface 이전 (BUILTIN_BLACKLIST `pub` for frozen contract + 3 helper fn `pub(crate)`). lib.rs 의 `pub use code_meta::BUILTIN_BLACKLIST` 1 줄 추가 (Option A — 다른 mod surface 무근거 확장 0).
- callsite migration: media.rs (1) + walker.rs (2) + connector.rs (2) 모두 `kebab_source_fs::code_meta::*` 로 갱신.
- kebab-parse-code 측 cleanup: skip.rs 삭제 + lang.rs narrow edit (code_lang_for_path body + unit test 2 + Path import 삭제, module_path_for_* 보존) + lib.rs 헤더 doc rewrite (migration breadcrumb 포함).
- tests/{lang,skip}.rs 13 test 이동 — 12 unit (`src/code_meta.rs::tests`) + 1 integration (`tests/code_meta.rs` for BUILTIN_BLACKLIST frozen contract).
- design §8 graph: edge 제거 + p10-2 inline note. ARCHITECTURE.md 산문 1 줄 갱신. kebab-core::metadata.rs:36 stale dep reference 정정.

G1+G5: cargo tree -p kebab-source-fs | grep tree-sitter = 0 줄.
G2+G3: workspace test 회귀 0 + 13 test 1:1 이동.
G4: design §8 + ARCHITECTURE.md 갱신.

Wire 영향: 없음 (internal Rust crate-API surface 만, user-facing 0). Cargo workspace.version bump 불필요.

Refs:
- docs/superpowers/specs/2026-05-26-source-fs-dep-lightening-spec.md (v3, 4-round APPROVE)
- docs/superpowers/plans/2026-05-26-source-fs-dep-lightening-plan.md (v4, 4-round ACCEPT)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 12:19:32 +00:00
b02ac8200e Merge pull request 'fix(rag): S3 NLI unavailable — hypothesis char budget + token-count fallback retry' (#184) from fix/s3-nli-model-unavailable-diagnose into main
Reviewed-on: #184
2026-05-26 09:17:12 +00:00
336962715a fix(rag): S3 NLI unavailable — hypothesis char budget + token-count fallback retry
S3 dogfood query 의 `nli_model_unavailable` consistent fail root cause = mDeBERTa-v3 tokenizer 의 `OnlyFirst` strategy + 949-token hypothesis. 기존 char-budget 단독 fix 의 KR-extreme density 미해결 → token-count fallback retry + RC1-residual trait dispatch 정합.

핵심 변경:
- kebab-nli::NliVerifier: `hypothesis_token_count(&str) -> Result<usize>` trait method 추가 (default `Ok(0)` backward-compat). `OnnxNliVerifier` 가 *trait impl block* 안에서 real mDeBERTa tokenize override — vtable 등록 보장 (round-3 critic RC1-residual closure).
- kebab-rag::pipeline: `MAX_NLI_HYPOTHESIS_CHARS_INITIAL = 1200` + `MAX_NLI_HYPOTHESIS_CHARS_MIN = 150` const + `pub(crate) fn truncate_chars` pure-fn + `pub fn truncate_hypothesis_for_nli_with_budget` retry helper (char budget 반감 retry, min floor 시 graceful unavailable). step 8.5 hook 의 callsite explicit `match` + `return self.refuse_nli_model_unavailable` 패턴 (`?` 금지 — round-2 plan critic CRITICAL #1 closure).
- SpyNliVerifier 신규 helper (closure score_fn + hypothesis_token_count_fn, 2-arg constructor).
- §5.1 의 2 ignored test (EN-long err + vtable dispatch RC1-residual pin) + §5.2 의 4 boundary test (truncate_chars) + §5.3 의 3 mock multi-hop test (long_en_grounded / long_kr_retries / unrelenting_fallback). +7 new tests (2 ignored default skip).
- tasks/HOTFIXES.md 신규 dated entry `## 2026-05-26 — S3 NLI unavailable ...` — Symptom / Root cause / Action / Amends 4-block.
- spec + plan (`docs/superpowers/{specs,plans}/2026-05-26-s3-nli-model-unavailable-diagnose-*.md`) — 4 round spec + 3 round plan OMC reviewer ACCEPT 산출물.

검증:
- cargo test -p kebab-nli -j 1 → 11/11 pass + 7 ignored default skip.
- cargo test -p kebab-rag -j 1 → 19+3+3+... 전체 pass + 3 new mock + 4 new boundary.
- cargo test --workspace --no-fail-fast -j 1 → **1313 pass (+7 new)**, 0 failed. 회귀 0 (HOTFIX #15 이미 fixed, no remaining flaky).
- cargo clippy --workspace --all-targets -j 1 -- -D warnings clean (type_complexity allow on Arc<dyn Fn> type aliases).

KR safe (token-count retry path) + graceful fallback (min floor 시 기존 unavailable wire 유지, regression 0). Wire 영향 없음 (additive trait method). Cargo bump 불필요.

Refs:
- spec: docs/superpowers/specs/2026-05-26-s3-nli-model-unavailable-diagnose-spec.md (4 round APPROVE — analyst → critic + verifier × 4 rounds)
- plan: docs/superpowers/plans/2026-05-26-s3-nli-model-unavailable-diagnose-plan.md (3 round ACCEPT — planner → critic-plan + verifier-plan × 3 rounds)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 09:12:21 +00:00
1a224bf983 Merge pull request 'fix(mcp): HOTFIX #15 — MCP ask multi_hop dispatch-divergence assertion (fixture 보강)' (#183) from fix/hotfix-15-mcp-ask-multi-hop-flaky into main
Reviewed-on: #183
2026-05-26 07:02:26 +00:00
a210bf5d52 docs(rag): HOTFIX #15 spec + plan (3 round OMC reviewer approve)
OMC team `hotfix-15-mcp-flaky` 의 spec + plan 작성 + 리뷰 산출물.

- spec: analyst 가 진단 (root cause = PR-7 probe-first 가 PR-5 test 의 stale empty-KB contract 와 mismatch) + Option A 권장 (test-only fix). 3 round review (critic + verifier): CRITICAL C1 (fixture/query FTS5 0 hits) + MAJOR M1/M2 + 등 closure.
- plan: planner 가 7 steps + subagent dispatch task 작성. 3 round review (critic-plan + verifier-plan): empirical SQLite REPL 검증, level-1 dated entry placement, actual KebabHandler/KebabAppState pattern 정합.

implementation = 429287f commit (executor).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 06:52:04 +00:00
429287f6cb fix(mcp,tests): HOTFIX #15 — MCP ask multi_hop dispatch-divergence assertion (fixture 보강)
PR-7 (v0.18 dogfood probe-first) 머지 후 PR-5 의 test `ask_tool_routes_multi_hop_true_to_decompose_first` 가 stale empty-KB contract 로 deterministic fail. test-only fix — production code 0 touch.

- `minimal_config`: `score_gate = 0.0` (probe 의 second gate `top_score < score_gate` 우회, test config isolation).
- fixture `workspace_root/note.md`: "This note is about a compound containing X and Y in detail." — build_match_string 의 token_and branch (FTS5 implicit-AND) 가 `compound` + `about` + `and` 셋 다 매칭 필요. empirical SQLite REPL (V007 trigram DDL) 로 1 hit 확정.
- 기존 assertion 보존, single-pass branch 도 query "anything" 으로 fixture 미매칭 → NoChunks refusal 유지.
- 신규 `_multi_hop_short_circuits_when_probe_empty` test (REQUIRED — round-1 critic HIGH + verifier 격상): probe-empty short-circuit 의 MCP-layer wire shape pin (kebab-rag::multi_hop_empty_probe_pool_refuses_before_any_llm_call 은 RAG-layer 만 pin, MCP-layer 안전망 부재).
- module doc 갱신: 두 test 가 각각 pin 하는 contract enumerate. inline 주석 (line 94-101) 도 새 contract 정합.
- HOTFIXES.md 신규 dated entry \`## 2026-05-26 — HOTFIX #15 ...\` (date-top convention).

검증: cargo test --workspace -j 1 — 회귀 0 (known flaky 1 → 0). cargo clippy --workspace --all-targets -j 1 -- -D warnings clean.

Wire / behavior / version cascade: 0.

Refs: docs/superpowers/specs/2026-05-26-hotfix-15-mcp-ask-multi-hop-flaky-spec.md (review 3 rounds APPROVE)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 06:51:06 +00:00
08495eb425 Merge pull request 'chore(release): bump version 0.17.2 → 0.18.0 + cut fb-41 multi-hop' (#182) from chore/v0-18-0-cut into main
Reviewed-on: #182
2026-05-26 05:36:05 +00:00
98cf4e8a04 chore(release): bump version 0.17.2 → 0.18.0 + cut fb-41 multi-hop
v0.18.0 cut PR. fb-41 multi-hop RAG + NLI verification 의 user-visible surface (PR #176-180) + post-PR9 cleanup/refactor (PR #181) ship 마무리.

## 변경 사항

### Version
- workspace `Cargo.toml`: 0.17.2 → 0.18.0. Cargo.lock 자동 cascade (24 kebab-* crate 모두 0.18.0).

### Frozen design contract
- `docs/superpowers/specs/2026-04-27-kebab-final-form-design.md`:
  - §3.8 RAG types — RefusalReason 에 NliVerificationFailed + NliModelUnavailable + MultiHopDecomposeFailed 추가 + Multi-hop RAG + NLI verification 의 ask_multi_hop facade + step 8.5 NLI hook + HopRecord / VerificationSummary 명시.
  - §9 versioning rules 표 — nli_model_version row 신규 (선택 — v0.19+ second adapter 시 wire surface candidate).

### Status transitions
- `docs/superpowers/specs/2026-05-25-p9-fb-41-finalize-spec.md`: status approved-by-team → completed.
- `docs/superpowers/plans/2026-05-25-p9-fb-41-finalize-plan.md`: status approved-by-team → completed (spec_status 도).

### User-facing docs
- `README.md`: 명령 표의 `kebab ask` row 에 `--multi-hop` flag + NLI 옵션 안내 한 단락 (mDeBERTa-v3 XNLI 280 MB 자동 다운로드 / RAM peak ~7-8 GB / threshold tuning 0.5 prod / 0.0 disable).
- `docs/SMOKE.md`: `[rag] nli_threshold = 0.0` config 예시 + 활성화 절차 + first-run download + RAM 권장 inline 안내.

### Handoff + dashboard
- `HANDOFF.md`: 한 줄 요약 의 현재 version 0.17.2 → 0.18.0. v0.18.0 cut entry 추가 (fb-41 multi-hop + NLI + cleanup ship). Component 카운트 단락에 fb-41 PR-9 의 kebab-nli + ask_multi_hop 추가 명시. 머지 후 결정 절 맨 위에 v0.18.0 fb-41 entry 신규.
- `tasks/INDEX.md`: p9-fb-41  머지 (v0.18.0). v0.18.0 subsection 신규 — PR #176-181 의 6 sub-PR + cleanup 각 한 줄 요약.

## 비범위 / 별 작업
- HOTFIXES.md 의 fb-41 entry 는 이미 PR #180 (PR-9d closure) 에서 작성 완료 — 본 cut PR 에서 추가 anchor 불필요.
- SKILL.md 의 v0.18+ NLI 안내는 이미 PR-9c-2 에서 inline 추가 완료.

## 검증
- `cargo check --workspace -j 1` 통과 (모든 24 crate v0.18.0 확인).
- frozen design 의 RefusalReason enum 확장이 kebab-core 의 production code 와 정합 (PR-9c-1 시점부터 동일 variants 있음).

Wire 영향: 없음 (additive minor 는 PR-9c-1 에서 이미 ship, 본 commit 은 documentation cascade only).
Behavior 영향: 없음.

머지 후 `gitea-release v0.18.0` 으로 tag + release notes 작성.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 05:18:08 +00:00
4030f04f37 Merge pull request 'chore: workspace-wide cleanup — clippy::pedantic baseline + auto-fix' (#181) from chore/workspace-wide-cleanup-pre-v0-18 into main
Reviewed-on: #181
2026-05-26 04:48:50 +00:00
7c27633df2 chore(rag): post-PR9 refactor — H1/H2/H3/D/E + test coverage + post-refactor dogfood retest
OMC team `post-pr9-refactor` 의 architectural cleanup. architect priorities 분석 후 executor + test-engineer 가 file edits, system-architect 가 component-level review 로 *pre-cut nothing — all v0.18.1+ defer* 결론.

## Executor 작업 (H1/H2/H3/D/E)

- **H1** (kebab-nli/src/onnx.rs): `[models.nli]` config wire 활성화. `DEFAULT_MODEL_ID` const 제거 (kebab-config 의 NliCfg::defaults 가 single source). OnnxNliVerifier::new 가 config.models.nli.model 읽고 config.models.nli.provider 가 "onnx" 아니면 anyhow::bail. 3 stale "PR-9c-1 will wire this" 코멘트 제거. 2 unit test 추가 (`new_uses_config_model_id`, `new_rejects_unsupported_provider`).
- **H2** (kebab-rag/src/pipeline.rs): `truncate_for_nli(premise: &str, _hypothesis: &str)` → `truncate_for_nli(premise: &str)`. v0.18.1 placeholder doc 제거. 4 callsite (tests/multi_hop.rs) 갱신 + test rename `multi_hop_truncate_for_nli_preserves_hypothesis` → `multi_hop_truncate_for_nli_char_budget` (contract 정합).
- **H3** (kebab-rag/src/pipeline.rs:1041): `was_truncated` 가 tracing::debug! 으로 surface (observability 추가, signature 보존 — caller logging contract).
- **D** (kebab-mcp/tests/tools_call_ask_multi_hop.rs): request_timeout_secs 2 → 5 (slow CI 안정성), `mh_code` discriminator 제거. dispatch contract = `mh.is_error.unwrap_or(false)` (기존 assertion 으로 충분).
- **E** (tasks/HOTFIXES.md + pipeline.rs:1633-1638): fb-41 PR-9 closure entry 의 sibling 으로 "### PR-9 NLI refusal: terminal Synthesize hop omitted from hops trace" subsection 추가. pipeline 의 "cleanup deferred to a follow-up" → "// See tasks/HOTFIXES.md ... for follow-up" cross-link.

## Test-engineer 작업 (T1/T2/T3/T4, 9 new tests)

- **T1** (kebab-nli/src/onnx.rs::tests): sanitize_model_id 3 unit (replaces_slash / idempotent / leaves_other_chars).
- **T2** (kebab-rag/tests/multi_hop_nli_panic.rs 신규): 2 panic-path tests — facade invariant (`expect("verifier must be Some when nli_threshold > 0.0")`) 의 #[should_panic] + threshold=0 의 companion.
- **T3** (kebab-rag/tests/multi_hop_nli_stream.rs 신규): 2 StreamEvent::Final tests — refuse_nli_verification + refuse_nli_model_unavailable 의 stream_sink Final 분기 wire shape pinning.
- **T4** (kebab-app/tests/open_with_config_nli.rs 신규): 2 NLI failure path — model_dir 가 unwritable 일 때 App::open_with_config 의 Result<App> Err (with "OnnxNliVerifier" in chain) + threshold=0 일 때 graceful skip.

## System-architect 결론

3 lenses (absorption / duplication / under-engineered interface) 분석 결과 — *pre-cut nothing*. Top-3 items 모두 v0.18.1+ defer:
- Lens 1: kebab-normalize + kebab-parse-types 흡수 가능 (parse-md 만 사용, 5 parsers 우회) → v0.18.1+.
- Lens 3: Extractor + Chunker trait 의 dead polymorphism (모든 callsite 가 hardcoded) → v0.18.1+.
- Lens 1 bundled: kebab-source-fs 가 kebab-parse-code 의 9 tree-sitter grammars drag → low-risk dep-graph win, v0.18.1+ bundled.
- Defer-with-intent: LanguageModel async refactor (cloud-LLM 시), NliVerifier::score_batch + typed NliError (2nd impl 시), compute_stale → kebab-core::stale.

보고서: /build/cache/tmp/post-pr9-refactor-priorities.md, /build/cache/tmp/system-architecture-priorities.md (둘 다 repo 외 — analysis 보존).

## 검증

- cargo test -p kebab-nli -j 1 → 11/11 pass.
- cargo test -p kebab-rag -j 1 → 75/75 pass (5 NLI multi-hop + 4 신규 T2/T3 포함).
- cargo test -p kebab-app -j 1 → 23 pass + 2 ignored (T4 의 2 포함).
- cargo test -p kebab-mcp --test tools_call_ask_multi_hop -j 1 → 1 pass + 1 pre-existing flaky (HOTFIX #15, no_chunks short-circuit, executor D fix 와 무관 — line 86 의 base assertion 이 fixture 없어서 fail).
- cargo clippy --workspace --all-targets -j 1 -- -D warnings clean.
- cargo test --workspace --no-fail-fast -j 1 → 1304 passed (+11 new) + 1 pre-existing flaky 동일.
- **Post-refactor dogfood retest byte-identical** (PR-9d / post-cleanup / post-refactor 3번 모두): S7 0.0035389824770390987, S1 0.058334656059741974, S10 0.0027875436935573816, S3 nli_model_unavailable.

docs/dogfood/v0.18.0/SUMMARY.md 에 "Post-architectural-refactor retest" section 추가.

Wire 영향: 없음.
Behavior 영향: 없음 (H1 의 config wiring 가 default 와 같은 model → byte-identical).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 04:42:37 +00:00
3712d005cc chore(rag): post-cleanup dogfood retest — byte-identical 회귀 0
workspace-wide cleanup commit 직후 동일 dogfood S7/S1/S10/S3 재실행. NLI score byte-identical 확인:

- S7: nli_verification_failed, 0.0035389824770390987 (PR-9d 동일)
- S1: nli_verification_failed, 0.058334656059741974 (PR-9d 동일)
- S10: nli_verification_failed, 0.0027875436935573816 (PR-9d 동일)
- S3: nli_model_unavailable (PR-9d 동일, cleanup 무관 — v0.18.1 follow-up)

cleanup = mechanical refactor only. behavior 회귀 0. cut PR v0.18.0 진행 가능.

docs/dogfood/v0.18.0/SUMMARY.md 에 "Post-cleanup retest" section 추가.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 03:19:15 +00:00
7c85de065a chore: workspace-wide cleanup — clippy::pedantic baseline + auto-fix
cut PR v0.18.0 전 마지막 정리. 사용자 요청: "전체 코드베이스를 깔끔하고 알아보기 쉽게".

## Workspace lints

- `Cargo.toml` 의 `[workspace.lints.clippy]` 에 `pedantic = "warn"` (priority -1) + 의도적 allow-list 추가:
  - cast_possible_truncation / cast_possible_wrap / cast_sign_loss / cast_precision_loss — ONNX i64 / hash modular reduction 등 의도적 truncation.
  - doc_markdown / missing_errors_doc / missing_panics_doc — cosmetic doc style.
  - too_many_lines / module_name_repetitions / must_use_candidate / needless_pass_by_value / manual_let_else / items_after_statements / similar_names — informational only.
  - format_collect / match_wildcard_for_single_variants / trivially_copy_pass_by_ref / unnecessary_wraps — intentional patterns (exhaustive match, future Result variants 등).
  - default_trait_access — `Foo::default()` 가 idiomatic.
  - float_cmp — NLI / RRF score 의 explicit threshold 비교 의도.
  - struct_excessive_bools / case_sensitive_file_extension_comparisons / naive_bytecount / ignore_without_reason — domain-specific 의도.
  - format_push_string / return_self_not_must_use / match_same_arms — builder / wire-label / hot-path 패턴 보존.
  - needless_continue / used_underscore_binding / nonminimal_bool / unreadable_literal / many_single_char_names / doc_link_with_quotes / assigning_clones / collapsible_str_replace / trivial_regex / elidable_lifetime_names / range_plus_one / explicit_iter_loop / implicit_hasher / ref_option — remaining low-value style.
- 각 24 crate `Cargo.toml` 에 `[lints] workspace = true` 추가.

## Auto-fix

`cargo clippy --workspace --all-targets --fix` 적용 — 128 files changed, 552 insertions / 472 deletions. 주로:
- uninlined_format_args (~18): `format!("{}", x)` → `format!("{x}")`.
- redundant_closure_for_method_calls (~33): `.map(|x| x.foo())` → `.map(T::foo)`.
- 그 외 mechanical refactor.

## 검증

- `cargo clippy --workspace --all-targets -j 1 -- -D warnings` clean (pedantic + 모든 lint group).
- `cargo test --workspace --no-fail-fast -j 1` — **1293 tests pass + 1 pre-existing flaky fail** (`kebab-mcp::tools_call_ask_multi_hop::ask_tool_routes_multi_hop_true_to_decompose_first`, HOTFIX candidate, cleanup 무관). 회귀 0.

Wire 영향: 없음.
Behavior 영향: 없음 (mechanical refactor only).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 03:01:58 +00:00
a0ccc7b021 Merge pull request 'feat(rag): fb-41 PR-9d — dogfood retest + HOTFIXES closure + corpus 보존' (#180) from feat/fb-41-pr-9d-dogfood-retest into main
Reviewed-on: #180
2026-05-26 02:01:56 +00:00
a8fd6994d2 chore(rag): PR-9d SUMMARY 의 latency 표 정정
PR-8 baseline 의 S1/S10 latency 추정값 (~150s, ~80s) 이 부정확. `results/s1-multihop.json` + `results/s10-multihop.json` 가 실제로 614s / 589s (`jq '.usage.latency_ms'` 측정) — *PR-8 시점 baseline 이 아닌 더 이전 timeline*. S7 만 `results/post-pr8/` 에 retest 보존되어 비교 의미 있음 (158s baseline → PR-9 241s with NLI first-run download).

SUMMARY.md 의 latency 표를 정정 — S1/S10 의 *동일 시점 baseline 부재* 명시 + S7 의 단일 비교만 의미 있음 caveat.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 01:47:38 +00:00
505b3889fb feat(rag): fb-41 PR-9d — dogfood retest + HOTFIXES PR-9 closure + docs/dogfood/v0.18.0/ 보존
PR-9 의 진짜 작동 확인 — PR-1~PR-9c-2 머지 후 `/build/cache/dogfood-v018/` corpus 의 S7/S1/S3/S10 multi-hop retest.

핵심 결과: **S7 hallucination root cause 해결 확정**.
- PR-8 baseline: `grounded=true, refusal_reason=null`, **답변=Adam gradient 공식** (caffeine 질문에 무관 hallucination, silent).
- PR-9 retest: `refusal_reason=nli_verification_failed, nli_score=0.0035` (graceful refuse, NLI 가 entailment 0.35% 검출).

전체 비교 (4 case):
- S7  hallucination FIXED.
- S1  둘 다 reject, NLI 가 더 deterministic (0.058).
- S3 ⚠ consistent fail (`nli_model_unavailable`, 313s) — *v0.18.1 follow-up* (kebab-nli 의 특정 input 의존 fail, debug log emit 안 됨 → 진단 어려움).
- S10  둘 다 reject, NLI 가 더 deterministic (0.0028).

- docs/dogfood/v0.18.0/SUMMARY.md (sanitized 보고서) + s{1,3,7,10}-multihop-post-pr9.json (sample wire output, repo 보존).
- tasks/HOTFIXES.md 의 fb-41 PR-9 entry: "예정" → "완료 (2026-05-26)" + 비교 표 inline + S3 follow-up subsection (v0.18.1 candidate).

RAM: 5-6 GB → 7-8 GB (ONNX session ~600 MB), 16 GB 안전.
Disk: NLI model cache 1.1 GB (XDG default 또는 storage.model_dir).
Wire 영향: 없음 (PR-9c-1 의 schema 변경만 + 측정값 sample 보존).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 01:44:57 +00:00
772575d8f0 Merge pull request 'feat(rag): fb-41 PR-9c-2 — pipeline integration + mock test + SKILL.md (★ NLI 실 활성화)' (#179) from feat/fb-41-pr-9c-2-pipeline-integration into main
Reviewed-on: #179
2026-05-26 01:03:18 +00:00
00ffe9c792 feat(rag): fb-41 PR-9c-2 — pipeline integration + mock test + SKILL.md (★ NLI 실 활성화)
PR-9c-1 의 wire surface 위에 behavior 활성화 — `ask_multi_hop` 의 step 8.5 hook 가 `[rag] nli_threshold > 0` 일 때 NLI 검증 실 수행. **첫 user-visible behavior change** in PR-9.

- crates/kebab-rag/src/pipeline.rs:
  - ask_multi_hop step 8.5 NLI hook (empty answer 가드 + truncate_for_nli + verifier.score + verification field + refusal 분기).
  - refuse_nli_verification helper (verification: Some(...) + RefusalReason::NliVerificationFailed).
  - refuse_nli_model_unavailable helper (verification: None + RefusalReason::NliModelUnavailable).
  - truncate_for_nli helper (module-level pub fn, MAX_NLI_PREMISE_CHARS = 4 * 400 = 1600 chars 의 chars-based budget, _hypothesis 미사용 placeholder — v0.18.1 token-budget 갱신 candidate).
  - PR-9c-1 의 #[allow(dead_code)] 두 곳 제거 (verifier field + with_verifier builder; doc 의 transitional sentence 도 정리). round-1 PR-9c-1 review N1 carry-forward closure.
- crates/kebab-app/src/app.rs:
  - App::open_with_config 의 NliVerifier construction — config.rag.nli_threshold > 0 → OnnxNliVerifier::new + Arc::new wrap + 후속 RagPipeline 초기화 시 with_verifier 호출. 실패 시 ? 전파 (시그니처 Result<Self> 그대로 — caller cascading 0).
  - kebab-app/Cargo.toml 에 kebab-nli path 의존 추가.
- crates/kebab-rag/tests/multi_hop.rs + tests/common/mod.rs:
  - MockNliVerifier (pass / fail / err 생성자 + score call_count instrumented).
  - multi_hop_nli_pass_keeps_grounded — entailment 0.9 → grounded=true, verification.nli_passed=true.
  - multi_hop_nli_fail_refuses — entailment 0.1 → refusal=NliVerificationFailed.
  - multi_hop_nli_disabled_skip_verify — threshold 0.0 → verify skip, verification=None.
  - multi_hop_nli_model_unavailable_refuses — verifier Err → refusal=NliModelUnavailable.
  - multi_hop_truncate_for_nli_preserves_hypothesis — long premise truncation + hypothesis 보전.
- integrations/claude-code/kebab/SKILL.md: mcp__kebab__ask 절에 NLI 안내 한 단락 (verification.nli_passed 의미 + threshold tuning + nli_verification_failed/nli_model_unavailable refusal handling).

검증: cargo test --workspace -j 1 — 5 신규 multi-hop pass + 회귀 0 (pre-existing kebab-mcp::tools_call_ask_multi_hop 동일 flaky). cargo clippy --workspace --all-targets -j 1 -- -D warnings clean.
Wire 영향: PR-9c-1 의 schema 변경에 *behavior wiring* — answer.v1.verification field 가 multi-hop happy path + refuse path 양쪽에서 채움.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 00:55:02 +00:00
681c48b2a3 Merge pull request 'feat(rag): fb-41 PR-9c-1 — core types + wire scaffolding (NLI verification)' (#178) from feat/fb-41-pr-9c-1-core-types-wire into main
Reviewed-on: #178
2026-05-26 00:09:03 +00:00
546c1564b0 feat(rag): fb-41 PR-9c-1 — core types + wire scaffolding (NLI verification)
Surface-only PR (no behavior wiring — that's PR-9c-2):
- kebab-core: RefusalReason::NliVerificationFailed + NliModelUnavailable (serde rename_all="snake_case", wire = identical strings).
- kebab-core: Answer.verification: Option<VerificationSummary> field (additive minor wire — pre-v0.18 reader 무영향).
- kebab-core: VerificationSummary { nli_score: f32, nli_threshold: f32, nli_passed: bool } struct + lib.rs 재-export.
- kebab-config: NliCfg { model, provider } + ModelsCfg.nli (default Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7).
- kebab-config: RagCfg.nli_threshold: f32 (default 0.0 = disabled, spec §2.6 single gate).
- kebab-config: env override KEBAB_MODELS_NLI_MODEL/PROVIDER + KEBAB_RAG_NLI_THRESHOLD (parse 실패 시 tracing::warn + default 유지).
- kebab-rag: RagPipeline.verifier: Option<Arc<dyn NliVerifier>> field + with_verifier builder (모두 #[allow(dead_code)] — PR-9c-2 의 step 8.5 hook 가 활성화 시 제거). RagPipeline::new signature 유지 (round-2 NEW-M1 Option B).
- kebab-rag: Cargo.toml 에 kebab-nli path 의존 추가.
- kebab-store-sqlite + kebab-tui: 두 신규 RefusalReason variant 에 대한 exhaustive match arm 추가 (snake_case label / 표시 문구).
- 모든 Answer 구축 site (rag 6 + cli/tui/eval 3 fixture) 에 verification: None 추가.
- wire schemas: answer.schema.json verification field + \$defs.VerificationSummary + refusal_reason.enum 2 추가. error.schema.json code.enum + details.description 2 추가 (forward-looking reserved).
- docs/ARCHITECTURE.md: Mermaid Adapters subgraph 의 nli 노드 + rag→nli + app→nli (forward-looking) + nli→config edges. nli→core edge 는 skip (kebab-nli/Cargo.toml direct dep 가 config 만, ARCHITECTURE 컨벤션 = direct deps only). 디렉토리 트리에 crates/kebab-nli/ 추가.

Tests: kebab-core 3 (serde rename + verification skip + struct shape) + kebab-config 6 (defaults + legacy + env + malformed env) + kebab-cli wire 5 (schema verification + enum 검증).
검증: cargo test --workspace -j 1 회귀 0 (pre-existing kebab-mcp::tools_call_ask_multi_hop flaky 1개 동일 — spec 에 명시된 known-flaky). cargo clippy --workspace --all-targets -D warnings clean.
Wire 영향: additive minor — answer.v1 의 verification optional + refusal_reason.enum 확장 + error.v1.code 확장.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 23:27:36 +00:00
79ad6e376f Merge pull request 'feat(nli): fb-41 PR-9b — OnnxNliVerifier ONNX inference + model download' (#177) from feat/fb-41-pr-9b-onnx-nli-inference into main
Reviewed-on: #177
2026-05-25 22:24:01 +00:00
6ffbe0a5a3 chore(nli): PR #177 회차 1 리뷰 반영 (N1 cache-hit probe + N2 test pollution)
- N1: fetch 의 cache-hit 검사 경로가 실제로는 download 트리거 (ApiRepo::get 가 cache miss 시 download 후 path 반환). log 의 "NLI artifact cache hit" 가 *방금 download 한 직후* 출력 — misleading. hf_hub::Cache::new(cache_dir).repo(repo).get(filename).is_some() 로 변경 — Cache::get 은 fs lookup only, 네트워크 안 탐. actual download 횟수는 변화 없음 (1번), log accuracy 만 개선.
- N2: new_succeeds_on_default_config / score_empty_hypothesis_returns_err 가 XDG 실 디렉토리 (`~/.local/share/kebab/models/nli/...`) 를 create_dir_all → test pollution. tempdir_config() 헬퍼 추가 — TempDir 으로 storage.data_dir override, model_dir 는 `{data_dir}/models` 그대로 두어 expand_path 의 substitution 검증도 유지.

cargo test -p kebab-nli -j 1 → 6 passed / 0 failed (unit) + 5 ignored (integration, manual).
cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings clean.
inference.rs 미수정 → manual --ignored smoke 결과 (5/5 PASS) 그대로 유효.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 22:22:30 +00:00
ab3408cb49 chore(nli): PR-9b inference test 2 의 expectation 정정
기존 expectation `entailment < 0.3` 가 너무 strict — mDeBERTa-v3 multilingual NLI 가 두 caffeine 사실 (premise: "Caffeine is a stimulant.", hypothesis: "The chemical formula of caffeine is C8H10N4O2.") 의 *neutral* 을 0.53 으로, entailment 를 0.43 으로 판단함 (서로 entail 안 하지만 모순도 아님 = 정확히 neutral).

spec §3 PR-9b 의 "entailment 낮음 — neutral/contradiction 이 winning channel" 의 *spirit* 은 *neutral 이 max* 임. expectation 을 `s.neutral > s.entailment && s.neutral > s.contradiction` 로 변경.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 22:10:51 +00:00
b807fd5aa5 feat(nli): fb-41 PR-9b — OnnxNliVerifier 의 ONNX inference + model download
- OnnxNliVerifier fields: model_id, cache_dir (XDG model_dir/nli/<sanitized>), session/tokenizer OnceLock.
- new(): eager cache_dir stamp만 — actual model download + Session::commit_from_file 는 첫 score 호출 시 ensure_loaded() 가 lazy 수행.
- score(): ensure_loaded → tokenizer.encode(pair, OnlyFirst truncation max_length=512) → ndarray Array2<i64> → ort::Session::run → logits[1,3] → NliScores::from_xnli_logits.
- empty hypothesis edge: defense-in-depth bail (spec §2.3 의 caller-side skip 외 추가).
- sanitize_model_id helper: "/" → "_".
- 5 #[ignore] integration tests (EN self-entailment, EN unrelated, KR entailment, long premise truncation, empty hypothesis err) — manual smoke 가 PR description 첨부.

Cargo.toml: `download-binaries` feature 를 kebab-nli 의 ort dep 에 활성화 (PR-9b prep commit 의 후속). 단독 `cargo test -p kebab-nli` 의 per-crate feature 유니온은 fastembed 없이 ort/download-binaries 가 OFF 되어 ort-sys link 가 실패 — kebab-nli 측에서 명시적으로 켜 줘야 standalone build 가 ONNX 런타임 link 됨. workspace 전체 빌드에서는 fastembed 의 동일 opt-in 과 union 되어 부작용 없음.

Verification:
- cargo test -p kebab-nli -j 1 — PR-9a 의 6 unit pass (`score_returns_err_in_skeleton` → `score_empty_hypothesis_returns_err` 로 stub→실 path 갱신, 갯수 유지).
- cargo clippy -p kebab-nli --all-targets -- -D warnings clean.
- cargo build --workspace -j 1 — 회귀 0.
- Manual --ignored smoke 결과 PR body 첨부.

Wire 영향: 없음 (crate-internal).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 21:56:22 +00:00
93436f9eca feat(nli): fb-41 PR-9b prep — activate ort/tokenizers/hf-hub/ndarray/tracing deps in kebab-nli
PR-9a 의 workspace.dependencies 만 declared 였던 5 crate 의존을 kebab-nli/Cargo.toml 에 활성화. PR-9b 의 OnnxNliVerifier 실 구현이 본 commit 위에서 빌드 가능.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 21:42:07 +00:00
11ce7847a1 Merge pull request 'feat(nli): fb-41 PR-9a — kebab-nli crate skeleton + workspace deps' (#176) from feat/fb-41-pr-9a-kebab-nli-crate into main
Reviewed-on: #176
2026-05-25 21:34:49 +00:00
1d88dccf8a chore(nli): PR #176 회차 1 리뷰 반영
- lib.rs::NliScores::faithfulness doc 의 `rag.nli_faithfulness_min` → `rag.nli_threshold` (spec §2.5/§2.6 의 실 config knob 이름 정합).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 21:25:44 +00:00
1eb0bbecb3 feat(nli): fb-41 PR-9a — kebab-nli crate skeleton + workspace deps
- 신규 crate kebab-nli (trait + impl 동일 crate, v0.18 scope = ONNX adapter 1개).
- NliVerifier trait + NliScores struct (XNLI 3-channel: entailment/neutral/contradiction).
- private softmax3 (log-sum-exp 안전).
- OnnxNliVerifier placeholder (PR-9b 가 ONNX inference + model download 추가).
- workspace.dependencies 추가: ort 2.0-rc.9, tokenizers 0.21 (default-features=false, onig), hf-hub 0.4, ndarray 0.16.

Pre-flight (PR-9 design contract 의 gate):
- HF Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 model.onnx + tokenizer.json → HTTP/2 302 (HF S3 routing, file 존재).
- tokenizers --no-default-features -F onig 의 standalone repro: SentencePiece mDeBERTa tokenizer.json 로드 OK (KR 9 tokens / EN 11 tokens 정상 encode).
- Cargo features 결정 trace: tokenizers = { default-features = false, features = ["onig"] } lock.

Tests: 6 unit (softmax3 정규화 + 불변성 + XNLI logits 변환 + faithfulness + new + score stub) — 통과.
Verification: cargo test -p kebab-nli -j 1 (6/6) + cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings clean.
Workspace: cargo test --workspace -j 1 — pre-existing kebab-mcp::tools_call_ask_multi_hop 1 fail (main baseline 동일 fail, PR-9a 무관 — ingest fixture/Ollama 의존 flaky).

Wire 영향: 없음 (crate 도입만).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 21:22:38 +00:00
44fbffff26 docs(rag): fb-41 PR-9 spec + plan — NLI verification + v0.18.0 cut
fb-41 multi-hop RAG 의 dogfood S7 hallucination root cause = LLM-self-judge ceiling.
대응 = NLI-based post-synthesis verification (mDeBERTa-v3 XNLI, 280 MB ONNX).

산출물:
- docs/superpowers/specs/2026-05-25-p9-fb-41-finalize-spec.md (review_round=5,
  4 OMC reviewer APPROVE: 1 CRITICAL + 9 MAJOR + 3 MINOR → 1 NIT carry-forward).
- docs/superpowers/plans/2026-05-25-p9-fb-41-finalize-plan.md (plan_review_round=3,
  4 OMC reviewer APPROVE: 15 issues → 0 actionable).

5 sub-PR (PR-9a~9d) + cut PR. 작업 21-31h / wall time 28-44h.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 21:22:20 +00:00
63aece3ea1 Merge pull request 'fix(rag): fb-41 PR-8 multi-hop synthesize safety in depth (pool 15 + self-check rule)' (#175) from feat/fb-41-pr-8-multi-hop-synthesize-safety into main 2026-05-25 12:51:46 +00:00
28a8bbeace chore(rag): PR #175 회차 1 리뷰 반영
HOTFIXES.md 의 fb-41 entry 에 *post-PR-7 dogfood retest + PR-8 partial
mitigation* sub-section 추가 + *PR-9 NLI plan* anchor + 사용자 영향
절 갱신. config.rs 의 doc reference 가 정확한 entry sub-section
가리키도록 조정 — dangling reference 해소.

검증
- `cargo test -p kebab-config -j 1` — 모든 test 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:51:15 +00:00
52a97303dc fix(rag): fb-41 PR-8 — multi-hop synthesize safety in depth (pool 15 + self-check rule)
v0.18 cut 전 fb-41 multi-hop RAG **layered defense** — PR-7 의 pre-decompose
probe gate 위에 추가 safety. PR-7 의 fix 만으로는 hybrid mode 의 RRF
top_score 가 gate 통과 시 (도그푸딩 S7 의 caffeine query) hallucination
여전히 발생 — synthesize 단계 자체의 safety 보강 필요.

**중요**: 본 PR 만으로는 S7 hallucination 완전 차단 안 됨 (gemma3:4b 의
prompt-following 한계 — 추가 dogfood S7 retest 에서 확인). 진짜 fix 는
PR-9 (NLI-based post-synthesis verification). PR-8 은 그 사이의 *partial
mitigation + safety in depth* — latency 4× 개선 (614s → 158s) + future
larger LLM 용 prompt rule.

설계: docs/superpowers/specs/2026-05-25-p9-fb-41-multi-hop-rag-design.md
계획: /build/cache/dogfood-v018/results/PR-9-DESIGN.md (사용자 결정 후
spec/plan 으로 promotion)

## 변경

- `crates/kebab-config/src/lib.rs`:
  - `RagCfg::multi_hop_max_pool_chunks` default **30 → 15**.
  - rationale doc — gemma3:4b 가 30-chunk large prompt 에서 citation
    rule 잃는 측정 결과.
  - 2 unit test (`default_*` rename + `legacy_*` assert) 갱신.
- `crates/kebab-rag/src/pipeline.rs`:
  - `MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT` 에 **답하기 전 self-check** rule
    추가 — "[원본 질문] 의 핵심 entity (고유명사, 화학식, 수치 단위,
    코드명, 약자) 가 [근거] 본문에 literal 으로 등장하지 않으면 다른
    entity 의 정보로 답을 합성하지 말고 '근거가 부족하다' 답한다". example
    (caffeine + Adam optimizer chunk) 도 명시.

## 도그푸딩 결과 (retest with PR-7 + PR-8)

| query | path | grounded | latency | answer |
|---|---|---|---|---|
| caffeine formula | single-pass | false (LlmSelfJudge) | 30s | "근거가 부족하다" ✓ |
| caffeine formula | multi-hop pre-fix | true ✗ | 141s | hallucination |
| caffeine formula | multi-hop PR-7 | true ✗ | 143s | hallucination (probe gate top_score 0.5 > 0.30) |
| caffeine formula | multi-hop PR-8 | true ✗ | **158s** | hallucination (LLM 가 새 rule 무시) — **latency 4× 개선** |

PR-8 의 부분 성과:
- pool 30→15 로 synthesize prompt size ↓ → latency 614s → 158s.
- prompt rule 은 future larger LLM (gemma2:9b, qwen2.5:7b 등) 에서 가치 ↑.

PR-8 의 한계:
- gemma3:4b 의 prompt-following 한계 — strong rule 도 무시하고 다른 entity
  chunk (Adam optimizer formula) 의 본문을 caffeine 화학식 출처로 인용.
- LLM-self-judge 기반 safety 의 ceiling.

## 진짜 fix → PR-9 (별 PR)

학계 / industry 표준 검색 결과 (Self-RAG, CRAG, Auto-GDA, MedTrust-RAG):
deterministic post-synthesis verification 이 정답 path. **NLI-based
groundedness check** — mDeBERTa-v3-base-xnli (280 MB multilingual) ONNX
model 이 (premise=packed_chunks, hypothesis=answer) entailment 검사. score
< 0.5 면 refuse. PR-8 위에 layered defense.

## 검증

- `cargo test -p kebab-config -p kebab-rag -j 1` — 모든 test 통과
  (config default test 2개 갱신, rag tests 영향 없음).
- `cargo clippy -p kebab-config -p kebab-rag --all-targets -j 1 --
  -D warnings` clean.
- 단일 crate 직렬 build (16 GB RAM 제약).
- S7 dogfood retest — hallucination 여전 (PR 본문에 정직 명시).

## 변경 없음

- Wire schema — additive (config knob default 만 변경).
- PR-7 의 probe gate — 그대로 작동 (gate 통과 시 PR-8 의 추가 safety
  layer).
- 다른 도그푸딩 P1 항목 (citation 일관성, binary path) — 별 PR.

## 다음

- **PR-9a/b/c**: NLI-based post-synthesis verification — 진짜 fix.
- PR-9 머지 후 dogfood S7 재검증 (예상: refuse + nli_score < 0.5).
- v0.18.0 cut.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:44:31 +00:00
71fb2cbcb3 Merge pull request 'fix(rag): fb-41 PR-7 multi-hop pre-decompose score-gate (S7 hallucination 회귀 핀)' (#174) from feat/fb-41-pr-7-multi-hop-score-gate-fix into main 2026-05-25 12:05:23 +00:00
85855ef596 chore(rag): PR #174 회차 1 리뷰 반영
`ask_multi_hop` 의 probe_hits 가 gate 검사 후 throw away 되는 의도
명시 — pool 초기값으로 재사용 안 하는 *invariant clarity* rationale 을
코드 안에 doc. 향후 retrieve cost 가 multi-hop bottleneck 이 될 경우
재검토 hint 도 함께.

검증
- `cargo test -p kebab-rag -j 1 --test multi_hop` 10 모두 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:04:53 +00:00
da25ce330b fix(rag): fb-41 PR-7 — multi-hop pre-decompose score-gate (S7 hallucination 회귀 핀)
v0.18 cut 전 fb-41 multi-hop RAG 도그푸딩에서 발견된 **safety regression**
fix. 자세한 도그푸딩 결과는 `tasks/HOTFIXES.md` 의 2026-05-25 fb-41
pre-v0.18 entry + `/build/cache/dogfood-v018/results/SUMMARY.md` 참조.

## 문제 (S7)

Query: `What is the chemical formula of caffeine?` (KB 에 없는 fact).

- Single-pass `kebab ask`: retrieve top score 가 default `rag.score_gate
  = 0.30` 미만 → `refuse_score_gate` → 안전한 refusal.
- Multi-hop `kebab ask --multi-hop`: **`grounded = true`**, 본문
  `"카페인의 화학식은 C₉H₁₅N₃O 입니다 [#6]"` (hallucination — 실제
  C₈H₁₀N₄O₂) + `[#6]` 가 Adam optimizer chunk 의 `g_t = ∂L/∂θ_i` 본문을
  인용 (시각적 short structured token 매칭 trigger).

원인: `ask_multi_hop` 의 score-gate 검사가 *pool 의 top_score* 만 봤다.
multi-hop 의 pool 은 5 sub-queries 의 union — 한 sub-query 의 top score
가 gate 위면 다른 chunks 가 원본 query 와 무관해도 gate 통과 + synth →
LLM hallucinate.

## Fix

`ask_multi_hop` entry 에 **pre-decompose probe** 추가:

1. *원본 query* 로 retrieve 한 번 (LLM call 0회, ~ms).
2. probe empty → `refuse_no_chunks(None)` (decompose 안 함, hops=None).
3. probe top_score < gate → `refuse_score_gate(None)` (decompose 안 함).
4. probe pass → 기존 decompose / decide / synthesize flow 그대로.

Multi-hop 의 safety floor 가 single-pass 와 정확히 일치 — multi-hop 은
*원본 query 가 이미 KB 범위 내* 일 때만 cross-doc reasoning 추가.

비용: 한 번의 retrieve (수 ms), LLM call 없음. multi-hop 의 LLM-dominated
latency 대비 무시 가능.

## Tests

신규 3 회귀 핀 (`crates/kebab-rag/tests/multi_hop.rs`):

- `multi_hop_below_probe_gate_refuses_before_any_llm_call` — **S7 직접
  회귀 핀**. low-score chunk + empty LM script → score_gate refusal, LM
  calls 0회, hops=None. fix revert 시 즉시 panic.
- `multi_hop_empty_probe_pool_refuses_before_any_llm_call` — empty
  retrieve 시 NoChunks refusal, LM calls 0회.
- `multi_hop_above_probe_gate_proceeds_to_decompose` — probe pass 시
  full multi-hop flow 정상 (decompose + decide + synth).

기존 7 multi-hop test 의 `ScriptedRetriever` 에 *probe-pass entry*
prepend + `retriever_handle.calls()` expectation +1. test 2 / test 4
처럼 entry 두 개였던 곳도 prepend (3 entries).

`multi_hop_refuse_no_chunks_preserves_hops_trace` /
`multi_hop_refuse_score_gate_preserves_hops_trace` 의 의미 좁힘 — 이제
*decompose-driven* refusal (probe pass 후 sub-query retrieve 가 empty
또는 below-gate) 만 검증. *probe-driven* refusal 은 hops=None
(decompose 안 함) — 신규 test 가 그 path 핀.

## 검증

- `cargo test -p kebab-rag -j 1` — 10 multi-hop (7 갱신 + 3 신규) + 19
  pipeline + 31 unit + 3 prompt_template + 3 streaming 모두 통과. 회귀
  없음.
- `cargo clippy -p kebab-rag --all-targets -j 1 -- -D warnings` clean.
- 단일 crate 직렬 build (16 GB RAM 제약).

## 변경 없음

- Wire schema — `Answer.hops` shape 동일, `refusal_reason` enum 동일.
- 다른 도그푸딩 발견 (synthesize citation 일관성, latency, binary path
  confusion) — v0.18.1 또는 별 PR 의 책임. HOTFIXES 의 "다른 도그푸딩
  발견" 절에 명시.

## 다음

PR-7 머지 후:
1. Workspace `Cargo.toml` version 0.17.2 → 0.18.0 (minor bump).
2. HANDOFF.md / INDEX.md 갱신 + frozen design §3.8 multi-hop sub-section.
3. `gitea-release v0.18.0 --auto-notes`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:02:11 +00:00
5bfea3c28b Merge pull request 'feat(tui): fb-41 PR-6 TUI Ask multi-hop toggle + hop trace summary' (#173) from feat/fb-41-pr-6-tui-multi-hop-toggle into main 2026-05-25 09:30:06 +00:00
b6756f8ce3 chore(tui): PR #173 회차 1 리뷰 반영
test `spawn_snapshot_multi_hop_into_askopts` →
`ask_state_multi_hop_field_default_false_and_round_trips` 로 rename.
이전 이름은 spawn 동작 검증을 약속했으나 본문은 단순 field
default + setter round-trip 만 검증 — name 과 실제 의도의 mismatch.
새 이름이 실제 검증 (field shape pin) 과 정확히 일치.

doc string 도 spawn 동작은 별 path (live dogfood) 로 검증된다고
명확히 표기 — test 의 책임 범위가 무엇인지 reader 가 즉시 파악.

검증
- `cargo test -p kebab-tui -j 1 --test ask` — 42 test (6 multi-hop
  포함) 모두 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 09:29:36 +00:00
016f380428 feat(tui): fb-41 PR-6 — TUI Ask multi-hop toggle + hop trace summary
fb-41 multi-hop RAG 의 **마지막 component PR** (PR-5 머지 직후). TUI Ask
패널의 user-facing surface — F2 toggle, multi-hop badge, status panel
의 hop count summary, cheatsheet 안내. v0.18.0 cut 준비.

설계: docs/superpowers/specs/2026-05-25-p9-fb-41-multi-hop-rag-design.md
계획: docs/superpowers/plans/2026-05-25-p9-fb-41-multi-hop-rag.md (PR-6 단락)

## TUI surface

- `crates/kebab-tui/src/app.rs`:
  - `AskState.multi_hop: bool` field + Default false. 사용자 토글
    상태를 인-패널 보존, 대화 history 와 직교 — F2 flipping mid-
    conversation 도 turns 보존 (다음 turn 만 다른 pipeline 으로 route).
- `crates/kebab-tui/src/ask.rs`:
  - `handle_key_ask` 에 `(KeyCode::F(2), _) → s.multi_hop = !s.multi_hop`.
    Mode-agnostic (physical function key — Normal/Insert 양쪽 작동, typing
    ambiguity 없음). Briefing 의 candidate (F2 vs Ctrl-T) 중 F2 채택 —
    Ctrl-M 은 Enter 와 collision 이미 명시, F2 가 cleanest.
  - `spawn_ask_worker` 의 `AskOpts.multi_hop` 가 spawn 시점에 토글값
    snapshot. 이후 F2 flip 은 다음 Enter 부터 적용 (in-flight turn 무영향).
  - `render_input` 의 input pane title 에 `F2=multi-hop` binding 안내
    추가 + prompt row 에 `multi-hop` badge (Success 녹색, toggled-on 일
    때만). 사용자가 어떤 pipeline 으로 다음 query 를 보낼지 항상 가시.
  - `render_status` 의 status panel 에 `multi-hop: N hops` line 추가
    (last_answer.hops 가 Some 일 때만). forced_stop 발생 시
    `forced_stop=K` suffix — depth/pool cap tuning 단서.
- `crates/kebab-tui/src/cheatsheet.rs`:
  - Ask section 에 `F2 toggle multi-hop pipeline` entry 추가.

## 변경 없음 (의도된 deferral)

- `InspectTarget::Hop(turn_index)` variant — plan 의 PR-6 stretch goal.
  per-iter hop trace detail 을 Inspect 패널에 노출하는 기능은 별 PR
  (PR-6b 또는 v0.18 dogfood follow-up). PR-6 의 핵심 가치
  (사용자가 multi-hop pipeline 을 토글하고 결과의 hop count 를 본다)
  는 status panel 의 한 줄 summary 로 100% cover. Inspect 진입은
  multi-hop 사용자가 *드물게* 필요한 surface — v0.18 cut 부담 회피.
- prompt_template_version (`rag-multi-hop-v1`) — 그대로.
- MCP / CLI surface — PR-4 / PR-5 의 책임.

## Tests (`tests/ask.rs` 신규 6 multi-hop pins)

- `f2_toggles_multi_hop_flag_from_insert_mode`: Insert 에서 F2 toggle
  (fresh_app default mode).
- `f2_toggles_multi_hop_flag_from_normal_mode`: Normal 에서도 동일
  — mode-agnostic 회귀 핀.
- `input_pane_shows_multi_hop_badge_when_toggled_on`: 토글 on 시
  prompt row 에 `multi-hop` 등장 + title 의 `F2=multi-hop` binding
  hint 등장.
- `input_pane_omits_multi_hop_badge_when_toggled_off`: 토글 off 시
  prompt row 의 badge 부재 (title hint 는 유지 — 사용자 discoverability).
- `status_panel_summarizes_hops_when_answer_has_trace`: 3-hop trace
  (Decompose + Decide + Synthesize) → `multi-hop: 3 hops` line.
- `status_panel_omits_hops_summary_for_single_pass`: hops=None → 본문
  에 summary line 부재 (title binding hint 만).
- `spawn_snapshot_multi_hop_into_askopts`: AskState.multi_hop 의
  field shape 회귀 핀 (default false / settable / round-trip).

## 검증

- `cargo test -p kebab-tui -j 1` — 신규 6 multi-hop + 기존 ask /
  search / library / mode / cheatsheet / inspect / status_bar 모두
  통과 (42 ask test + 10 mode + 기타). 회귀 없음.
- `cargo clippy -p kebab-tui --all-targets -j 1 -- -D warnings` clean.
- 단일 crate 직렬 build (16 GB RAM 제약).

## v0.18.0 cut (다음 단계)

- Workspace `Cargo.toml` version 0.17.2 → 0.18.0 (minor — surface
  확장 + new prompt_template_version `rag-multi-hop-v1`).
- HANDOFF.md / HOTFIXES.md / INDEX.md 갱신 (fb-41 entry 정리).
- `gitea-release v0.18.0 --auto-notes`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 09:26:29 +00:00
bf28a1e4d9 Merge pull request 'feat(mcp): fb-41 PR-5 MCP ask multi_hop arg + SKILL.md 안내' (#172) from feat/fb-41-pr-5-mcp-multi-hop-arg into main 2026-05-25 09:09:06 +00:00
24221826ed chore(mcp): PR #172 회차 1 리뷰 반영
`ask_tool_routes_multi_hop_true_to_decompose_first` 의 error code
검증을 더 견고하게 — `model_unreachable | timeout` 둘 다 accept.
환경 차이 (즉시 ECONNREFUSED vs connect timeout) 가 다른 wire code
로 분류돼도 dispatch divergence 자체 (schema_version=error.v1 +
isError=true vs single-pass 의 answer.v1 grounded=false) 는 동일하게
검증.

검증
- `cargo test -p kebab-mcp -j 1 --test tools_call_ask_multi_hop` 2 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 09:08:40 +00:00
8a2f7affa6 feat(mcp): fb-41 PR-5 — MCP ask multi_hop arg + SKILL.md 안내
fb-41 multi-hop RAG 의 **PR-5** (PR-4 머지 직후). PR-4 의 CLI `--multi-hop`
flag 와 sister surface — agent (Claude Code 등 MCP host) 가 `mcp__kebab__ask`
호출 시 `multi_hop: true` 옵션 사용 가능.

설계: docs/superpowers/specs/2026-05-25-p9-fb-41-multi-hop-rag-design.md
계획: docs/superpowers/plans/2026-05-25-p9-fb-41-multi-hop-rag.md (PR-5 단락)

## MCP surface

- `crates/kebab-mcp/src/tools/ask.rs`:
  - `AskInput.multi_hop: Option<bool>` 추가. JsonSchema derive 가 tools/list
    에 자동 반영 — agent capability discovery 가 새 필드 인식.
  - `handle()` 가 `AskOpts.multi_hop = input.multi_hop.unwrap_or(false)` —
    기존 caller (필드 누락 / null) 는 single-pass 그대로.
- `crates/kebab-mcp/src/lib.rs` (tools/list):
  - `ask` tool description 에 multi-hop 한 줄 (decompose → retrieve →
    synthesize, 2-5× LLM cost, per-hop trace on Answer.hops).

## SKILL.md 안내

- `integrations/claude-code/kebab/SKILL.md` 의 `mcp__kebab__ask` 절:
  - Input shape JSON 예제에 `multi_hop: false` 추가.
  - Returns 절에 `hops` (multi-hop only) 추가.
  - 신규 bullet (p9-fb-41) — opt-in 조건 / 비용 trade-off / 사용 케이스
    (compound questions / prereq chains / cross-doc reasoning) /
    `Answer.hops` 의 per-hop trace shape / `multi_hop_decompose_failed`
    refusal 처리.

## Tests (`tests/tools_call_ask_multi_hop.rs` 신규, 2 Ollama-free pins)

- `ask_tool_routes_multi_hop_true_to_decompose_first`: dispatch
  divergence 핀. invalid LLM endpoint (`http://127.0.0.1:1`,
  request_timeout_secs=2) 로 force unreachable. multi_hop=true 는
  decompose 먼저 호출 → `error.v1` (code=model_unreachable) /
  isError=true. multi_hop=false (single-pass) 는 empty KB 에서 retrieve
  먼저 → no LLM call → `answer.v1` grounded=false / isError=false. 두
  shape 의 분기가 dispatch 가 실제로 다른 path 로 라우팅됨의 증거.
- `ask_input_schema_advertises_multi_hop_field`: AskInput 의 JsonSchema
  가 `multi_hop` property 노출 — MCP host capability discovery
  (tools/list 의 input schema) 회귀 핀.

기존 `tools_call_ask.rs` 의 AskInput literal 도 `multi_hop: None`
추가 (struct field 추가에 따른 minimal cascade).

## 변경 없음

- `prompt_template_version` (`rag-multi-hop-v1`) — 그대로.
- TUI surface — PR-6 의 책임.
- error.v1 매핑 — PR-4 의 enum reservation 그대로 (no error_wire
  promotion).

## 검증

- `cargo test -p kebab-mcp -j 1` — 신규 tools_call_ask_multi_hop 2 +
  기존 ask / search / bulk_search / fetch / ingest / schema / doctor /
  tools_list / initialize 등 모두 통과 (회귀 없음).
- `cargo clippy -p kebab-mcp --all-targets -j 1 -- -D warnings` clean.
- 단일 crate 직렬 build (16 GB RAM 제약).

## 다음 PR

- PR-6: TUI Ask 패널 multi-hop toggle (F2 / Ctrl-T) + hop trace render +
  cheatsheet 갱신.
- v0.18.0 cut (PR-6 머지 후).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 09:06:28 +00:00
f28a422f79 Merge pull request 'feat(cli): fb-41 PR-4 CLI --multi-hop flag + answer.v1 / error.v1 wire' (#171) from feat/fb-41-pr-4-cli-multi-hop-flag into main 2026-05-25 08:48:08 +00:00
c56242d04f chore(cli): PR #171 회차 1 리뷰 반영
`answer.schema.json` 의 `refusal_reason` description 의 PR 번호 정정:
`multi_hop_decompose_failed` 도입 시점 = PR-2 (#167, RefusalReason variant
+ ask_multi_hop decompose-failure 분기). PR-3a (#168) 는 `Answer.hops`
field + RagCfg knob 만 — refusal variant 와 무관.

검증
- `cargo test -p kebab-cli -j 1 --test wire_ask_multi_hop` 4 모두 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 08:47:35 +00:00
17c48a0ee6 feat(cli): fb-41 PR-4 — CLI --multi-hop flag + answer.v1 / error.v1 wire 확장
fb-41 multi-hop RAG 의 **PR-4** (PR-3b-ii 의 ScriptedLm + tests 위에서
user-facing CLI surface + JSON Schema 확장). PR-3b-i / PR-3b-ii 의 multi-hop
pipeline 을 `kebab ask --multi-hop` 으로 사용자에게 노출.

설계: docs/superpowers/specs/2026-05-25-p9-fb-41-multi-hop-rag-design.md
계획: docs/superpowers/plans/2026-05-25-p9-fb-41-multi-hop-rag.md (PR-4 단락)

## CLI surface

- `kebab ask --multi-hop <query>` — 새 flag (default false). `AskOpts.multi_hop`
  로 전달, stream + non-stream 두 callsite 모두 갱신.
- `--show-citations` / `--hide-citations` / `--stream` / `--session` 등 기존
  flag 와 orthogonal.
- `--json` 모드에서 `Answer.hops` 배열이 multi-hop happy path / refusal-with-
  partial-trace 양쪽 경로에서 노출됨 (PR-3b-i + PR-3b-ii 의 wiring).

## Wire schema 확장

- `docs/wire-schema/v1/answer.schema.json`:
  - 신규 `hops: array | null` 필드 (optional, additive). `HopRecord` 의
    `$defs` 추가 — `iter` / `kind` (decompose|decide|synthesize) /
    `sub_queries` / `context_chunks_added` / `forced_stop` / `llm_call_ms`
    6 필드 + per-field doc.
  - `refusal_reason` 필드를 `anyOf [enum, null]` 로 명시 — 6 variant
    (`score_gate`, `llm_self_judge`, `no_index`, `no_chunks`,
    `llm_stream_aborted`, `multi_hop_decompose_failed`). 이전 schema 는
    `type: string|null` 만 명시 → enum 명시는 agent / consumer 의 strict
    validate 강화 (additive — 기존 producer 값 모두 enum 안).
  - `$id` / `schema_version` 변경 없음 — additive minor.
- `docs/wire-schema/v1/error.schema.json`:
  - `code` enum 에 `multi_hop_decompose_failed` 추가. **이는 forward-looking
    enum extension** — 현재 RefusalReason 은 `Answer.refusal_reason` (stdout)
    으로만 노출되고 `error.v1` (stderr) 경로 안 거침. 미래 PR 에서 fatal
    promotion 정책 결정 시 trigger 가능하도록 enum 만 미리 reserve.
  - details.description 의 per-code 안내에 `multi_hop_decompose_failed: {}`
    note 추가 — reserved 상태 명시.

## Tests

- `crates/kebab-cli/tests/wire_ask_multi_hop.rs` 신규 (4 Ollama-free pins):
  - `cli_ask_help_advertises_multi_hop_flag`: clap-level smoke, `kebab ask
    --help` 출력에 `--multi-hop` 등장 확인.
  - `answer_schema_declares_hops_property_with_hop_record_defs`: `hops`
    property 존재 + `$defs.HopRecord` 의 `kind` enum 3 variant
    (decompose/decide/synthesize) 회귀 핀.
  - `answer_schema_refusal_reason_enum_includes_multi_hop_decompose_failed`:
    6 variant 모두 enum 에 존재 — 기존 5 도 함께 핀 (회귀 방지).
  - `error_schema_code_enum_includes_multi_hop_decompose_failed`: 신규
    code enum 확장 + 기존 code (config_invalid / not_indexed / ...) 보존 핀.

End-to-end multi-hop ask 의 live Ollama 검증은 후속 `#[ignore]` test 로
(같은 `wire_ask_stale.rs` 패턴). PR-4 의 범위 = clap + schema 정합성 만.

## 변경 없음

- `crates/kebab-app/src/error_wire.rs` — plan 의 "error_wire 매핑" 항목은
  현재 RefusalReason 가 `Answer.refusal_reason` 로만 노출 (anyhow chain 안
  거침) 라 trigger 가 없음. enum reservation 만으로 충분, 매핑 코드는 dead
  code 회피. 향후 fatal-promotion 정책 (refusal → error.v1) 결정 시 PR-4b
  로 split.
- `prompt_template_version` — `rag-multi-hop-v1` 그대로.
- TUI / MCP surface — PR-5 / PR-6 에서.

## 검증

- `cargo test -p kebab-cli -j 1` — 모든 test 통과 (신규 wire_ask_multi_hop 4 +
  기존 ask / search / schema / ingest / mcp / reset 등 모두).
- `cargo clippy -p kebab-cli --all-targets -j 1 -- -D warnings` clean.
- 단일 crate 직렬 build (16 GB RAM 제약).

## 다음 PR

- PR-5: MCP `ask` tool 의 `multi_hop: bool` argument + `integrations/claude-
  code/kebab/SKILL.md` 의 ask 절 갱신.
- PR-6: TUI Ask 패널 multi-hop toggle (F2 / Ctrl-T) + hop trace render.
- v0.18.0 cut (PR-6 머지 후): `Cargo.toml` 0.17.2 → 0.18.0 + HANDOFF /
  HOTFIXES / INDEX 갱신 + gitea-release.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 08:45:01 +00:00
64a009314c Merge pull request 'feat(rag): fb-41 PR-3b-ii ScriptedLm + multi-hop tests + refusal hop trace' (#170) from feat/fb-41-pr-3b-ii-scripted-lm-tests into main 2026-05-25 08:25:42 +00:00
ddfe7ba099 chore(rag): PR #170 회차 2 리뷰 반영
test 7 의 `i32_below_gate_chunk` helper rename → `seed_low_score_chunk` +
반환 shape 을 `(chunk_id, doc_id)` tuple 로 확장. `i32` prefix 가 Rust
integer 타입과 충돌하던 가독성 문제 해소 + 호출자가 `id32("d_low")` 를
재계산하지 않도록 id 페어를 single source of truth 로 통합.

검증
- `cargo test -p kebab-rag -j 1 --test multi_hop` — 7 모두 통과.
- `cargo clippy -p kebab-rag --all-targets -j 1 -- -D warnings` clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 08:24:36 +00:00
104363a0db chore(rag): PR #170 회차 1 리뷰 반영
(A) ScriptedLm doc 의 `Arc<Vec<String>>` 표기 → 실제 구현 (`Vec<String>` +
    `AtomicUsize`, 외부에서 `Arc::new(ScriptedLm::new(...))` 로 wrap)
    반영.
(B) ScriptedLm::new doc 의 미존재 `with_*` builder 언급 제거.
(C) refuse path 의 hops 보존 회귀 핀 2 건 추가 (`tests/multi_hop.rs`):
    - `multi_hop_refuse_no_chunks_preserves_hops_trace`: empty pool →
      `refuse_no_chunks(Some(hops))` → Answer.hops = Some([Decompose,
      Decide]).
    - `multi_hop_refuse_score_gate_preserves_hops_trace`: top score 0.10
      < 0.30 gate → `refuse_score_gate(Some(hops))` → 같은 shape.
    refuse_* widening + ask_multi_hop 의 forwarding wiring 이 reverting
    되면 두 test 가 회귀 잡음.
(D) test 5 의 redundant `assert_ne!(.., Some(MultiHopDecomposeFailed))`
    제거 — `assert_eq!(.., None)` 이미 함의. 메시지에 의도 통합.

검증
- `cargo test -p kebab-rag -j 1 --test multi_hop` — 7 (5+2) 모두 통과.
- `cargo clippy -p kebab-rag --all-targets -j 1 -- -D warnings` clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 08:22:58 +00:00
6188a50c1c feat(rag): fb-41 PR-3b-ii — ScriptedLm + 5 multi-hop tests + refusal hop trace + carry-over
PR-3b 의 분할 두 번째 PR — PR-3b-i 의 dynamic decide loop 위에서:

1. **ScriptedLm + ScriptedRetriever helper** (kebab-rag tests/common/mod.rs)
   per-call 다른 response 반환. decompose / decide×N / synthesize 의 각
   LLM call 을 구분하는 다단계 multi-hop 시나리오를 mock-only 로 exercise
   가능. `Vec<&str>` / `Vec<Vec<SearchHit>>` 받아 call sequence 순서대로
   emit. Send + Sync.

2. **5 multi-hop integration tests** (kebab-rag tests/multi_hop.rs 신규)
   - decide_stop_triggers_synthesize: decide [] → 즉시 synthesize
   - decide_continue_adds_more_chunks: decide ["q2"] → iter 2 retrieve + pool 확장
   - max_depth_force_stops: depth cap → forced_stop + decide LLM call skip
   - pool_chunks_dedup_by_chunk_id: 같은 chunk_id 두 sub-query 에서 1 회
   - decide_parse_failure_falls_through_to_synthesize: parse fail = graceful
     synthesize (refusal 아님, spec §9)

3. **refuse_* helper hops trace 보존** (회차 1 carry-over)
   refuse_no_chunks / refuse_score_gate 시그니처에 `hops:
   Option<Vec<HopRecord>>` 인자 추가. ask_multi_hop 의 score-gate /
   no-chunks refusal 시 누적된 hops 그대로 Answer.hops 에 보존.
   single-pass ask 는 None 전달 — wire 변동 없음 (skip_serializing_if).

4. **HopRecord doc 보강** (회차 1 carry-over)
   sub_queries 의 per-kind 의미 명시 (Decompose=initial / Decide=next-iter
   or empty=stop / Synthesize=always empty). llm_call_ms=0 의 ambiguity
   (no call vs 0ms call) doc 명시.

5. **MULTI_HOP_MAX_SUB_QUERIES_DEFAULT → _HARD_CAP rename** (회차 1 carry-over)
   const 의 의도 명확화 — config knob `multi_hop_max_sub_queries_per_iter`
   (5, prompt-side soft hint) 와 const (10, parse-side hard ceiling)
   분리. 두 layer 의 책임 doc 동기화. test 도 rename.

6. **decide guard 단순화 + preview budget doc** (회차 1 carry-over)
   parse_decompose_response 의 post-condition (Some=non-empty 보장)
   doc 명시. defensive `Some(qs) if !qs.is_empty()` →
   `decide_result.unwrap_or_default()` 단순화. decide preview 의
   snippet-only path (full chunk text 안 fetch) 의도 doc.

검증
- `cargo test -p kebab-rag -j 1` — 31 unit + 19 pipeline + 5 multi_hop
  + 3 prompt_template + 3 streaming 모두 통과.
- `cargo clippy -p kebab-rag --all-targets -j 1 -- -D warnings` clean.

Spec / plan
- design: docs/superpowers/specs/2026-05-25-p9-fb-41-multi-hop-rag-design.md
- plan: docs/superpowers/plans/2026-05-25-p9-fb-41-multi-hop-rag.md (PR-3b 단락)

다음 단계 = PR-4 (CLI --multi-hop + wire schema + error_wire).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 08:17:37 +00:00
94e6146013 Merge pull request 'feat(rag): fb-41 PR-3b-i dynamic decide loop + helpers' (#169) from feat/fb-41-pr-3b-decide-loop into main 2026-05-25 07:32:25 +00:00
12c7dc9efb feat(rag): fb-41 PR-3b-i — dynamic decide loop + helpers + format! named arg
PR-3b 의 분할 첫 PR. ask_multi_hop 의 fixed depth=2 → dynamic N-hop.
ScriptedLm helper + 5+ integration tests (happy-path 통합 검증) 는
PR-3b-ii 분리. 본 PR 의 회귀 핀 = 기존 PR-2 의 2 integration test
통과 (decompose garbage refusal + multi_hop=false single-pass keep).

- `RagPipeline::multi_hop_decompose` 시그니처 변경 — `Result<
  (Option<Vec<String>>, u32)>` (parsed result + LLM call latency_ms).
  caller (`ask_multi_hop`) 가 hop trace 의 `llm_call_ms` stamp.
- `RagPipeline::multi_hop_decide` helper 신규. decide LLM call →
  `parse_decompose_response` 으로 `Option<Vec<String>>` 반환. None
  또는 empty array 가 stop signal (refusal 아닌 graceful degrade).
- `MULTI_HOP_DECIDE_SYSTEM_PROMPT` const 신규.
- `MULTI_HOP_DECOMPOSE_USER_TEMPLATE` const 제거 + `format!` named
  arg 사용 (PR-2 회차 1 carry-over fix). compile-time substitution
  check — 사용자 query 안에 `{max_sub_queries}` literal 있어도
  mis-replace 회피.
- `ask_multi_hop` 의 §1 (Decompose) + §2 (Retrieve) 영역을 dynamic
  loop 으로 재작성:
  - iter 0 = decompose, HopRecord 추가 (kind=Decompose).
  - iter 1..=max_depth = retrieve current_sub_queries → pool dedup
    → decide LLM call (forced_stop / pool_cap_hit 시 skip).
    HopRecord 추가 (kind=Decide, sub_queries=new_sub_queries,
    context_chunks_added, forced_stop, llm_call_ms).
  - `max_pool_chunks` 도달 시 `pool_cap_hit = true` → 그 iter 의
    HopRecord 가 `forced_stop = true` + decide LLM call skip.
  - depth 도달 (`iter >= max_depth`) 시 동일하게 forced_stop.
  - decide parse failure 또는 empty array → loop break (early
    synthesize, NOT refusal — spec §9 graceful degrade).
- §6 (Generate) 시작 시 `synthesize_started: Instant::now()` 별
  stamp → §8 Build Answer 직전 `HopRecord { kind=Synthesize,
  llm_call_ms = synth_ms }` 추가. happy path 의 Answer literal
  `hops: Some(hops)` 채움 (`hops: None` → `Some(...)` 변경).
- doc comment 갱신: "PR-2 scope (fixed depth=2)" → "PR-3b-i
  scope (dynamic N-hop)". refusal path 의 hops trace 손실 caveat
  명시 (PR-3b-ii / follow-up 에서 helper signature 확장 시 해결).

기존 회귀 핀 (PR-2 의 2 integration test):
- `ask_multi_hop_dispatches_and_decompose_garbage_refuses`:
  decompose garbage → RefusalReason::MultiHopDecomposeFailed +
  정확히 1 LLM call. PR-3b-i 의 시그니처 변경 후도 통과.
- `ask_with_multi_hop_false_keeps_single_pass_path`: 영향 없음.

56 unit + integration test 모두 통과 (kebab-rag).

Wire 영향: `Answer.hops` 가 multi-hop happy path 에서 emit. JSON
Schema additionalProperties default `true` 라 wire breaking 아님
(PR-3a 의 review 확인). schema.json 명시 갱신은 별 PR
(PR-3b-ii 또는 PR-4 의 schema sweep).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 07:29:46 +00:00
cd1d4fb807 Merge pull request 'feat(rag): fb-41 PR-3a HopRecord wire + RagCfg multi-hop knobs' (#168) from feat/fb-41-pr-3-dynamic-decide-loop into main 2026-05-25 07:18:27 +00:00
7150c376bb feat(rag): fb-41 PR-3a — HopRecord wire + RagCfg multi-hop knobs
PR-3 의 분할 첫 PR. wire additive (HopRecord + HopKind + Answer.hops
field) + RagCfg 의 multi_hop_* 3 노브. RAG pipeline 동작 미변경 —
모든 Answer literal 의 `hops = None`. PR-3b (후속) 가 ask_multi_hop
의 happy path 에서 dynamic decide loop 구현 + hops trace 채움.

분할 이유: 원래 PR-3 가 wire + cfg + decide loop + ScriptedLm +
helper refactor + 5+ tests 단일 PR 였는데 ~1500 줄 단일 patch 가
review 부담 + 회기 위험 ↑. additive foundation 부터 ship 후 decide
loop 별 PR — 사용자 결정 (2026-05-25).

- `kebab_core::HopRecord` (iter, kind, sub_queries,
  context_chunks_added, forced_stop, llm_call_ms) + `HopKind`
  (Decompose / Decide / Synthesize) — wire-additive shape.
- `kebab_core::Answer.hops: Option<Vec<HopRecord>>` —
  `#[serde(default, skip_serializing_if = "Option::is_none")]`,
  single-pass / refusal path 는 None, PR-3b 의 multi-hop happy
  path 가 Some.
- `kebab_config::RagCfg` 에 3 신규 노브:
  - `multi_hop_max_depth: u32` (default 3)
  - `multi_hop_max_sub_queries_per_iter: u32` (default 5)
  - `multi_hop_max_pool_chunks: u32` (default 30)
  3 모두 `#[serde(default)]` + env override
  (`KEBAB_RAG_MULTI_HOP_MAX_*`) + legacy parse 핀
  (`LEGACY_PRE_TIMEOUT_TOML` 공유).
- 9 Answer literal site (pipeline.rs ×6 + kebab-cli + kebab-tui
  tests + kebab-eval test) 에 `hops: None` 명시 추가. exhaustive
  field check 가 자동 guard — 빠진 site 시 compile fail.
- plan 의 PR-3 단락 → PR-3a / PR-3b 분할 명시 + scope 정정.

Tests (163 passing across kebab-config + kebab-core + kebab-rag):
- 5 신규 multi-hop knob test (default / env override / legacy parse).
- 기존 50+57+31+19+3+3 test 모두 hops:None 추가 후도 통과.

Wire 영향: `answer.v1` 의 optional `hops` 필드 — `skip_serializing_
if = None` 이라 single-pass response 에 emit 안 됨. wire breaking
아님, JSON Schema 갱신은 PR-3b 또는 PR-4 (실제 emit 시점).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 07:15:01 +00:00
6280abf2df Merge pull request 'feat(rag): fb-41 PR-2 ask_multi_hop skeleton (fixed depth=2)' (#167) from feat/fb-41-pr-2-ask-multi-hop-skeleton into main 2026-05-25 06:50:02 +00:00
192da45dbf chore(rag): PR #167 회차 1 리뷰 반영
- `parse_decompose_response_drops_partial_empty_keeps_valid` 신규
  회귀 핀 — `["", "valid q", "  "]` → `["valid q"]` (trim+filter
  chain 동작 pin).
- `multi_hop_decompose` 의 `stop: Vec::new()` 옆 doc comment 추가 —
  의도 명시 (instruction-following 모델 기대 + prose 추가 시
  MultiHopDecomposeFailed refusal 가 policy). 회차 1 question 의
  답변.
- plan 의 PR-3 implementation order 에 회차 1 carry-over 추가:
  1) ask + ask_multi_hop 의 §4-§9 mirror → 공통 helper 추출,
  2) decompose template 의 substitution corner case → format!
     named arg 으로 교체.

회차 1 의 다른 suggestion (mirror refactor, substitution corner
case, history block helper) 는 PR-3 합리적 timing 으로 plan 에
명시 — 회차 2 reply 에 정리.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 06:49:21 +00:00
cf35f36f88 feat(rag): fb-41 PR-2 — RagPipeline::ask_multi_hop skeleton (fixed depth=2)
PR-2 of fb-41 multi-hop RAG. Decompose + retrieve + synthesize 3-stage
pipeline가 `opts.multi_hop=true` 일 때 dispatch. Dynamic decide loop
는 PR-3.

- `AskOpts.multi_hop: bool` 필드 추가 + `impl Default for AskOpts`
  도입 (HOTFIXES 2026-05-07 의 known limitation 해소). 9 explicit
  init site 모두 `multi_hop: false` 추가 — Default 도입으로 향후
  `..Default::default()` 점진 migrate 가능.
- `RagPipeline::ask` 의 entry 에 dispatcher 한 줄
  (`if opts.multi_hop { return self.ask_multi_hop(...) }`).
- `RagPipeline::ask_multi_hop` 신규 method. 1) decompose LLM call
  → JSON array of strings parse, 2) 각 sub-query 로 retrieve +
  chunk_id dedup pool, 3) score gate / no-chunks 가드, 4)
  pack_context (single-pass 와 helper 공유), 5) synthesize LLM
  call w/ MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT, 6) citation extract
  + Answer build. `prompt_template_version` = "rag-multi-hop-v1"
  로 stamp — eval `compare` 가 single-pass vs multi-hop 분리.
- Prompt const 신규: MULTI_HOP_DECOMPOSE_SYSTEM_PROMPT +
  MULTI_HOP_DECOMPOSE_USER_TEMPLATE + MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT
  + PROMPT_TEMPLATE_VERSION_MULTI_HOP + MULTI_HOP_MAX_SUB_QUERIES_DEFAULT.
- `kebab_core::RefusalReason::MultiHopDecomposeFailed` variant 신규.
  Cascade: kebab-store-sqlite `refusal_reason_label` + kebab-tui `ask
  refusal render` exhaustive match 갱신.
- `parse_decompose_response` + `strip_markdown_json_fence` helper —
  markdown code fence (```json / ```) strip + JSON array of strings
  parse + trim + drop empty + cap at MULTI_HOP_MAX_SUB_QUERIES_DEFAULT.
  None 반환 시 caller 가 `MultiHopDecomposeFailed` refusal.

Tests (55 passing total, 8 신규):
- 6 unit (parse_decompose_response 의 bare array / fence variants /
  garbage / cap / trim 회귀 핀).
- 2 integration: `ask_multi_hop_dispatches_and_decompose_garbage_refuses`
  (decompose garbage → MultiHopDecomposeFailed + 정확히 1 LLM call) +
  `ask_with_multi_hop_false_keeps_single_pass_path` (회귀 핀, 기존
  caller 자동 backwards-compat).

Happy-path multi-hop (decompose 성공 → synthesize) 의 integration
test 는 ScriptedLm helper 가 PR-3 의 decide loop 와 함께 도입될
때 같이 추가. 현 `MockLanguageModel` 는 canned single response 라
2-LLM-call sequence 핀 불가.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 06:45:32 +00:00
ed34f2e03f Merge pull request 'feat(eval): fb-41 multi-hop golden set + spec/plan' (#166) from feat/fb-41-multi-hop-eval-golden into main 2026-05-25 06:27:06 +00:00
624b44c46b chore(eval): PR #166 회차 1 리뷰 반영
- `mh-s-004` 의 `must_contain: ["i"]` 한 글자 → `["INSERT", "i 입력모드"]`
  보강. trigram 0-hit + noise 매칭 위험 해소.
- 3 question 영어 변경 (`mh-c-005` / `mh-i-001` / `mh-s-002`) — fixture
  의 lang 다양성 mix (12 ko + 3 en). 영어 dogfood 시 measurement gap
  회피.
- plan 의 PR-1 단락이 outdated (kebab-eval crate 미survey 단계 작성 →
  실제 PR 와 deviation). actual 변경 명시 + 초안 대비 deviation 명시.

회차 1 의 다른 2 suggestion (mh-c-002 의 `v0.17.2` hard-coded, 15
question / 5-per-bucket 회귀 핀의 frozen size) 은 baseline anchor 의도
적 freeze — 회차 2 reply 에 명시.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 06:26:15 +00:00
caf690dc72 feat(eval): fb-41 multi-hop golden set + spec/plan
PR-1 of fb-41 multi-hop RAG (spec: docs/superpowers/specs/2026-05-25-
p9-fb-41-multi-hop-rag-design.md, plan: docs/superpowers/plans/2026-
05-25-p9-fb-41-multi-hop-rag.md).

XL 작업의 첫 PR — baseline 측정 anchor 만 추가. RAG pipeline 미변경,
fixture file + parse 회귀 핀.

사용자 결정 4 axis (2026-05-25):
- approach: query decomposition (LLM 서브-질문)
- trigger: explicit `--multi-hop` flag
- MVP scope: dynamic N-hop (LLM 이 depth 결정, decompose seed +
  ReAct-style decide loop hybrid)
- eval: multi-hop golden set 먼저 (본 PR)

본 PR:
- `fixtures/multi_hop_golden.yaml` 신규. 15 question (5 cross-doc +
  5 intra-doc + 5 single-fact negative). 기존 `GoldenQuery` struct
  그대로 사용 — 별 loader / type 변경 없음. `expected_chunk_ids`
  비어 있어 curator 가 `kebab ingest` 후 채울 수 있는 template
  형태. `must_contain` 으로 baseline 측정 가능 (P5-2 metric).
- `crates/kebab-eval/tests/loader.rs::loads_multi_hop_golden_fixture`
  신규 회귀 핀. fixture parse OK + 15 question + 5/5/5 bucket
  분포 + 모든 question 에 must_contain 최소 1 개.

baseline 측정 protocol (별 run, commit 에 artifact 안 포함):
1. v0.17.2 binary 로 single-pass `kebab eval run --fixture
   multi_hop_golden.yaml` 실행
2. P@5, P@10, must_contain pass rate, citation_coverage 캡처
3. PR-3 (dynamic iter 머지) 후 동일 fixture + `multi_hop=true` 로
   재실행 → Δ 비교

PR 분할 6 단계 (plan 참조): PR-1 (본 PR — fixture only), PR-2
(RagPipeline::ask_multi_hop fixed depth=2), PR-3 (dynamic iter),
PR-4 (CLI flag + wire), PR-5 (MCP + SKILL.md), PR-6 (TUI toggle +
trace render). 마지막 PR 후 v0.18.0 cut.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 06:22:08 +00:00
1640ecf288 chore: bump version 0.17.1 → 0.17.2
v0.17.1 post-dogfood polish cut. 두 PR 묶어 release:

- PR #164 — `[image.ocr] request_timeout_secs` 별 노브 (v0.17.1
  미진행 closure). LLM 패턴을 OCR 어댑터에 동일 적용, 별 노브로
  분리 (OCR vs LLM 의 cold start 패턴 차이로 독립 조절).
- PR #165 — `heading_path` FTS5 column filter 로 text-only 매칭
  + raw-mode escape hatch (2026-05-24 v0.17.0 trigram entry 의
  JSON 노이즈 closure). lexical.rs 가 non-raw 분기 결과를
  `text : (<expr>)` 로 wrap, 색인 자체는 V007 verbatim 그대로
  유지. raw mode `'heading_path : <token>'` 로 opt-in 가능.

둘 다 additive (옛 config 호환) + re-ingest 불필요. binary 교체만.

HANDOFF 한 줄 요약 + 머지 후 결정 절에 v0.17.2 entry 추가.
HOTFIXES 의 두 entry anchor 가 `post-v0.17.1 dogfood` → `v0.17.2`
로 갱신.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 05:55:50 +00:00
90e77631a8 Merge pull request 'feat(search): heading_path FTS5 text column filter' (#165) from feat/heading-text-column-filter into main 2026-05-25 05:48:22 +00:00
fa251db48f chore(search): PR #165 회차 2 리뷰 반영
HOTFIXES entry 의 **MCP / agent 가시성** 단락이 회차 1 의 SKILL.md
추가 결정과 contradiction (`별도 SKILL.md 갱신 불필요` 잘못된
표기). 갱신 사실 + 새 escape hatch 가 v0.17.0 raw mode pattern
위에 build 됐다는 점 명시.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 05:45:41 +00:00
3114c31841 chore(search): PR #165 회차 1 리뷰 반영
- HOTFIXES test 카운트 표기 정정: `9 신규 / 갱신 unit test` 의 산수
  ambiguity → `9 unit test (8 갱신 + 1 신규) + 2 신규 통합 test = 11
  total` 로 명시.
- SKILL.md (Claude Code integration) 의 search 절에 column scoping +
  heading_path raw-mode escape hatch 안내 한 bullet 추가. 회차 1
  의 follow-up suggestion 반영 — heading 검색 의도 agent 가 새
  escape hatch `'heading_path : <token>'` 를 발견 가능.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 05:44:21 +00:00
271329efbd feat(search): heading_path FTS5 text column filter (default text-only matching)
v0.17.0 trigram tokenizer entry 가 미수정으로 남겨둔
heading_path_json JSON 노이즈 (HOTFIXES 2026-05-24) closure.
trigram 이 chunks_fts.heading_path 컬럼 (V002/V007 트리거가
chunks.heading_path_json 그대로 INSERT) 의 JSON 표기 + 안의 path
세그먼트 (app, src) 까지 3-gram 색인해서 query 가 우연히 false
positive hit 하는 문제. column filter 채택 — heading 색인 유지
(V007 verbatim 불변), 매칭 대상만 text 컬럼 한정.

- build_match_string 가 non-raw 분기에서 combined expression 을
  `text : (<expr>)` 로 wrap. FTS5 column filter syntax 가 OR/AND
  sub-expression 허용.
- Raw mode (`'...'`) 는 그대로 — 사용자가 명시 의도로
  `'heading_path : agent'` 같은 explicit opt-in 가능 (escape hatch).
- 8 기존 build_match_string unit test expected string 갱신 +
  `build_match_string_raw_mode_preserves_heading_filter` 신규.
- `lexical_heading_only_token_does_not_hit_default_mode` 신규 회귀 핀
  (heading-only unique token 이 default mode 에서 0 hit).
- `lexical_raw_mode_can_opt_into_heading_path_filter` 신규 — 같은
  fixture 가 raw mode 로 hit 확인 (escape hatch 동작 핀).

사용자 영향: lexical / hybrid 검색의 본문 precision ↑. recall
변화 없음 (text 본문 token 매칭은 동일). re-ingest 불필요 (FTS
query 시점 매칭만 변경). lexical_snapshot_run_1 + hybrid_snapshot
도 fixture regenerate 불필요 (text 본문 매칭 query 라 BM25 동일).

HOTFIXES: 2026-05-24 v0.17.0 entry 의 `heading_path_json` 노이즈
항목 closure 표기 + 새 2026-05-25 post-v0.17.1 dogfood entry 추가.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 05:40:51 +00:00
f2867540d2 Merge pull request 'feat(ocr): request_timeout_secs config knob' (#164) from feat/ocr-timeout-config into main 2026-05-25 05:14:27 +00:00
e118844256 chore(ocr): PR #164 회차 1 리뷰 반영
- HOTFIXES 헤더 `v0.17.2` (vaporware) → `post-v0.17.1 dogfood`
  로 변경, release tag 결정과 무관하게 정확한 anchor.
- HOTFIXES caller 수 `6 (5+3)` → `9 call site (6+3)` 으로 정정.
- OcrCfg.request_timeout_secs doc 의 edge case 가 LlmCfg sister
  doc 과 동일한 구체 예제 (`u64::MAX`, `86400`) + reqwest 0.12.x
  명시 주석으로 강화.
- LLM + OCR 양쪽의 legacy TOML fixture (78 줄 거의 동일) 를
  module-level `LEGACY_PRE_TIMEOUT_TOML` const 로 추출. 두 test
  가 동일 source 공유 → 옛 schema 가 또 변하면 한 곳만 수정.

reqwest::Duration::ZERO fact-check (회차 1 점 5) 는 회차 2
reply 에서 검증 결과 보고.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 05:13:09 +00:00
41c5edc517 feat(image.ocr): request_timeout_secs config knob + closure of v0.17.1 미진행
v0.17.1 (PR #162) 가 LLM 쪽 hard-coded 300s 를 [models.llm]
request_timeout_secs 로 풀어준 것과 같은 패턴을 OCR 어댑터에 적용.
사용자 결정으로 별 노브 분리 ([image.ocr] request_timeout_secs) —
OCR 는 LLM 대비 cold start 패턴이 달라 독립 조절이 편함.

- OcrCfg.request_timeout_secs: u64 (serde default 300)
- KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS env override
- OllamaVisionOcr::build / from_parts 시그니처에 timeout 인자 추가
- REQUEST_TIMEOUT 상수 제거
- 3 신규 unit test (default / env / legacy parse) — LlmCfg 패턴 그대로
- HOTFIXES 2026-05-25 v0.17.1 entry 의 두 미진행 항목 모두 closure
  (OCR timeout = 본 PR, --stream docs = PR #163 에서 이미 완료)

기존 config / 옛 KB 영향 없음 — 새 필드는 default 로 채워지고
동작도 동일 (300s). vision 모델 cold start 가 길면 env 또는
config 로 늘릴 수 있음.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 05:06:53 +00:00
d02149c010 docs(v0.17.1): HANDOFF + INDEX — v0.17.1 cut sync
- HANDOFF 한 줄 요약 v0.17.0 → v0.17.1 + release URL 추가
  (v0.17.1 cut: PR #162 + #163 한 묶음 안내).
- 머지 후 발견 deviation 절: 2026-05-25 v0.17.1 entry 추가.
- INDEX P10 Dogfooding Feedback section 하단에 'v0.17.1
  post-dogfood polish' subsection 추가 (PR #162, #163 각 한 줄).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 03:35:58 +00:00
0c69b9621b chore: bump version 0.17.0 → 0.17.1
v0.17.1 patch release — v0.17.0 post-dogfood follow-up 두 PR 머지 후.

- PR #162: [models.llm] request_timeout_secs config + 권장 모델 가이드
- PR #163: sudo 없이 ollama 설치 가이드 + kebab ask --stream UX 권장

둘 다 additive only (config field) + docs only — wire breaking 없음,
기존 사용자 영향 없음. patch bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 03:34:12 +00:00
0d69d85757 Merge pull request 'docs: sudo 없이 ollama 설치 + ask --stream 권장 (v0.17.0 post-dogfood)' (#163) from docs/ollama-install-and-stream into main
Reviewed-on: #163
2026-05-25 03:26:24 +00:00
a67300317b docs(ollama): sudo 없이 설치 가이드 + ask --stream 권장 (v0.17.0 post-dogfood)
확장 도그푸딩에서 사용된 두 패턴을 README + SMOKE 에 옮김.

(1) sudo / systemd 없이 격리 디렉토리에 ollama 설치 — tarball 받아
    /opt/ollama/{bin,models,logs} 같은 사용자 디렉토리에 풀고
    OLLAMA_MODELS env 로 모델 위치 분리. 컨테이너 / WSL2 / 회사
    머신 등 root 권한 제약 환경에 유용. 도그푸딩 머신에서
    /build/cache/ollama 로 같은 패턴 검증.

(2) cold start 가 긴 모델 (8B+ 또는 첫 호출) 은 `kebab ask --stream`
    권장 — 동일 inference 시간이라도 progressive 토큰이 5분 timeout
    한도 안에서 빠르게 surface 됨. p9-fb-33 의 streaming 경로를
    UX 개선 권고로 명시.

코드 변경 없음 — docs only. README + SMOKE 두 군데 동일 패턴
sub-bullet + bash snippet.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 03:23:35 +00:00
abb05ebc23 Merge pull request 'feat: [models.llm] request_timeout_secs config + 권장 모델 가이드' (#162) from feat/llm-timeout-config into main
Reviewed-on: #162
2026-05-25 03:21:19 +00:00
26fdc4f344 docs(llm-timeout): 0-as-disable 함정 명시 + HOTFIXES typo + 용어 정리
PR #162 워커 리뷰 반영.

- MEDIUM (W2) + LOW (W1): request_timeout_secs = 0 이 reqwest 의
  의미상 disable 이 아닌 instant timeout (모든 요청 즉시 실패).
  LlmCfg field rustdoc + ollama.rs module-level comment + README
  세 군데에 명시 + u64::MAX / 86400 같은 large finite 값 권장.
- NIT (W1): HOTFIXES 2026-05-25 entry 의 '답변이 인 5분' typo →
  '답변이 5분' (1자 삭제).
- NIT (W1): README + HOTFIXES 의 '확장 도그푸딩' 내부 jargon →
  '후속 도그푸딩' 으로 통일.

코드 동작 변경 없음 — doc only. cargo test request_timeout 3 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 03:14:41 +00:00
3f5e0e6e90 feat(llm): [models.llm] request_timeout_secs config + 권장 모델 가이드
v0.17.0 확장 도그푸딩 (2026-05-25) 에서 발견된 두 가지를
한 PR 에 묶음.

(1) llm.generate_stream 의 hard-coded 300s timeout 을 config 노브로
    빼냄. 8B+ 모델 (gemma4:e4b 등) 은 CPU only 환경에서 5분
    안에 첫 RAG 답변 못 마치고 `error: kb-rag: llm.generate_stream`
    으로 떨어지던 문제.

    - kebab-config::LlmCfg 에 request_timeout_secs: u64 additive
      필드 (#[serde(default = "default_llm_request_timeout_secs")]
      default 300). 옛 config 가 키 누락해도 그대로 파싱 + 동일
      동작.
    - env override KEBAB_MODELS_LLM_REQUEST_TIMEOUT_SECS.
    - kebab-llm-local::ollama.rs 의 REQUEST_TIMEOUT 상수 제거 →
      OllamaLanguageModel::new 가 Duration::from_secs(
      llm.request_timeout_secs) 로 reqwest client 빌드. doc
      comment 도 동일 갱신.
    - 신규 unit test 3 — default 300 핀 / env override / legacy
      config (필드 누락) backward-compat.

(2) docs — README 사전 요구 절 + docs/SMOKE.md ollama 안내에 한 단락:
    CPU only / RAM ≤ 16 GB 환경 ⇒ ≤ 4B Q4 모델 권장
    (gemma3:4b / qwen2.5:3b / phi3:mini). 8B+ 시도 시 timeout
    패턴 사전 안내. request_timeout_secs 노브 사용법.

    HOTFIXES 2026-05-25 entry — 위 두 변경 + 미진행 사항
    (kebab-parse-image OCR 의 같은 hard-coded 300s 는 scope 외
    follow-up 으로 등재 + ask --stream 권장 강조 후속) 기록.

workspace cargo test -j 1 + clippy 통과. 코드 변경은 backwards-compat
(additive serde field) 라 기존 사용자 영향 없음.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 03:01:03 +00:00
578a60e3bb docs(v0.17.0): HANDOFF — version + PR-A/B/C closure entries (R1)
- 한 줄 요약 v0.16.1 → v0.17.0 + release notes URL + PR-A/B/C
  한 줄 요약.
- 머지 후 발견 deviation 절: PR-A 외 PR-B / PR-C 의 2026-05-24
  closure entry 추가.
- '다음 task 후보' 의 P10 round 2 follow-up 라인: 세 항목 모두
  v0.17.0 closure 표시.
- 'P10 dogfooding 백로그' 의 chunk_breakdown + C typedef 두
  항목도  v0.17.0 closure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:55:47 +00:00
64f518e08e docs(v0.17.0): HANDOFF + INDEX — v0.17.0 cut sync (R1)
- HANDOFF 한 줄 요약 v0.16.1 → v0.17.0, release notes URL,
  PR-A/B/C 셋 한 줄 요약. 머지 후 발견 deviation 절에 PR-B / PR-C
  closure entry 추가. "다음 task 후보" + "P10 백로그" 의 세 항목
   v0.17.0 closure 표시.
- INDEX 의 P10 섹션 하단에 신규 "P10 Dogfooding Feedback (v0.17.0)"
  subsection — PR-A/B/C 3 항목 listup (Gemini round 2 권장 형식).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:54:39 +00:00
fa9f91ead4 chore: bump version 0.16.1 → 0.17.0
v0.17.0 release cut — PR-A (한국어 trigram FTS tokenizer + lexical
builder + hint surface) + PR-B (C typedef alias unit + parser_version
cascade + orphan purge) + PR-C (code_lang_chunk_breakdown additive
wire field) 셋 머지 후.

Breaking changes:
- V007 migration (chunks_fts unicode61 → trigram) — chunks 원본 /
  embedding / vector 불변, FTS shadow 자동 backfill. 사용자는 다음
  open 시 V007 즉시 적용 (re-ingest 불필요). kebab.sqlite 파일 크기
  ~2-5배 또는 수백 MB 증가.
- 영어 lexical 검색이 substring 매칭으로 동작 변경 (token →
  tokenization/tokenizer 도 hit, recall ↑ / 단어 경계 ↓).
- C parser_version code-c-v1 → code-c-v2 (typedef alias 추출
  cascade). 같은 file 의 옛 doc/chunks/vector 는 same-workspace_path
  orphan purge 가 자동 정리.

Additive (backwards-compat):
- SearchResponse.hint additive field — 한국어 2자 query 등 trigram
  비호환 시 안내.
- schema.v1.stats.code_lang_chunk_breakdown additive field — chunk
  단위 언어별 분포.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:52:14 +00:00
9ee89c2a94 Merge pull request 'feat: v0.17.0 PR-C — code_lang_chunk_breakdown additive wire field' (#161) from feat/code-lang-chunk-breakdown into main
Reviewed-on: #161
2026-05-24 20:35:28 +00:00
13a3361ba2 docs(v0.17.0/PR-C): rustdoc — code_lang_breakdown / repo_breakdown 가
실제로 doc count 임을 명시 (PR #161 워커 리뷰 MEDIUM 반영)

JSON schema description 은 PR-C 본체에서 'code chunk count' →
'doc count' 로 정정했으나 Rust struct field 의 rustdoc 은 같은
오기재를 그대로 carry — Gemini round 2 가 JSON schema 만 봤고
rustdoc 은 miss. 워커 둘 다 동일 finding (MEDIUM).

implementation 변경 없음 — 의미가 doc count 였던 사실이 처음부터
일관. wording 만 맞춤.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:35:01 +00:00
0def913abd feat(v0.17.0/PR-C): code_lang_chunk_breakdown additive wire field
closure of HOTFIXES 2026-05-22 "code_lang_breakdown chunk granularity"
LOW. Chunk-level companion of the existing doc-count metric.

- crates/kebab-store-sqlite/src/store.rs: code_lang_chunk_breakdown()
  method. chunks INNER JOIN documents → COUNT(c.chunk_id) GROUP BY
  metadata_json.code_lang, NULL skipped. BTreeMap<String, u32>.
  + lib unit test code_lang_chunk_breakdown_counts_chunks_not_docs
  (1 rust doc + 3 chunks → rust=3 chunks vs rust=1 doc).
- crates/kebab-app/src/schema.rs: Stats.code_lang_chunk_breakdown
  additive field + collect_stats builder. tests_stats_ext 의
  stats_includes_code_lang_and_repo_breakdown_fields 가 신규 필드도
  검증.
- docs/wire-schema/v1/schema.schema.json: 신규 additive 필드
  명세 + 기존 code_lang_breakdown / repo_breakdown description
  정정 ("code chunk count" → "doc count", Gemini round 2 권고).
- tasks/HOTFIXES.md: 2026-05-24 PR-C closure entry.

wire additive, schema_version bump 불필요. v0.16.x 호출 호환.
cargo test --workspace --no-fail-fast -j 1 + clippy 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:35:01 +00:00
ff9d5f5f86 Merge pull request 'feat: v0.17.0 PR-B — C typedef-wrapped struct/enum/union → typedef alias unit' (#160) from feat/c-typedef-struct-unit into main
Reviewed-on: #160
2026-05-24 20:33:15 +00:00
70a5068c0d docs(v0.17.0/PR-B/B2): HOTFIXES 2026-05-24 closure + p10-1d Risks 갱신
- tasks/HOTFIXES.md: 새 2026-05-24 PR-B closure entry — extractor 의
  type_definition 분기, PARSER_VERSION bump, same-workspace_path
  orphan purge, 사용자 영향, 잔여 nested typedef Risks.
- tasks/HOTFIXES.md: 기존 2026-05-21 typedef 항목의 Status / Next step
  을 v0.17.0 closure 표현으로 갱신 (관찰 기록은 frozen 유지).
- tasks/p10/p10-1d-c-cpp-ast-chunker.md: Risks 의 typedef idiom 라인
  을 closure  + 잔여 nested typedef 안내로 갱신.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:32:36 +00:00
93ddece111 feat(v0.17.0/PR-B/B1): C typedef extractor + parser_version bump + orphan purge cascade
closure of HOTFIXES 2026-05-21. C typedef-wrapped anonymous
struct/enum/union 이 typedef alias 이름으로 symbol unit 방출.

- crates/kebab-parse-code/src/c.rs: type_definition 분기 추가.
  inner anonymous struct_specifier / enum_specifier / union_specifier
  탐지 → declarator field 의 type_identifier 재귀 추출 → synthetic
  unit (typedef alias). named inner aggregate / plain alias 는
  기존대로 glue. PARSER_VERSION code-c-v1 → code-c-v2.
  recover_typedef_alias + extract_typedef_alias_name helper 추가.

- crates/kebab-store-sqlite/src/store.rs: 두 helper 신규
  (parser_version bump cascade 용 doc-id 기반 orphan purge).
  - stale_chunk_ids_for_workspace_path_except_doc_id(workspace_path,
    keep_doc_id) — sister of stale_chunk_ids_at, doc_id 기반.
  - purge_document_at_workspace_path_except_doc_id(workspace_path,
    keep_doc_id) — CASCADE document/chunks 제거, assets 보존.
  keep_doc_id="" 가 "모든 doc 제거" 사용.

- crates/kebab-app/src/lib.rs: try_skip_unchanged 의 parser_mismatch
  분기에서 purge_workspace_path_for_parser_bump 호출. helper 가
  app.vector() 로 lazy 접근 + delete_by_chunk_ids + SQLite document
  row 제거. Ok(None) 반환 전 cleanup 끝나서 caller 의 새 INSERT 시
  idx_docs_workspace_path UNIQUE 충돌 회피.

- tests:
  - c.rs unit tests 4 신규 — typedef_struct_emits_unit /
    typedef_enum_emits_unit / typedef_union_emits_unit /
    typedef_to_existing_type_stays_glue (negative).
  - tier1_c_ingest_searchable: parser_version assertion code-c-v1 →
    code-c-v2.
- 회귀: bytes-edit 경로 (asset_id 변경) 의 기존 purge_orphan_at_workspace_path
  + purge_vector_orphans_for_workspace_path 는 그대로 — 신규 분기와
  공존, 기존 test 모두 PASS.

미해결 (Risks): nested typedef (typedef struct { struct {...} inner; }
Outer;) 의 inner 익명 struct 는 여전히 glue — v2 의 1차 범위는
top-level typedef alias 만.

cargo test --workspace --no-fail-fast -j 1 + clippy 통과.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 20:30:57 +00:00
67559fb3ce Merge pull request 'feat: v0.17.0 한국어 trigram FTS tokenizer + lexical builder + hint surface' (#159) from feat/korean-trigram-tokenizer into main
Reviewed-on: #159
2026-05-24 20:29:00 +00:00
d79e432916 test(v0.17.0/A5): CLI hint surface e2e coverage (worker-1 nit)
PR #159 worker-1 review 의 LOW 가독성 nit 반영 — CLI stderr [hint]
line + --json hint shape 통합 test 가 없었음.

- search_plain_emits_short_query_hint_to_stderr — 빈 KB + 2자 query
  → stderr 가 "[hint]" + "3자 이상" 포함 확인.
- search_json_emits_hint_field_for_short_query — 동일 입력 --json
  → search_response.v1.hint 필드 set + 표준 advisory 문자열 정합.
- search_json_omits_hint_field_when_query_is_long_enough — 3자
  query → hint 필드 absent (additive serializer 의 None 제외 동작).

wire_search_response 5 → 8 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:45:11 +00:00
0ee18149e7 test(v0.17.0/A5 follow-up): trigram tokenizer downstream test fixes
trigram tokenizer 가 snippet 단위 + 단어 경계 + BM25 raw score 분포를
모두 바꿔서 unicode61 assumption 기반의 3 test 가 regression.

- wire_search_response::search_json_truncates_with_max_tokens +
  search_plain_emits_truncated_hint_to_stderr: 단일 doc + 작은
  max_tokens 로는 snippet 이 짧아서 budget loop 가 trip 안 함.
  다중 doc fixture (5 doc) + budget 30 token 으로 hit-pop 경로
  통해 truncated=true 보장.
- fetch_integration::fetch_chunk_with_context_returns_neighbors:
  fixture body 의 2-char tokens (A1/A3 등) 가 trigram 비호환으로
  0-hit. apples/banana/cherry/durian/elder 5-char unique words
  로 갱신, query 도 cherry 로 deterministic pin.
- eval/runner::runner_per_query_snapshot_matches_fixture: trigram
  token stream 으로 BM25 raw score 변동. UPDATE_SNAPSHOTS=1 로
  regenerate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 12:21:34 +00:00
8a68289499 docs(v0.17.0/A6): HANDOFF + HOTFIXES + README + SMOKE + SKILL — 한국어 trigram closure
- HOTFIXES: 새 2026-05-24 절 — v0.17.0 closure 영향 (한국어
  lexical 3-gram, 영어 substring 변경, BM25 분포, 디스크 용량,
  heading_path JSON 노이즈 관찰). 기존 2026-05-22 한국어 lexical
  항목의 Status / Next step 을 closure 표현으로 갱신.
- HANDOFF: 머지 후 발견 deviation 절에 2026-05-24 entry +
  기존 2026-05-22 항목을 closure cross-link 로 정리. P10
  백로그 한국어 tokenizer 항목  v0.17.0 + "다음 task 후보"
  follow-up 라인의 상태 갱신.
- README: 검색 명령 행에 trigram 동작 + hint + 디스크 용량 한 줄.
- SMOKE: 새 "한국어 trigram 검색 (v0.17.0)" 절 — 도그푸딩 query
  시퀀스 (충돌은 raw / 해시 충돌 multi-token / Rust 충돌은
  mixed / 충돌 2자 + stderr / --json hint 검증) + 영어 substring
  동작 변경 안내.
- SKILL.md: search 절에 hint 필드 안내 한 줄 — agent 가
  short query 케이스에서 같은 query 재시도 대신 사용자에게
  surface 하도록.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 11:54:44 +00:00
6ac7fea7b9 feat(v0.17.0/A5): trigram-aware build_match_string + SearchResponse.hint
PR-A 본체. plan Task A4 Step 1c + A5.

- lexical.rs::build_match_string 재설계: whole-phrase + token-AND
  OR-combined, 3자 미만 토큰 drop, 후보 없음 시 None (빈 MATCH
  회피). raw single-quote mode 유지.
- SearchResponse.hint additive — empty result + trimmed < 3 chars
  + non-raw 케이스에 short_query_hint helper 가 set.
- CLI 'kebab search' 가 [hint] stderr 한 줄 (text mode).
- TUI SearchState.short_query_hint + poll_worker stale-aware set
  + fire_search/mark_input_changed reset + dynamic_status 표시.
- docs/wire-schema/v1/search_response.schema.json hint additive.
- 신규 unit tests (lexical 9 PASS, 기존 2 expectation 갱신) +
  통합 회귀 (search_korean: multi_token + mixed, 3 PASS) +
  BM25 snapshot regen (trigram token stream).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-24 11:54:25 +00:00
fe123c0c6d test(A4): korean + english trigram matching at FTS level
3개 신규 unit tests in tests/fts.rs §7:

1. fts_trigram_korean_3char_substring_hits — Codex sqlite 3.45.1 검증
   동작 5개 assert pin: raw 3자 substring hit (충돌은/발생한),
   quoted phrase hit (\"해시 충돌\"/\"시 충\"), raw 해시충 0-hit (원문
   미존재).
2. fts_trigram_korean_short_query_zero_hit_pinned — 2자 한국어 query
   (충돌·키) 0-hit 회귀 감지. trigram 구조 변경 시 먼저 fail.
3. fts_trigram_english_substring_hits — substring recall 동작 변경
   pin (token→tokenizer, to 0-hit).

검증: cargo test -p kebab-store-sqlite --test fts → 13/13 PASS
(신규 3 + 기존 10).

Step 1c (multi-token 한국어 query e.g. \"해시 충돌\") 와 Step 5
(lexical BM25 snapshot 갱신) 는 Task A5 의 build_match_string()
재설계 후 진행.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:57:37 +00:00
753b1ff5e5 task(A4-step0): synthetic korean fixture for trigram tests
도그푸딩 실 한국어 위키 문서 (hash-table.md, 4512줄 mediawiki HTML,
CC-BY-SA) 는 크기·라이선스 부담으로 직접 commit 회피. 대신 도그푸딩
query 들 (해시 충돌·충돌은·시 충·해시충·충돌) 을 모두 cover 하는 합성
fixture 작성. trigram tokenizer 의 정확한 매칭 동작 (3자 substring
hit, 2자 0-hit, raw vs quoted phrase) 검증용.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:54:30 +00:00
8dcedc4b11 feat(p10-r2): V007 trigram migration + design §5.5 + fts diff-check
Task A2 + A3 한 묶음.

migrations/V007__fts_trigram.sql 신규:
- chunks_fts shadow 를 DROP + 재생성 (tokenize = trigram).
- chunks_ai/ad/au trigger 재생성 (V002 와 동일).
- chunks 에서 backfill INSERT — 사용자 re-ingest 불필요, V007 자동.
- V002 는 historical cold-upgrade replay 위해 그대로 유지.

design §5.5 갱신:
- verbatim block 의 tokenize 만 trigram 으로 교체.
- §5.5 본문 상단에 한국어 채택 사유 + trade-off (영어 lexical 변경,
  BM25 분포, 디스크 ~2-10x, contentless 아님) prose 한 단락 추가.

crates/kebab-store-sqlite/tests/fts.rs:
- fts_v002_matches_design_section_5_5_verbatim →
  fts_v007_matches_design_section_5_5_verbatim 으로 rename.
- extract_migration_5_5_verbatim_block() 의 include_str! path 를
  V007__fts_trigram.sql 로 변경. 주석/assertion msg V007 로.
- V002 cold-upgrade test 들 (fts_v002_backfill_*) 은 그대로 유지.

검증: cargo test -p kebab-store-sqlite --test fts → 10/10 PASS
(`fts_v007_matches_design_section_5_5_verbatim` 포함).

Codex round 1/2 의 design §5.5 contentless 정정·trigram tokenizer
채택 사유 명시 발견 반영.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:52:40 +00:00
8781c6112b task(A1): builder baseline + sqlite version + snapshot locations
Task A1 step 1-3 완료. plan A5 의 baseline 노트 슬롯 채움.

핵심 발견:
- build_match_string() (lexical.rs:177-200): trim → strip_single_quotes
  raw FTS verbatim / 그 외 whitespace split + escape_fts5_token (\"...\"
  + inner doubling) + space join (implicit AND).
- raw mode = single quote '...' 가 trimmed 전체 감쌈 (lexical.rs:167).
- SQLite: rusqlite 0.32 + libsqlite3-sys 0.30.1 bundled (in-tree, SQLite
  ~3.46.x) → trigram 사용 가능.
- Snapshot: tests/lexical.rs::lexical_snapshot_run_1 + tests/hybrid.rs::
  hybrid_snapshot_run_1 (KEBAB_UPDATE_SNAPSHOTS=1 로 regenerate).
  inline normalize_bm25_top_score 는 numerical 무관.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:47:24 +00:00
14197b5e02 docs(p10-round-2): HANDOFF + HOTFIXES sync for v0.17.0 follow-up
P10 도그푸딩 round 2 의 follow-up 후보를 HANDOFF "다음 task" /
"P10 백로그" 절에 반영. HOTFIXES 의 round 2 항목 (한국어 lexical
한계 + code_lang_breakdown + ranking deferred) 정합.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:43:31 +00:00
584247f1ea spec+plan(v0.17.0): korean trigram tokenizer + dogfood fixes
P10 도그푸딩 round 2 (2026-05-22) follow-up. SQLite FTS5 tokenizer
unicode61 → trigram 으로 교체해 한국어 lexical 검색 지원 + 작은
버그픽스 2 (C typedef-wrapped struct 미노출, code_lang_breakdown
집계 단위).

Codex + Gemini round 1/2/3 리뷰 반영:
- [r1] 2자 한국어 query 0-hit, build_match_string() multi-token 깨짐,
  contentless → shadow, parser_version cascade, BM25/heading_path/디스크
- [r2] same-workspace_path orphan purge (parser bump cascade 실제 동작),
  trigram 테스트 예시 sqlite 3.45.1 검증, builder 권장안 (whole phrase OR)
- [r3] SMOKE 시나리오 정정, TUI stale hint 방지, search_response.v1 hint
  필드, new purge helpers, single quote raw mode 통일, fixture 도입

PR 구성: PR-A (trigram + builder + 안내), PR-B (C typedef + orphan
purge), PR-C (stats + wire). 셋 머지 후 v0.17.0 release cut.

design: docs/superpowers/specs/2026-05-22-korean-trigram-tokenizer-design.md
plan:   docs/superpowers/plans/2026-05-22-korean-trigram-tokenizer.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 00:43:31 +00:00
a0c0dca321 fix(dogfood): k8s multi-resource YAML chunk_id collision (#158) 2026-05-21 23:57:49 +00:00
667495ae6a docs(dogfood): HOTFIXES entry for k8s multi-resource chunk_id collision
PR #158 code-reviewer recommendation. Records the dogfood-discovered
k8s multi-resource chunk_id collision + the deliberate decision NOT to
bump chunker_version (dogfood-only stage, single-resource k8s chunk_id
shift is benign churn). Cross-link added to p10-2 spec Risks/notes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 23:57:34 +00:00
08d72a12e0 chore: bump version 0.16.0 → 0.16.1 (k8s multi-resource chunk_id fix)
Patch bump — bug fix only (P10 dogfood-discovered k8s multi-resource
chunk_id collision). New binary needed to resume dogfooding. No wire
schema change, no DB migration.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 23:54:33 +00:00
1969c8e3b5 fix(dogfood): k8s multi-resource YAML chunk_id collision
P10 dogfooding found that a k8s manifest with 2+ documents (e.g.
Deployment + Service in one file) fails to ingest:
  UNIQUE constraint failed: chunks.chunk_id

Root cause: tier2_shared::push_chunks_with_oversize's non-oversize branch
hardcoded split_key = None. K8sManifestResourceV1Chunker calls it once per
resource; with split_key None every resource from the same document gets
the same id_hash (= base_policy_hash) → identical chunk_id. p10-3's
code_text_paragraph_v1 had the same bug (fixed in df3c5b8) but it calls
build_chunk_no_symbol directly — the push_chunks_with_oversize path was
never fixed.

Fix: push_chunks_with_oversize gains a base_split_key parameter for the
non-oversize single-chunk case. k8s chunker passes Some(resource.line_start)
so each resource gets a distinct chunk_id; dockerfile / manifest pass None
(1 chunk per file — no sibling collision, chunk_id stays stable).

Regression coverage: k8s_multi_doc_emits_one_chunk_per_resource now asserts
chunk_id distinctness; new integration test
tier2_k8s_multi_resource_yaml_ingests_without_collision ingests a real
2-document YAML end-to-end.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-21 23:49:37 +00:00
c6207d196e chore(p10-1d-followup): reviewer nit cleanup — C extractor tests + HOTFIXES + cpp snapshot (#157) 2026-05-21 22:47:38 +00:00
840c6c40a6 test(p10-1d-followup): cpp snapshot exercises actual CppAstExtractor
Reviewer nit #3: the hand-built fixed_doc() only verified chunker 1:1
mapping. New tests invoke CppAstExtractor against tests/fixtures/sample.cpp
and snapshot the real extractor → chunker pipeline (14 blocks emitted
covering namespace::chunk::Class, ctor/dtor/operator/template/free-fn
convention, glue <top-level> blocks between units).

Adds kebab-parse-code as a dev-dep of kebab-chunk (same precedent as
kebab-parse-md). Both the existing hand-built test AND the new
extractor-driven tests are kept — the former for fast chunker-only
validation, the latter for end-to-end regression detection.

Added tests:
- code_cpp_ast_extractor_snapshot: asserts all 8 named symbol units are present
- code_cpp_ast_extractor_chunks_deterministic: chunker output is stable

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:43:57 +00:00
b81574afa9 docs(p10-1d-followup): HOTFIXES entry — typedef-wrapped struct/enum in C falls into glue
PR #156 reviewer nit #2. Documents the tension between spec body
("struct_specifier (named, top-level) → 1 unit") and the actual behavior
for the C idiom `typedef struct { ... } Foo;` — the inner struct_specifier
is anonymous, so the extractor falls into glue. Workaround: dogfood-driven
revisit if frequent pain point emerges.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:40:04 +00:00
6beff35a2f test(p10-1d-followup): add in-file unit tests to C AST extractor
Mirrors the cpp.rs 15-test pattern. Covers function_definition (incl.
pointer-return, static/extern/inline), struct_specifier / enum_specifier /
union_specifier (named), anonymous struct/enum/union → glue, typedef-wrapped
struct → glue (per spec risks note), preprocessor directives → glue, empty
file → <module> post-pass, preprocessor-only → <module>, mixed fn + glue →
<top-level> present, determinism (20 runs). 17 tests total.

Reviewer nit #1 (PR #156 code-reviewer).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:39:36 +00:00
75a4207aa1 feat(p10-1d): C + C++ AST chunkers — P10 Tier 1 chunker family complete (#156) 2026-05-21 15:48:34 +00:00
86aa180ad7 chore: bump version 0.15.0 → 0.16.0 (p10-1d C + C++ AST chunkers)
Minor bump — additive new chunker_versions code-c-ast-v1 + code-cpp-ast-v1
+ new routing langs c / cpp + new tree-sitter-c / tree-sitter-cpp workspace
deps. P10 Tier 1 chunker family complete. No DB migration, no wire schema
major bump.

Also lands the missing p10-3 try_skip_unchanged fallback-aware fix (Option
B1 — 7th param) that PR #155 was supposed to ship but never made it to main
(implementer reported commit SHA 2a39513 that didn't exist in the merged
branch). Same commit extends tier3_fallback_cv to include c/cpp.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 15:38:00 +00:00
802c573c07 docs(p10-1d): README/HANDOFF/ARCHITECTURE/SMOKE/INDEX sync
P10 Tier 1 chunker family complete (Rust + Python + TS + JS + Go + Java +
Kotlin + C + C++).

- README adds C/C++ to the ingest row + --code-lang c/cpp + Mermaid brace.
- HANDOFF flips p10-1D to  (v0.16.0), updates 한 줄 요약 + 다음 후보.
- ARCHITECTURE adds C/C++ to the code-parser row, extends flowchart pcode
  node, adds chunker tree entries.
- SMOKE adds P10-1D walkthrough section + verification checklist entry.
- tasks/INDEX + tasks/p10/INDEX flip p10-1D to .

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:35:59 +00:00
438870ee25 docs(p10-1d): activate C + C++ in frozen design §10
P10 Tier 1 chunker family complete (Rust + Python + TS + JS + Go + Java +
Kotlin + C + C++).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:32:26 +00:00
192835e5bf test(p10-1d): integration smoke tests for C + C++
Verifies end-to-end ingest + search + Citation::Code shape:
- tier1_c_ingest_searchable: .c file → --code-lang c search → symbol
  = function name (no nesting), lang = "c", chunker_version = "code-c-ast-v1".
- tier1_cpp_ingest_searchable: .cpp file → --code-lang cpp search →
  symbol starts with namespace::Class prefix, lang = "cpp",
  chunker_version = "code-cpp-ast-v1".

Brings code_ingest_smoke to 18 tests (Tier 1: 9 → 11, Tier 2: 3,
Tier 3: 4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:31:35 +00:00
1034de25a2 fix(p10-3+p10-1d): land the missing try_skip_unchanged fallback-aware fix
PR #155 (p10-3) merged WITHOUT the reviewer's required Option B1 fix —
the implementer reported a commit SHA (2a39513) that never made it to main.
Result: every reingest of a Tier 3-fallback file (non-k8s YAML, invalid
YAML, AST extractor failure) re-runs full extract + chunk + embed because
the parser/chunker version comparison can never match (stored is
code-text-paragraph-v1 / none-v1, but caller uses Tier 1/2 dispatch
values).

This commit:
1. Adds the 7th param `fallback_chunker_version: Option<&ChunkerVersion>`
   to try_skip_unchanged + the stored_is_tier3_fallback detection branch
   (skip parser/chunker equality, keep embedder check).
2. Threads `None` through non-code call sites (md / image / pdf).
3. Code call site computes tier3_fallback_cv covering all Tier 1/2 langs
   that can fall back: rust / python / ts / js / go / java / kotlin /
   yaml / dockerfile / toml / json / xml / groovy / go-mod / c / cpp
   (p10-1D additions).
4. Adds tier3_yaml_fallback_reingest_is_unchanged + tier3_shell_reingest_is_unchanged
   regression tests (the originally-promised PR #155 regression coverage
   that also never made it to main).

Smoke tests: 14 + 2 = 16 PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:19:17 +00:00
d1560be80d feat(p10-1d): activate C + C++ in ingest_one_code_asset dispatch
Extends 4-arm match (parser_version / chunker_version / extract / chunks)
+ allowlist + tier3_fallback_cv with "c" + "cpp" arms. C uses CAstExtractor
+ CodeCAstV1Chunker; C++ uses CppAstExtractor + CodeCppAstV1Chunker. Both
langs are Tier 3-fallback-eligible (e.g. .h file with C++ syntax may fail
tree-sitter-c parse → Tier 3 paragraph fallback per p10-3 wrapper).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:56:45 +00:00
b2a2902e38 feat(p10-1d): code-cpp-ast-v1 chunker + snapshot test
Identical chunker body to code-c-ast-v1 (per-language work happens in the
CppAstExtractor, Task C). Snapshot fixture covers nested namespace + class
+ ctor/dtor + method + operator overload + template fn + free fn + top-level
main, verifying namespace::Class::method symbol convention per design §3.4.

5 chunks emitted:
- <top-level> (includes, namespace opening)
- kebab::chunk::MdHeadingV1Chunker (class unit)
- kebab::identity (template function)
- kebab::global_helper (free function in namespace)
- main (top-level main function)

Template function symbols emit without <T> parameters per spec convention.
Namespace::Class::method pattern verified. All tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:46:12 +00:00
03cd41c48f feat(p10-1d): code-c-ast-v1 chunker + snapshot test
Mirrors code-go-ast-v1's chunker pattern. Snapshot test against
tests/fixtures/sample.c (function + typedef struct + typedef enum +
preprocessor) verifies symbol list + lang=c stamping.

Chunks produced (4 total):
- <top-level> glue: includes, defines, static vars, typedefs (lines 1-18)
- parse_record function (lines 20-23)
- print_record function (lines 25-27)
- main function (lines 29-33)

All chunks stamped with lang=c and chunker_version=code-c-ast-v1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:41:19 +00:00
926042049c feat(p10-1d): C++ AST extractor (tree-sitter-cpp)
Symbol = namespace::Class::method via recursive build_blocks. namespace_definition
pushes namespace name (anonymous → <anonymous>). nested_namespace_specifier
(outer::inner) flattens all segments and pushes them. class_specifier / struct_specifier
(named) emit class unit + recurse with class name pushed. function_definition emits
method unit; symbol resolution unpacks declarator chain (pointer_declarator /
reference_declarator → function_declarator → identifier / field_identifier /
qualified_identifier / operator_name / destructor_name).

operator_cast (conversion operators, e.g. operator bool) handled as a direct
declarator kind on function_definition. template_declaration recurses with same
prefix (template params NOT in symbol). enum_specifier + concept_definition emit
type-level units. linkage_specification (extern "C") recurses into body with same
prefix. Other top-level nodes → <top-level> glue.

All 15 unit tests pass; build and clippy clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:37:58 +00:00
e0a29225da feat(p10-1d): C AST extractor (tree-sitter-c)
Top-level units: function_definition (symbol = fn name from declarator's
innermost identifier), struct_specifier, enum_specifier, union_specifier
(each emits 1 unit with the named identifier as symbol). Preprocessor
directives + top-level declarations group into a <top-level> glue chunk.
Empty file or zero units → <module> post-pass.

C symbol = function name only — no namespace, no class nesting (design §3.4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:29:36 +00:00
b541567946 build(p10-1d): add tree-sitter-c + tree-sitter-cpp workspace deps
Standard crate names resolved cleanly: tree-sitter-c v0.24.2 and
tree-sitter-cpp v0.23.4 are both compatible with workspace tree-sitter 0.26.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:19:00 +00:00
a58d400abd docs(p10-1d): implementation plan (11 tasks A-K, subagent-driven)
Tasks: workspace deps / C extractor / C++ extractor / C chunker + snapshot /
C++ chunker + snapshot / ingest dispatch + tier3_fallback_cv extension /
2 smoke tests / frozen design §10 / docs sync / workspace test gate /
version bump 0.15.0 → 0.16.0 + gitea PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:15:22 +00:00
8add684ffc docs(p10-1d): task spec for C + C++ AST chunkers
Frozen contract: single PR with code-c-ast-v1 + code-cpp-ast-v1. C symbol
= function name only (no nesting). C++ symbol = namespace::Class::method
(recursion). .h → C (design §3.5); C++ headers' parse failure picked up
by p10-3 Tier 3 fallback. tree-sitter-c + tree-sitter-cpp workspace deps,
version bump 0.15.0 → 0.16.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:12:11 +00:00
222 changed files with 20524 additions and 1106 deletions

98
Cargo.lock generated
View File

@@ -4127,7 +4127,7 @@ dependencies = [
[[package]]
name = "kebab-app"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"base64 0.22.1",
@@ -4142,12 +4142,11 @@ dependencies = [
"kebab-embed-local",
"kebab-llm",
"kebab-llm-local",
"kebab-normalize",
"kebab-nli",
"kebab-parse-code",
"kebab-parse-image",
"kebab-parse-md",
"kebab-parse-pdf",
"kebab-parse-types",
"kebab-rag",
"kebab-search",
"kebab-source-fs",
@@ -4172,12 +4171,12 @@ dependencies = [
[[package]]
name = "kebab-chunk"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
"kebab-core",
"kebab-normalize",
"kebab-parse-code",
"kebab-parse-md",
"serde_json",
"serde_json_canonicalizer",
@@ -4188,7 +4187,7 @@ dependencies = [
[[package]]
name = "kebab-cli"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"clap",
@@ -4209,7 +4208,7 @@ dependencies = [
[[package]]
name = "kebab-config"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"dirs 5.0.1",
@@ -4224,7 +4223,7 @@ dependencies = [
[[package]]
name = "kebab-core"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
@@ -4238,7 +4237,7 @@ dependencies = [
[[package]]
name = "kebab-embed"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
@@ -4252,7 +4251,7 @@ dependencies = [
[[package]]
name = "kebab-embed-local"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"fastembed",
@@ -4265,7 +4264,7 @@ dependencies = [
[[package]]
name = "kebab-eval"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"kebab-app",
@@ -4284,7 +4283,7 @@ dependencies = [
[[package]]
name = "kebab-llm"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"kebab-core",
@@ -4293,7 +4292,7 @@ dependencies = [
[[package]]
name = "kebab-llm-local"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"kebab-config",
@@ -4310,7 +4309,7 @@ dependencies = [
[[package]]
name = "kebab-mcp"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"kebab-app",
@@ -4327,23 +4326,23 @@ dependencies = [
]
[[package]]
name = "kebab-normalize"
version = "0.15.0"
name = "kebab-nli"
version = "0.19.0"
dependencies = [
"anyhow",
"kebab-core",
"kebab-parse-md",
"kebab-parse-types",
"hf-hub",
"kebab-config",
"ndarray",
"ort",
"serde",
"serde_json",
"time",
"tempfile",
"tokenizers",
"tracing",
"unicode-normalization",
]
[[package]]
name = "kebab-parse-code"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"gix",
@@ -4353,6 +4352,8 @@ dependencies = [
"time",
"tracing",
"tree-sitter",
"tree-sitter-c",
"tree-sitter-cpp",
"tree-sitter-go",
"tree-sitter-java",
"tree-sitter-javascript",
@@ -4364,7 +4365,7 @@ dependencies = [
[[package]]
name = "kebab-parse-image"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"ab_glyph",
"anyhow",
@@ -4388,11 +4389,10 @@ dependencies = [
[[package]]
name = "kebab-parse-md"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"kebab-core",
"kebab-parse-types",
"lingua",
"pulldown-cmark",
"serde",
@@ -4401,11 +4401,12 @@ dependencies = [
"time",
"toml",
"tracing",
"unicode-normalization",
]
[[package]]
name = "kebab-parse-pdf"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
@@ -4416,23 +4417,16 @@ dependencies = [
"tracing",
]
[[package]]
name = "kebab-parse-types"
version = "0.15.0"
dependencies = [
"kebab-core",
"serde",
]
[[package]]
name = "kebab-rag"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
"kebab-config",
"kebab-core",
"kebab-llm",
"kebab-nli",
"kebab-search",
"kebab-store-sqlite",
"regex",
@@ -4447,7 +4441,7 @@ dependencies = [
[[package]]
name = "kebab-search"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"globset",
@@ -4466,7 +4460,7 @@ dependencies = [
[[package]]
name = "kebab-source-fs"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
@@ -4474,7 +4468,6 @@ dependencies = [
"ignore",
"kebab-config",
"kebab-core",
"kebab-parse-code",
"serde",
"serde_json",
"tempfile",
@@ -4485,7 +4478,7 @@ dependencies = [
[[package]]
name = "kebab-store-sqlite"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"blake3",
@@ -4493,7 +4486,6 @@ dependencies = [
"kebab-chunk",
"kebab-config",
"kebab-core",
"kebab-normalize",
"kebab-parse-md",
"refinery",
"rusqlite",
@@ -4506,7 +4498,7 @@ dependencies = [
[[package]]
name = "kebab-store-vector"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"arrow",
@@ -4530,7 +4522,7 @@ dependencies = [
[[package]]
name = "kebab-tui"
version = "0.15.0"
version = "0.19.0"
dependencies = [
"anyhow",
"crossterm",
@@ -8531,6 +8523,26 @@ dependencies = [
"tree-sitter-language",
]
[[package]]
name = "tree-sitter-c"
version = "0.24.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a9b2eb57a55fed6b00812912e730b7a275cf4fe98bfd6a5d76263d4438371728"
dependencies = [
"cc",
"tree-sitter-language",
]
[[package]]
name = "tree-sitter-cpp"
version = "0.23.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df2196ea9d47b4ab4a31b9297eaa5a5d19a0b121dceb9f118f6790ad0ab94743"
dependencies = [
"cc",
"tree-sitter-language",
]
[[package]]
name = "tree-sitter-go"
version = "0.25.0"

View File

@@ -2,11 +2,9 @@
resolver = "3"
members = [
"crates/kebab-core",
"crates/kebab-parse-types",
"crates/kebab-config",
"crates/kebab-source-fs",
"crates/kebab-parse-md",
"crates/kebab-normalize",
"crates/kebab-chunk",
"crates/kebab-store-sqlite",
"crates/kebab-store-vector",
@@ -24,6 +22,7 @@ members = [
"crates/kebab-tui",
"crates/kebab-mcp",
"crates/kebab-parse-code",
"crates/kebab-nli",
]
[workspace.package]
@@ -31,7 +30,95 @@ edition = "2024"
rust-version = "1.85"
license = "MIT OR Apache-2.0"
repository = "https://github.com/altair823/kebab"
version = "0.15.0"
version = "0.19.0"
# pre-v0.18 workspace-wide cleanup: enable clippy::pedantic group with
# intentional allow-list. The allowed lints are either cosmetic (doc style),
# informational (function size), or carry intentional truncation we accept
# (numeric casts in tokenizer/ONNX inputs, hash modular reduction, etc).
[workspace.lints.clippy]
pedantic = { level = "warn", priority = -1 }
# Intentional u32 ↔ i64 casts in kebab-nli (ONNX i64 inputs from tokenizer u32 ids).
# u64 ↔ usize across kebab-store-sqlite row counts. Wide truncation is auditable
# at use site, not lint-wide.
cast_possible_truncation = "allow"
cast_possible_wrap = "allow"
cast_sign_loss = "allow"
cast_precision_loss = "allow"
# Doc markdown style is cosmetic; we run rustdoc on demand.
doc_markdown = "allow"
missing_errors_doc = "allow"
missing_panics_doc = "allow"
# Informational only — splitting a long pipeline function isn't always cleaner.
too_many_lines = "allow"
# `Foo::default()` is concise and idiomatic here; `<Foo as Default>::default()`
# adds noise without surfacing intent.
default_trait_access = "allow"
# Module name prefix on public items keeps the wire/log surface readable
# (`refusal_reason::no_chunks` etc).
module_name_repetitions = "allow"
# We use `#[must_use]` deliberately on public results, not blanket.
must_use_candidate = "allow"
# `String` arg sometimes signals "I'll consume this" — let signature decide.
needless_pass_by_value = "allow"
# Idiomatic single-line bindings stay; let-else expansion isn't always clearer.
manual_let_else = "allow"
# `use` after `let` is a common kebab pattern (scoped imports next to use site).
items_after_statements = "allow"
# Naming pairs like `chunk_id` / `chunks_id` are intentional domain terms.
similar_names = "allow"
# `iter.map(format!).collect::<String>()` is idiomatic when the per-element
# string is genuinely independent — `fold` only wins on accumulation patterns.
format_collect = "allow"
# Exhaustive `match` with explicit variant arms (vs `_`) catches future
# variant additions at compile time (kebab core's `RefusalReason` pattern).
match_wildcard_for_single_variants = "allow"
# Copy types under `&self` keep call-site discipline; auto-deref noise > tiny perf gain.
trivially_copy_pass_by_ref = "allow"
# `unnecessary_wraps` flags helpers that could drop `Result`, but keeping the
# Result allows future error variants without churning callers.
unnecessary_wraps = "allow"
# NLI score / RRF fusion / similarity threshold comparisons are intentional —
# floats live in the `[0, 1]` band and are compared with explicit thresholds.
float_cmp = "allow"
# File-extension dispatch is keyed on ASCII conventions; case sensitivity
# is part of the spec for `.md`, `.pdf`, etc.
case_sensitive_file_extension_comparisons = "allow"
# Config / opts structs intentionally bundle boolean flags (ingest options,
# search modes, etc) — splitting them into enums would obscure the wire shape.
struct_excessive_bools = "allow"
# `bytecount` crate would be a new dep just for one-off ASCII counts.
naive_bytecount = "allow"
# `#[ignore]` annotations on tests document via the test name + nearby comment.
ignore_without_reason = "allow"
# `format!` push patterns are a hot path for kebab-tui's progressive rendering;
# `write!` rewrite needs a verified-equal benchmark before swapping.
format_push_string = "allow"
# Builder-style `with_*` methods return `Self`; the existing `#[must_use]`
# discipline lives on aggregate constructors, not every chainable setter.
return_self_not_must_use = "allow"
# Match arms grouped by side-effect over body equality (e.g. snake_case wire
# label tables) — fanning them out keeps adding a new variant trivial.
match_same_arms = "allow"
# Remaining style-only warnings: trailing `continue` is sometimes clearer than
# rewriting, `_x` underscored bindings document intent at the use site, and
# `!(a == b)` reads better than `a != b` when paired with a complementary check.
needless_continue = "allow"
used_underscore_binding = "allow"
nonminimal_bool = "allow"
# Other one-off cosmetic items: large literal formatting, doc link quoting,
# `Clone::clone_from` swap, `str::replace` chaining, `Iterator::any` ergonomics.
unreadable_literal = "allow"
many_single_char_names = "allow"
doc_link_with_quotes = "allow"
assigning_clones = "allow"
collapsible_str_replace = "allow"
trivial_regex = "allow"
elidable_lifetime_names = "allow"
range_plus_one = "allow"
explicit_iter_loop = "allow"
implicit_hasher = "allow"
ref_option = "allow"
[workspace.dependencies]
anyhow = "1"
@@ -99,6 +186,22 @@ tree-sitter-go = "0.25.0"
# JVM family grammars for code ingest (kebab-parse-code, p10-1C-JK).
tree-sitter-java = "0.23.5"
tree-sitter-kotlin-ng = "1.1.0" # bare tree-sitter-kotlin requires ts <0.23; -ng uses tree-sitter-language 0.1 (ts 0.26 compat)
# C/C++ family grammars for code ingest (kebab-parse-code, p10-1D).
tree-sitter-c = "0.24.2"
tree-sitter-cpp = "0.23.4"
# fb-41 PR-9 (kebab-nli): mDeBERTa-v3 XNLI verifier deps. Versions match
# the fastembed 4.9 transitive set so the ONNX Runtime + tokenizer stack
# stays single-versioned across the workspace. ort `default-features=false`
# drops the bundled binary downloader (fastembed already provides one);
# tokenizers `default-features=false, onig` swaps the default `esaxx` regex
# backend for `onig` so the build doesn't need libstdc++ headers (verified
# via PR-9a pre-flight: SentencePiece tokenizer.json loads + KR/EN encode).
# hf-hub uses `ureq + rustls-tls` to stay aligned with kebab-embed-local's
# pure-Rust TLS stack.
ort = { version = "=2.0.0-rc.9", default-features = false, features = ["ndarray"] }
tokenizers = { version = "0.21", default-features = false, features = ["onig"] }
hf-hub = { version = "0.4", default-features = false, features = ["ureq", "rustls-tls"] }
ndarray = "0.16"
# Disk-footprint trim for dev / test builds. Codegen, opt-level, and
# behavior are unchanged — only DWARF debug info is reduced (line

View File

@@ -4,7 +4,7 @@
## 한 줄 요약
P0P5 + P6 + P7 + P9-1/2/3/4 (Library / Search / Ask / Inspect) 머지 완료. `kebab ingest` 가 markdown / image / PDF / 소스코드 (Rust / Python / TS / JS / Go / Java / Kotlin) / Tier 2 리소스 파일 (yaml/k8s / dockerfile / toml / json / xml / groovy / go-mod) + Tier 3 paragraph fallback (shell / 비-k8s YAML / AST 실패 케이스) 처리. `kebab search` / `kebab ask` 가 매체 가로질러 결과 + page / code citation 반환. `kebab tui` 가 4 패널 (Library + Search + Ask + Inspect) 제공. P10-3 (Tier 3 paragraph fallback) 완료 — 다음 후보 = P10-1D (C/C++) 또는 P9-5 (desktop tauri) 또는 보류 중인 P8 (audio).
P0P5 + P6 + P7 + P9-1/2/3/4 (Library / Search / Ask / Inspect) + P10 전체 머지 완료 (현재 **v0.18.0**). `kebab ingest` 가 markdown / image / PDF / 소스코드 (Rust / Python / TS / JS / Go / Java / Kotlin / C / C++) / Tier 2 리소스 파일 (yaml/k8s / dockerfile / toml / json / xml / groovy / go-mod) + Tier 3 paragraph fallback (shell / 비-k8s YAML / AST 실패 케이스) 처리. `kebab search` / `kebab ask` 가 매체 가로질러 결과 + page / code citation 반환. `kebab tui` 가 4 패널 (Library + Search + Ask + Inspect) 제공. **v0.17.0 cut (2026-05-24)**: 한국어 trigram FTS5 tokenizer (PR #159) + C typedef alias unit (PR #160) + `code_lang_chunk_breakdown` additive (PR #161). **v0.17.1 cut (2026-05-25)**: 확장 도그푸딩 후 `[models.llm] request_timeout_secs` config 노브 (PR #162) + sudo 없이 ollama 설치 + `kebab ask --stream` UX 권장 docs (PR #163). **v0.17.2 cut (2026-05-25)**: v0.17.1 post-dogfood polish — `[image.ocr] request_timeout_secs` 별 노브 (PR #164, v0.17.1 미진행 closure) + `heading_path` FTS5 column filter 로 text-only 매칭 + raw-mode escape hatch (PR #165, 2026-05-24 v0.17.0 trigram entry 의 JSON 노이즈 closure). **v0.18.0 cut (2026-05-26)**: fb-41 multi-hop RAG + NLI verification ship (PR #176-180) — `kebab ask --multi-hop` 의 decompose → decide → synthesize loop + mDeBERTa-v3 XNLI ONNX post-synthesize entailment 검사. dogfood S7 caffeine hallucination 의 silent LLM-self-judge ceiling 해결 (nli_score 0.0035 graceful refuse). 추가 `chore: workspace-wide cleanup + post-PR9 refactor` (PR #181) — clippy::pedantic baseline + H1 config wiring + 9 new tests. 자세한 영향은 [v0.17.0 release notes](https://gitea.altair823.xyz/altair823-org/kebab/releases/tag/v0.17.0) + [v0.17.1 release notes](https://gitea.altair823.xyz/altair823-org/kebab/releases/tag/v0.17.1) + [v0.17.2 release notes](https://gitea.altair823.xyz/altair823-org/kebab/releases/tag/v0.17.2) + [v0.18.0 release notes](https://gitea.altair823.xyz/altair823-org/kebab/releases/tag/v0.18.0). 구조적으로 남은 component 는 P9-5 (desktop tauri) 하나뿐, P8 (audio) 는 사용자 보류.
## Phase 로드맵
@@ -20,18 +20,26 @@ P0P5 + P6 + P7 + P9-1/2/3/4 (Library / Search / Ask / Inspect) 머지 완료.
| **P7** | PDF text + page citation | `kebab-parse-pdf` | P5 | ✅ 완료 (3/3 component, page-level chunker + ingest wiring) |
| **P8** | 음성 transcription + timestamp citation | `kebab-parse-audio` | P5 | ⏸ 보류 (whisper-rs 시스템 dep brainstorm 필요) |
| **P9** | TUI + desktop app | `kebab-tui`, `kebab-desktop` | P5 | 🟡 진행 (4/5 component — P9-1/2/3/4 완료 [Library / Search / Ask / Inspect], P9-5 desktop 예정 · 도그푸딩 피드백 **20/20 ✅**) |
| **P10** | code ingest framework | `kebab-parse-code` | P5 | 🟡 진행 중 — 1A-1 ✅ (wire schema + parse-code skeleton + filter flags), 1A-2 ✅ (Rust AST chunker, `code-rust-ast-v1` — v0.7.0), 1B ✅ (Python/TS/JS AST chunkers — v0.8.0 이후), **1C-Go ✅ (Go AST chunker, `code-go-ast-v1` — v0.12.0)**, **1C-JavaKotlin ✅ (Java + Kotlin AST chunkers, `code-java-ast-v1` / `code-kotlin-ast-v1` — v0.13.0)**, **2 ✅ (Tier 2 resource-aware: yaml/k8s + dockerfile + manifest, `k8s-manifest-resource-v1` / `dockerfile-file-v1` / `manifest-file-v1` — v0.14.0)**, **3 ✅ (Tier 3 paragraph fallback: code-text-paragraph-v1 — v0.15.0)** |
| **P10** | code ingest framework | `kebab-parse-code` | P5 | 🟡 진행 중 — 1A-1 ✅ (wire schema + parse-code skeleton + filter flags), 1A-2 ✅ (Rust AST chunker, `code-rust-ast-v1` — v0.7.0), 1B ✅ (Python/TS/JS AST chunkers — v0.8.0 이후), **1C-Go ✅ (Go AST chunker, `code-go-ast-v1` — v0.12.0)**, **1C-JavaKotlin ✅ (Java + Kotlin AST chunkers, `code-java-ast-v1` / `code-kotlin-ast-v1` — v0.13.0)**, **2 ✅ (Tier 2 resource-aware: yaml/k8s + dockerfile + manifest, `k8s-manifest-resource-v1` / `dockerfile-file-v1` / `manifest-file-v1` — v0.14.0)**, **3 ✅ (Tier 3 paragraph fallback: code-text-paragraph-v1 — v0.15.0)**, **1D ✅ (C + C++ AST chunkers, code-c-ast-v1 + code-cpp-ast-v1 — v0.16.0)** |
P0~P5 직렬. P6~P9 P5 이후 병렬 가능.
## Component 카운트
총 33 component task — spec 시점 31 개 + 후속 wiring task 3 (P3-5 / P6-4 / P7-3) 가 머지 시점에 추가됨. per-component 진행 + status 는 [tasks/INDEX.md](tasks/INDEX.md).
총 33 component task — spec 시점 31 개 + 후속 wiring task 3 (P3-5 / P6-4 / P7-3) 가 머지 시점에 추가됨. v0.18.0 cut 시점에 fb-41 multi-hop RAG + NLI verification (PR-9 5 sub-PRs) 가 P9 추가 component 로 ship — `kebab-nli` 신규 crate (mDeBERTa-v3 XNLI ONNX verifier) + `kebab-rag::ask_multi_hop` (decompose/decide/synthesize loop + step 8.5 NLI hook). per-component 진행 + status 는 [tasks/INDEX.md](tasks/INDEX.md).
## 머지 후 발견된 버그 / 결정 (요약)
머지 후 발견된 모든 deviation / hotfix 의 dated 로그는 [tasks/HOTFIXES.md](tasks/HOTFIXES.md). 본 요약은 \"누군가가 인수받을 때 알아두면 시간을 많이 절약하는\" 항목만:
- **2026-05-26 kebab-normalize + kebab-parse-types 흡수 (24 → 22 crates, design §3.7b 재작성)** — v0.19.0 cut. 4 parser 중 markdown 한 갈래만 lift 를 경유하는 reality 가 design §3.7b 의 fan-in ≥ 2 가정과 diverge → thin layer (`kebab-parse-types`) + `kebab-normalize` 두 crate 가 `kebab-parse-md` 로 흡수. 5 사용 type + 3 forward-declared struct 모두 `kebab-parse-md::{types,normalize}` module 의 `pub` re-export 로 보존. wire / surface impact = 0 (CLI / TUI / MCP / `--json` / config / XDG / parser_version 모두 unchanged). 자세한 내용: `tasks/HOTFIXES.md` (2026-05-26 design deviation entry).
- **2026-05-26 v0.18.0 fb-41 multi-hop RAG + NLI verification ship (PR #176-180) + post-PR9 cleanup (PR #181)** — pre-v0.18.0 dogfood (`/build/cache/dogfood-v018/`, 33 assets / 205 chunks, gemma3:4b CPU only / 16 GB RAM) 에서 발견된 S7 caffeine hallucination 의 root cause = LLM-self-judge ceiling (synthesize 가 chunks 와 무관한 Adam optimizer gradient 식을 silent emit, self-judge 가 reject 못함). 학계 표준 (Self-RAG, CRAG, Auto-GDA, MedTrust-RAG) 결론 = deterministic post-synthesis verification. mDeBERTa-v3 XNLI ONNX (280 MB, Xenova HF) 가 `(packed_chunks, answer)` entailment 검사 — `[rag] nli_threshold > 0` (default 0.0 = disabled, production 권장 0.5) 일 때 활성. dogfood retest 측정 — S7 PR-8 baseline `grounded=true + Adam hallucination` → PR-9 `nli_verification_failed, nli_score 0.0035`. wire additive minor — `answer.v1.verification` field + `refusal_reason``nli_verification_failed` / `nli_model_unavailable` 추가, pre-v0.18 reader 무영향. 5 sub-PR 시퀀스 + cleanup PR (clippy::pedantic baseline + 의도적 30+ allow + H1 `[models.nli].model` config wiring + 9 new tests). post-refactor retest = PR-9d byte-identical (deterministic 확인). 자세한 내용: `tasks/HOTFIXES.md` (2026-05-25 fb-41 PR-9 closure entry + S3 follow-up).
- **2026-05-25 v0.17.2 post-v0.17.1 polish (PR #164 + #165)** — v0.17.1 의 두 follow-up closure. (1) `[image.ocr] request_timeout_secs` 별 노브 — `crates/kebab-parse-image/src/ocr.rs::REQUEST_TIMEOUT` hard 300s 제거, LLM 쪽 패턴 (PR #162) 을 OCR 어댑터에 동일 적용. 사용자 결정으로 별 노브 분리 (OCR vs LLM 의 cold start 패턴이 달라 독립 조절). v0.17.1 미진행 항목 closure. (2) `chunks_fts``heading_path` 컬럼이 JSON 표기 + path 세그먼트 까지 trigram 색인 → query false positive 가능 문제 closure. `lexical.rs::build_match_string` 가 non-raw 분기 결과를 `text : (<expr>)` 로 wrap — heading 색인 V007 verbatim 유지, 매칭만 text 한정. 사용자가 명시 heading 검색 하려면 raw mode `'heading_path : <token>'` escape hatch (SKILL.md 갱신). 둘 다 additive (옛 config 호환) / re-ingest 불필요. 자세한 내용: `tasks/HOTFIXES.md` (2026-05-25 v0.17.2 두 entry).
- **2026-05-25 v0.17.1 post-dogfood (PR #162 + #163)** — 확장 도그푸딩 (16 GB CPU only, gemma4:e4b 시도) 에서 발견된 두 follow-up 한 묶음. (1) `crates/kebab-llm-local/src/ollama.rs::REQUEST_TIMEOUT` hard 300s → `[models.llm] request_timeout_secs` config + env override (additive, default 300, `=0` 은 disable 아닌 "즉시 timeout" 이라 doc 명시). (2) README + SMOKE 에 sudo / systemd 없이 ollama 설치 + ≤4B Q4 권장 모델 + `kebab ask --stream` UX 권장 docs. additive only — 옛 config / wire 호환. 자세한 내용: `tasks/HOTFIXES.md` (2026-05-25).
- **2026-05-24 v0.17.0 PR-C `code_lang_chunk_breakdown` additive (closure of 2026-05-22 LOW)** — `schema.v1.stats` 에 chunk 수 집계 신규 키. 기존 `code_lang_breakdown` (doc count) 와 sister. 또 기존 두 필드 JSON schema description 의 "chunk count" 오기재 → "doc count" 로 정정. wire additive — schema_version bump 불필요. 자세한 내용: `tasks/HOTFIXES.md` (2026-05-24 PR-C).
- **2026-05-24 v0.17.0 PR-B C typedef alias unit (closure of 2026-05-21)** — `kebab-parse-code::c::extract_blocks``type_definition` 분기로 inner anonymous struct/enum/union → declarator 의 typedef alias 이름으로 synthetic unit 방출. `PARSER_VERSION code-c-v1``code-c-v2` bump + 같은-asset/다른-doc_id 케이스용 `purge_workspace_path_for_parser_bump` cascade (`stale_chunk_ids_for_workspace_path_except_doc_id` + `purge_document_at_workspace_path_except_doc_id` helper 신규). 사용자 작업 불필요 (다음 ingest 가 자동 재처리). 자세한 내용: `tasks/HOTFIXES.md` (2026-05-24 PR-B).
- **2026-05-24 v0.17.0 PR-A 한국어 trigram tokenizer 채택 (closure of 2026-05-22 한국어 lexical)** — `chunks_fts` 가 FTS5 `unicode61``trigram` 으로 V007 migration (자동 backfill, re-ingest 불필요). `lexical.rs::build_match_string` trigram-aware 재설계 — multi-token 한국어 query (`해시 충돌`) 가 whole-phrase 후보로 hit, 한영 혼합 (`Rust 충돌은`) 도 OR-combined. 2자 이하 query 는 0-hit + CLI/TUI/wire `hint` 안내. 영어 lexical 도 substring 매칭으로 바뀜 (recall ↑ / 단어 경계 ↓). `kebab.sqlite` 크기 ~2-5배 증가 (trigram index). 자세한 내용: `tasks/HOTFIXES.md` (2026-05-24).
- **2026-05-22 P10 종합 도그푸딩 round 2 (한국어 lexical 검색 한계)** — `kebab search --mode lexical` 의 한국어 query 가 FTS5 `unicode61` 토크나이저에서 거의 0 hit (어절 단위 토큰화 → 부분 매칭 불가). 기본 hybrid 모드는 `multilingual-e5-small` vector 가 carry 해 한국어 검색 정상. **closure**: 위 2026-05-24 v0.17.0 entry.
- **2026-05-20 P10-1B (Rust 1A symbol path 비일관 + expression-level 함수 미방출)** — (a) Rust `code-rust-ast-v1` 은 file-scope nesting 만 (workspace path prefix 없음), 1B 의 Python/TypeScript/JavaScript 는 workspace 경로 → module path prefix 사용 (비일관 수용, retrofit = chunker_version bump + reindex 필요, 사용자 명시 요청까지 보류); (b) TS/JS 의 `const foo = () => {...}` 같은 expression-level 함수는 `<top-level>` glue 로 처리됨 (declaration-level 단위만 1B 1차 범위). 자세한 내용: `tasks/HOTFIXES.md` (2026-05-20) 두 항목.
- **2026-05-19 P10-1A-2 (code_rust_ast_v1.rs + SourceType)** — `AST_CHUNK_MAX_LINES` 상수가 `IngestCodeCfg.ast_chunk_max_lines` 를 읽지 않고 모듈 상수 200 고정 (Chunker trait 이 per-medium config 미노출); `SourceType::Code` variant 부재로 code 파일이 `SourceType::Note` 로 분류됨 — 두 항목 모두 `tasks/HOTFIXES.md` (2026-05-19) 에 기록.
- **2026-05-07 fb-26 (progress.rs)** — `Aborted` unconditional writeln (TTY duplicate) + `Completed` TTY no summary fixed; `KEBAB_PROGRESS=plain` env + quiet suppression added
@@ -81,13 +89,13 @@ P0~P5 직렬. P6~P9 P5 이후 병렬 가능.
## 다음 task 후보
- **P9-2 TUI search** — `App.search` slot 채움. Library 의 `/` 가 enable 됨.
- **P9-3 TUI ask** — `App.ask` slot 채움. `?` enable.
- **P9-4 TUI inspect** — `App.inspect` slot 채움. `Enter` enable.
- **P9-5 desktop tauri** — 별도 분기. PDF citation rendering UI 가치 큼.
- **P8 audio brainstorm** — whisper-rs 시스템 dep 받을지 / 외부 transcription endpoint 사용할지 사용자 결정 필요. 사용자 패턴 (책+PDF 위주, audio 의향 없음) 상 후순위.
구조적으로 미완인 component 는 P9-5 하나뿐. 나머지는 도그푸딩 follow-up (아래 "P10 dogfooding 백로그") 또는 사용자 결정 대기.
P9-2/3/4 는 P9-1 의 parallel-safety contract (sub-state slot 패턴) 덕에 병렬 진행 가능 — 같은 `App` 손대지 않음.
- **P9-5 desktop tauri** — 마지막 남은 P9 component. `kebab-desktop` crate + Tauri 앱, 별도 분기. PDF citation rendering UI 가치 큼. 사용자 우선순위 (P9 우선 · 책/PDF 위주) 와 부합.
- **P10 도그푸딩 round 2 follow-up** — ✅ v0.17.0 cut (2026-05-24) 으로 세 항목 모두 closure (한국어 trigram PR-A + C typedef alias PR-B + code_lang_chunk_breakdown additive PR-C). 상세 cross-link: 아래 "P10 dogfooding 백로그" 절 + `tasks/HOTFIXES.md` (2026-05-24 PR-A/B/C).
- **P8 audio brainstorm** — whisper-rs 시스템 dep 받을지 / 외부 transcription endpoint 사용할지 사용자 결정 필요. 사용자 패턴 (책+PDF 위주, audio 의향 없음) 상 보류.
- **fb-41 multi-hop reasoning** — ⏳ 미구현, XL, eval 인프라 선행 + brainstorm 필요.
- **Rust symbol path retrofit** — Rust `code-rust-ast-v1` symbol 이 file-scope-only (1B+ 는 module prefix). `code-rust-ast-v2` bump + Rust corpus re-ingest 비용 → 사용자 명시 요청까지 보류. HOTFIXES `2026-05-20`.
### P9 dogfooding 백로그 (fb-26 ~ fb-42) — release 분할
@@ -96,11 +104,20 @@ P9-2/3/4 는 P9-1 의 parallel-safety contract (sub-state slot 패턴) 덕에
- **0.3.0 — agent foundation** ✅ cut 2026-05-07: fb-26 (log), fb-27 (introspection/error wire), fb-28 (readonly/quiet). ~~fb-29 (daemon)~~ → 🚫 **deferred** — fb-30 stdio MCP 가 동일 가치를 daemon 복잡도 없이 제공.
- **0.4.0 — agent integration (MCP)** ✅ cut: fb-30 (MCP stdio), fb-31 (single-file/stdin ingest).
- **0.5.0 — agent surface refinement (additive)** ✅ cut 2026-05-10: fb-32 (stale doc indicator), fb-33 (streaming ask), fb-34 (output budget controls), fb-35 (verbatim fetch), fb-36 (search filter args), fb-37 (trace + stats). 모두 wire schema additive minor.
- **0.6.0 — RAG quality** 🟡 진행: fb-38 (score semantics) ✅ 머지 (2026-05-10), fb-40 (fact-grounded answer / rag-v2 prompt) ✅ 머지 (2026-05-10), fb-39 (retrieval precision tuning, embedding_version cascade) — 미진행 (eval golden set 선행 필요).
- **0.7.0 또는 P+**: fb-41 (multi-hop reasoning, XL), fb-42 (bulk multi-query / rerank, Nice).
- **0.6.0 — RAG quality** ✅ 대부분 머지 (2026-05-10): fb-38 (score semantics) ✅, fb-39 (eval foundation — `precision_at_k_chunk` metric) ✅, fb-39b (embedding upgrade — multilingual-e5-large default) ✅, fb-40 (fact-grounded answer / rag-v2 prompt) ✅. 잔여 = fb-39 retrieval precision lever 실제 적용 (eval golden set 확장 선행 필요).
- **0.7.0 또는 P+**: fb-41 (multi-hop reasoning, XL) — ⏳ 미구현 · brainstorm 필요; fb-42 (bulk multi-query) ✅ 머지 (2026-05-10, bulk only — rerank hint 은 deferred).
각 fb spec frontmatter 의 `target_version` 필드가 source of truth. INDEX.md 의 release subheader 도 동일 grouping.
### P10 dogfooding 백로그 (2026-05-22 round 2)
P10 종합 도그푸딩 round 2 (`/build/cache/dogfood-p10b/`, OSS 8 repo + 한국어 위키 문서 10편) 에서 발견된 follow-up 후보. 자세한 내용 + 우선순위 근거는 `tasks/HOTFIXES.md` (2026-05-22).
- **한국어 lexical tokenizer** — ✅ v0.17.0 (2026-05-24) PR-A 머지 (#159). V007 trigram migration 자동 backfill + `build_match_string` 재설계 + CLI/TUI/wire hint. HOTFIXES `2026-05-24 PR-A` 참조.
- **code_lang_chunk_breakdown chunk 단위 집계 (LOW)** — ✅ v0.17.0 (2026-05-24) PR-C 머지 (#161). `schema.v1.stats` additive 필드. HOTFIXES `2026-05-24 PR-C` 참조.
- **C typedef-wrapped struct (LOW)** — ✅ v0.17.0 (2026-05-24) PR-B 머지 (#160). `type_definition` 분기 + `PARSER_VERSION code-c-v2` bump + orphan purge cascade. HOTFIXES `2026-05-24 PR-B` 참조.
- **ranking glue chunk 편향 (deferred)** — 자동 heuristic 은 user intent misalignment 위험. 사용자 명시 요청 전까지 surface 변경 0 유지. 1주+ 실사용 후 재 brainstorm.
## 검증된 운영 동작 (release binary, fastembed enabled)
P7-3 머지 직후 25 시나리오 smoke 통과 — markdown + image + PDF 5 자산 워크스페이스에서 doctor / ingest / list / inspect / search (lex/vec/hybrid) / re-ingest / byte-edit re-ingest / corrupt PDF / RAG ask + page citation 모두. 자세한 시나리오 표는 conversation 기록 참조; 워크스페이스에 직접 돌려보는 절차는 [docs/SMOKE.md](docs/SMOKE.md).

View File

@@ -6,6 +6,20 @@
- **Rust toolchain** ≥ 1.85 (workspace 가 edition 2024 + resolver 3 사용). [rustup](https://rustup.rs) 권장.
- **Ollama** — `kebab ask` 와 이미지 OCR/caption 가 사용. `https://ollama.com/download` 에서 설치 후 `ollama serve` 실행. 기본 LLM 은 gemma4 계열 (`ollama pull gemma4:e4b`) — OCR / caption 도 같은 family 라 모델 하나만 pull 하면 됨. 더 큰 variant 원하면 `gemma4:26b` 등으로 config override. config 의 `[models.llm].endpoint` 에 host:port 명시.
- **CPU only / RAM ≤ 16 GB 환경 권장 모델**: gemma4:e4b (8B) 는 CPU 추론에 무거워 RAG 한 답변이 5분을 넘기기 쉽다 — `[models.llm] request_timeout_secs` 의 기본 300 s 한도에 걸려 `error: kb-rag: llm.generate_stream` 으로 떨어진다 (HOTFIXES 2026-05-25). `gemma3:4b` / `qwen2.5:3b` / `phi3:mini` 같은 ≤ 4B Q4 모델로 바꾸면 답변 1-3 분에 안정 동작 (확장 도그푸딩에서 검증). 모델 storage 가 부담이면 `OLLAMA_MODELS=/path` env 로 위치 분리 가능.
- **`request_timeout_secs` 노브 (v0.17.0)**: `[models.llm] request_timeout_secs = 1200` (또는 `KEBAB_MODELS_LLM_REQUEST_TIMEOUT_SECS=1200`) 로 한도를 늘려 큰 모델도 시도 가능. 단 응답 동안 RAM 점유가 길어진다. **`= 0` 은 disable 이 아니라 "즉시 timeout"** (reqwest 의 의미상) — "사실상 무제한" 의도면 `u64::MAX` 또는 `86400` 같이 큰 finite 값 사용.
- **sudo 없이 설치 (격리 디렉토리 사용)**: `install.sh``/usr/local/bin/ollama` + `systemd` 유닛까지 건드리는 게 부담이면 binary tarball 만 받아 사용자 디렉토리에 풀고 env 로 모델 위치 분리하면 된다.
```bash
mkdir -p /opt/ollama/{models,logs}
curl -fL https://ollama.com/download/ollama-linux-amd64.tar.zst -o /tmp/ollama.tar.zst
zstd -d /tmp/ollama.tar.zst -o /tmp/ollama.tar && tar -xf /tmp/ollama.tar -C /opt/ollama/
# bin/ollama + lib/ollama/ 가 풀린다. 모델 디렉토리는 OLLAMA_MODELS 로 분리.
OLLAMA_MODELS=/opt/ollama/models OLLAMA_HOST=127.0.0.1:11434 \
/opt/ollama/bin/ollama serve > /opt/ollama/logs/serve.log 2>&1 &
/opt/ollama/bin/ollama pull gemma3:4b
```
루트 디스크 부담을 분리하고 싶을 때 (`~/.ollama/models` 가 기본) 그대로 활용. systemd 가 없는 컨테이너 / WSL2 / 회사 머신 등에서 유용.
- **`kebab ask --stream` 권장 (fb-33)**: 모델 cold start 가 길 때 (8B+ 또는 첫 호출) `--stream` 으로 토큰을 stderr 에 ndjson 으로 흘려 받으면 5 분 timeout 한도 안에서도 첫 토큰이 빨리 보여 사용자 체감이 개선된다. 동일 inference 시간이라도 wait-and-pray 보다 progressive 가 안정적. CLI: `kebab ask "..." --stream 2> events.ndjson > final.json`. MCP host 도 `streaming_ask` capability flag 가 `true` 면 자동 사용 권장.
- **빌드 디스크** — 첫 빌드 시 `target/` 가 610 GB (Lance + DataFusion + fastembed). 여유 확인.
- **fastembed 모델** — 첫 `kebab ingest` 시 `multilingual-e5-large` (~1.3 GB, fb-39b) 자동 다운로드. `config.toml` 에서 `model = "multilingual-e5-small"` 로 명시하면 이전 모델 사용.
@@ -70,12 +84,12 @@ kebab doctor
| 명령 | 동작 |
|------|------|
| `kebab init` | XDG 경로에 데이터 디렉토리 + config.toml 생성 |
| `kebab ingest [<path>]` | Markdown / 이미지 / PDF / Rust 소스코드 색인 (idempotent). TTY 에서는 stderr 진행 바, non-TTY (CI / pipe) 는 stderr 한 줄씩, `--json` 은 stdout 에 `ingest_progress.v1` 라인 streaming 후 마지막에 `ingest_report.v1`. Ctrl-C 한 번이면 현재 asset 마무리 후 abort (부분 commit 보존, idempotent re-run), 두 번째 Ctrl-C 는 hard exit. Markdown title 이 frontmatter 에 없어도 첫 H1 → H2 → 첫 paragraph 80 자 → 파일명 순으로 자동 채움 (parser_version `md-frontmatter-v2`) — 기존 색인된 doc 도 다음 ingest 에서 새 title 로 갱신. **Incremental** (p9-fb-23): 두 번째 이후의 ingest 는 변하지 않은 doc (blake3 + parser/chunker/embedder version 모두 동일) 의 parse/chunk/embed/vector upsert 를 자동 스킵. final summary 에 `N unchanged` 카운트 표시. `--force-reingest` 로 skip 무시 강제 재처리. **지원 형식** (extractor 자동 결정 — config 에 명시 불가): Markdown (`.md`), 이미지 (`.png` / `.jpg` / `.jpeg`, OCR + caption), PDF (`.pdf`), **소스코드** (`.rs``code-rust-ast-v1`, `.py``code-python-ast-v1`, `.ts`/`.tsx``code-ts-ast-v1`, `.js`/`.mjs`/`.cjs`/`.jsx``code-js-ast-v1`, `.go``code-go-ast-v1`, `.java``code-java-ast-v1`, `.kt`/`.kts``code-kotlin-ast-v1` — 모두 tree-sitter AST chunker; **Tier 2 리소스 파일**: `.yaml`/`.yml``k8s-manifest-resource-v1` (apiVersion+kind 파싱), `Dockerfile`/`Dockerfile.*`/`*.dockerfile``dockerfile-file-v1` (전체 파일), `Cargo.toml`/`pyproject.toml`/`.toml`/`package.json`/`tsconfig.json`/`.json`/`pom.xml`/`.xml`/`build.gradle`/`.gradle`/`go.mod``manifest-file-v1` (전체 파일) — yaml (k8s) / dockerfile / toml / json / xml / groovy / go-mod 지원); **Tier 3 paragraph fallback** (`.sh`/`.bash`/`.zsh``code-text-paragraph-v1`, blank-line paragraph split + 80-line/20-overlap line-window. Tier 1/2 가 0 chunk 또는 Err 시 자동 fallback — 비-k8s YAML 같은 케이스 picked up. symbol = None, lang 은 원본 보존.). 다른 확장자는 자동 skip — `IngestItem.warnings` 에 사유 (`"unsupported media type: .docx"` 등), `IngestReport.skipped_by_extension` 에 카운트 분류, CLI / TUI summary 에 breakdown 표시. 코드 chunk 는 `citation.kind = "code"``citation.lang = "<lang>"` + `symbol` + line range 를 담고, SearchHit top-level 에 `code_lang` + `repo` (`.git/` walk-up 의 디렉토리 이름) 가 backfill 됨. `--code-lang rust` / `--code-lang python` / `--code-lang typescript` / `--code-lang javascript` / `--code-lang go` / `--code-lang java` / `--code-lang kotlin` / `--code-lang yaml` / `--code-lang dockerfile` / `--code-lang toml` / `--code-lang json` / `--code-lang xml` / `--code-lang groovy` / `--code-lang go-mod` / `--code-lang shell` / `--media code` filter 로 언어별·코드 전용 검색 가능 (p10-1A-1 filter flags). Python symbol 은 workspace 경로 → dotted module path prefix (예: `kebab_eval.metrics.compute_mrr`), TS/JS symbol 은 slash-style module path prefix (예: `src/Foo.Foo.search`), Go symbol 은 `package.Func` / `package.(*Receiver).Method` 형식, Java / Kotlin symbol 은 `com.foo.Foo.bar` 형식 (패키지 + 클래스 + 메서드/필드). |
| `kebab search --mode {lexical,vector,hybrid} "<query>" [--no-cache] [--max-tokens N] [--snippet-chars N] [--cursor <opaque>] [--tag T] [--lang L] [--path-glob G] [--trust-min LEVEL] [--media TYPE] [--ingested-after RFC3339] [--doc-id ID] [--trace] [--bulk] [--repo NAME ...] [--code-lang LIST]` | 검색. hybrid는 RRF fusion, citation 포함. 같은 process 안에서 동일 query (NFKC + trim + lowercase 정규화) 반복 시 in-process LRU 캐시 hit (capacity = `[search] cache_capacity`, default 256). `--no-cache` 로 강제 bypass — 디버깅용. ingest commit 발생 시 `kv['corpus_revision']` bump 으로 모든 entry 자동 stale. **`--max-tokens` / `--snippet-chars` / `--cursor` (p9-fb-34)** — agent budget controls. `--json` 출력은 `search_response.v1` wrapper (`{hits, next_cursor, truncated}`) — pre-fb-34 의 bare array 와 호환 안 됨. mismatched cursor → `error.v1.code = stale_cursor`. **filter flags (p9-fb-36):** `--tag` 는 반복 가능 flag (`--tag rust --tag async`) 로 OR 매칭, `--media``,` 구분 다중 값 OR 매칭, 나머지 flags 간은 AND 조합. `--trust-min``primary\|secondary\|generated` 중 하나 (해당 level 이상 포함). `--ingested-after` 는 RFC3339 UTC — 파싱 실패 시 `error.v1.code = config_invalid` (exit 2). `--media md``markdown` alias 로 정규화. 알 수 없는 `--media` 값은 무조건 empty hits (오류 아님). **`--trace` (p9-fb-37)** — `search_response.v1.trace` 에 lexical / vector pre-fusion 후보 + RRF union + per-stage timing (`lexical_ms` / `vector_ms` / `fusion_ms` / `total_ms`) 노출. trace 요청은 캐시 우회 (`--no-cache` 없이도 항상 cold). **`--bulk` (p9-fb-42)** — stdin ndjson 으로 N query 한 번에 실행. `--json` 면 stdout per-query ndjson (`bulk_search_item.v1`) + stderr summary (`bulk_summary: total=N succeeded=S failed=F`). Cap 100. agent 가 query decomposition 후 sub-query 일괄 실행 시 single round-trip — App instance 재사용으로 캐시 / embedder cold-start 비용 한 번만. Per-query failure 는 item 의 `error` (error.v1) 에 격리, 다른 query 계속 진행. **code corpus filters (p10-1A-1):** `--repo` 는 반복 가능 (`--repo kebab --repo other`) OR 매칭. `--code-lang` 는 반복 또는 comma 다중 값 (`--code-lang rust,python`), 알 수 없는 값은 빈 hits. `--media code` 는 Tier 1/2/3 모든 code chunk 포함. 1A-1 시점에서는 indexed 된 code chunk 가 없어 filter 가 항상 빈 결과 — 1A-2 (Rust AST chunker) 머지 이후 실효. |
| `kebab ingest [<path>]` | Markdown / 이미지 / PDF / Rust 소스코드 색인 (idempotent). TTY 에서는 stderr 진행 바, non-TTY (CI / pipe) 는 stderr 한 줄씩, `--json` 은 stdout 에 `ingest_progress.v1` 라인 streaming 후 마지막에 `ingest_report.v1`. Ctrl-C 한 번이면 현재 asset 마무리 후 abort (부분 commit 보존, idempotent re-run), 두 번째 Ctrl-C 는 hard exit. Markdown title 이 frontmatter 에 없어도 첫 H1 → H2 → 첫 paragraph 80 자 → 파일명 순으로 자동 채움 (parser_version `md-frontmatter-v2`) — 기존 색인된 doc 도 다음 ingest 에서 새 title 로 갱신. **Incremental** (p9-fb-23): 두 번째 이후의 ingest 는 변하지 않은 doc (blake3 + parser/chunker/embedder version 모두 동일) 의 parse/chunk/embed/vector upsert 를 자동 스킵. final summary 에 `N unchanged` 카운트 표시. `--force-reingest` 로 skip 무시 강제 재처리. **지원 형식** (extractor 자동 결정 — config 에 명시 불가): Markdown (`.md`), 이미지 (`.png` / `.jpg` / `.jpeg`, OCR + caption), PDF (`.pdf`), **소스코드** (`.rs` → `code-rust-ast-v1`, `.py` → `code-python-ast-v1`, `.ts`/`.tsx` → `code-ts-ast-v1`, `.js`/`.mjs`/`.cjs`/`.jsx` → `code-js-ast-v1`, `.go` → `code-go-ast-v1`, `.java` → `code-java-ast-v1`, `.kt`/`.kts` → `code-kotlin-ast-v1`, `.c`/`.h` → `code-c-ast-v1`, `.cpp`/`.cc`/`.cxx`/`.hpp`/`.hh`/`.hxx` → `code-cpp-ast-v1` — 모두 tree-sitter AST chunker; **Tier 2 리소스 파일**: `.yaml`/`.yml` → `k8s-manifest-resource-v1` (apiVersion+kind 파싱), `Dockerfile`/`Dockerfile.*`/`*.dockerfile` → `dockerfile-file-v1` (전체 파일), `Cargo.toml`/`pyproject.toml`/`.toml`/`package.json`/`tsconfig.json`/`.json`/`pom.xml`/`.xml`/`build.gradle`/`.gradle`/`go.mod` → `manifest-file-v1` (전체 파일) — yaml (k8s) / dockerfile / toml / json / xml / groovy / go-mod 지원); **Tier 3 paragraph fallback** (`.sh`/`.bash`/`.zsh` → `code-text-paragraph-v1`, blank-line paragraph split + 80-line/20-overlap line-window. Tier 1/2 가 0 chunk 또는 Err 시 자동 fallback — 비-k8s YAML 같은 케이스 picked up. symbol = None, lang 은 원본 보존.). 다른 확장자는 자동 skip — `IngestItem.warnings` 에 사유 (`"unsupported media type: .docx"` 등), `IngestReport.skipped_by_extension` 에 카운트 분류, CLI / TUI summary 에 breakdown 표시. 코드 chunk 는 `citation.kind = "code"` 에 `citation.lang = "<lang>"` + `symbol` + line range 를 담고, SearchHit top-level 에 `code_lang` + `repo` (`.git/` walk-up 의 디렉토리 이름) 가 backfill 됨. `--code-lang rust` / `--code-lang python` / `--code-lang typescript` / `--code-lang javascript` / `--code-lang go` / `--code-lang java` / `--code-lang kotlin` / `--code-lang yaml` / `--code-lang dockerfile` / `--code-lang toml` / `--code-lang json` / `--code-lang xml` / `--code-lang groovy` / `--code-lang go-mod` / `--code-lang shell` / `--code-lang c` / `--code-lang cpp` / `--media code` filter 로 언어별·코드 전용 검색 가능 (p10-1A-1 filter flags). Python symbol 은 workspace 경로 → dotted module path prefix (예: `kebab_eval.metrics.compute_mrr`), TS/JS symbol 은 slash-style module path prefix (예: `src/Foo.Foo.search`), Go symbol 은 `package.Func` / `package.(*Receiver).Method` 형식, Java / Kotlin symbol 은 `com.foo.Foo.bar` 형식 (패키지 + 클래스 + 메서드/필드). |
| `kebab search --mode {lexical,vector,hybrid} "<query>" [--no-cache] [--max-tokens N] [--snippet-chars N] [--cursor <opaque>] [--tag T] [--lang L] [--path-glob G] [--trust-min LEVEL] [--media TYPE] [--ingested-after RFC3339] [--doc-id ID] [--trace] [--bulk] [--repo NAME ...] [--code-lang LIST]` | 검색. hybrid는 RRF fusion, citation 포함. 같은 process 안에서 동일 query (NFKC + trim + lowercase 정규화) 반복 시 in-process LRU 캐시 hit (capacity = `[search] cache_capacity`, default 256). `--no-cache` 로 강제 bypass — 디버깅용. ingest commit 발생 시 `kv['corpus_revision']` bump 으로 모든 entry 자동 stale. **`--max-tokens` / `--snippet-chars` / `--cursor` (p9-fb-34)** — agent budget controls. `--json` 출력은 `search_response.v1` wrapper (`{hits, next_cursor, truncated}`) — pre-fb-34 의 bare array 와 호환 안 됨. mismatched cursor → `error.v1.code = stale_cursor`. **filter flags (p9-fb-36):** `--tag` 는 반복 가능 flag (`--tag rust --tag async`) 로 OR 매칭, `--media` 는 `,` 구분 다중 값 OR 매칭, 나머지 flags 간은 AND 조합. `--trust-min` 은 `primary\|secondary\|generated` 중 하나 (해당 level 이상 포함). `--ingested-after` 는 RFC3339 UTC — 파싱 실패 시 `error.v1.code = config_invalid` (exit 2). `--media md` 는 `markdown` alias 로 정규화. 알 수 없는 `--media` 값은 무조건 empty hits (오류 아님). **`--trace` (p9-fb-37)** — `search_response.v1.trace` 에 lexical / vector pre-fusion 후보 + RRF union + per-stage timing (`lexical_ms` / `vector_ms` / `fusion_ms` / `total_ms`) 노출. trace 요청은 캐시 우회 (`--no-cache` 없이도 항상 cold). **`--bulk` (p9-fb-42)** — stdin ndjson 으로 N query 한 번에 실행. `--json` 면 stdout per-query ndjson (`bulk_search_item.v1`) + stderr summary (`bulk_summary: total=N succeeded=S failed=F`). Cap 100. agent 가 query decomposition 후 sub-query 일괄 실행 시 single round-trip — App instance 재사용으로 캐시 / embedder cold-start 비용 한 번만. Per-query failure 는 item 의 `error` (error.v1) 에 격리, 다른 query 계속 진행. **code corpus filters (p10-1A-1):** `--repo` 는 반복 가능 (`--repo kebab --repo other`) OR 매칭. `--code-lang` 는 반복 또는 comma 다중 값 (`--code-lang rust,python`), 알 수 없는 값은 빈 hits. `--media code` 는 Tier 1/2/3 모든 code chunk 포함. 1A-1 시점에서는 indexed 된 code chunk 가 없어 filter 가 항상 빈 결과 — 1A-2 (Rust AST chunker) 머지 이후 실효. **v0.17.0 trigram tokenizer (한국어 + 영어 동작 변경):** `chunks_fts` 가 FTS5 `trigram` 으로 동작 — 한국어 query 는 3자 이상 substring 매칭 (`해시 충돌` 같은 multi-token 도 whole-phrase 후보로 hit), 영어도 substring 매칭 (`token` 이 `tokenizer` 도 hit, recall ↑ / 단어 경계 ↓). 2자 이하 query 는 0-hit + stderr `[hint] 3자 이상 키워드 권장` + `search_response.v1.hint` 필드 (raw FTS5 mode `'...'` 제외). `kebab.sqlite` 파일 크기는 trigram index 비대화로 ~2-5배 또는 수백 MB 증가 (V007 자동 backfill, re-ingest 불필요). |
| `kebab list docs` | 색인된 문서 목록 |
| `kebab inspect doc <id>` / `kebab inspect chunk <id>` | raw record 보기 |
| `kebab fetch chunk <id> [--context N]` / `kebab fetch doc <id> [--max-tokens N]` / `kebab fetch span <doc_id> <ls> <le> [--max-tokens N]` | (p9-fb-35) verbatim text fetch from indexed corpus. wire = `fetch_result.v1` (kind discriminator). chunk: target + ±N ordinal-context chunks. doc: full normalized markdown. span: 1-based line range (PDF/audio rejected as `error.v1.code = span_not_supported`). chars/4 budget on doc/span. |
| `kebab ask "<query>" [--show-citations / --hide-citations] [--session <id>] [--stream]` | RAG 답변 + 근거 인용. 답변 후 `근거:` block 으로 full path / line range / score 한 줄씩 (default ON — `--hide-citations` 로 끄기, pipe 시 유용). 근거 부족 시 거절. Ollama 필요. `--session <id>` 로 multi-turn — 첫 호출에서 SQLite `chat_sessions` 에 자동 생성, 이후 호출은 prior turns 를 history 로 받아 follow-up. session id 는 사용자 지정 (e.g. `kb-rust-async-2026-05`) — `kebab reset --data-only` 로 모든 session wipe. **`--stream` (p9-fb-33)** 로 ndjson `answer_event.v1` event (retrieval_done → token* → final) 를 stderr 에 흘리고 stdout 마지막 줄에 기존 `answer.v1` — agent 가 token 즉시 소비 가능 |
| `kebab ask "<query>" [--show-citations / --hide-citations] [--session <id>] [--stream] [--multi-hop]` | RAG 답변 + 근거 인용. 답변 후 `근거:` block 으로 full path / line range / score 한 줄씩 (default ON — `--hide-citations` 로 끄기, pipe 시 유용). 근거 부족 시 거절. Ollama 필요. `--session <id>` 로 multi-turn — 첫 호출에서 SQLite `chat_sessions` 에 자동 생성, 이후 호출은 prior turns 를 history 로 받아 follow-up. session id 는 사용자 지정 (e.g. `kb-rust-async-2026-05`) — `kebab reset --data-only` 로 모든 session wipe. **`--stream` (p9-fb-33)** 로 ndjson `answer_event.v1` event (retrieval_done → token* → final) 를 stderr 에 흘리고 stdout 마지막 줄에 기존 `answer.v1` — agent 가 token 즉시 소비 가능. **`--multi-hop` (v0.18.0 fb-41)** — single-pass 대신 decompose → decide → synthesize 의 N-hop loop. compound 질문 (cross-doc / prereq chain) 에 효과적. 최종 답변 후 mDeBERTa-v3 XNLI 가 `(packed_chunks, generated_answer)` entailment 검사 — `[rag] nli_threshold > 0` (default 0.0 = disabled, production 권장 0.5) 일 때 활성. entailment < threshold → `refusal_reason = "nli_verification_failed"` (LLM-self-judge ceiling 극복, S7 caffeine hallucination 같은 케이스 catch). 첫 호출 시 ~280 MB ONNX model 자동 다운로드 + RAM peak ~7-8 GB (gemma3:4b 기준). model unavailable 시 `refusal_reason = "nli_model_unavailable"`, 우회는 `[rag] nli_threshold = 0` 임시 disable. |
| `kebab doctor` | 설정/모델/DB 헬스 체크 |
| `kebab tui` | Ratatui 셸 (Library + Search + Ask + Inspect 패널, desktop 진행 중). Library 에서 `r` 키로 background ingest 시작 — 화면 하단 status bar 가 진행 표시, 완료/abort 시 final 라인 잠시 유지 후 자동 hide. ingest 진행 중 `Esc` / `Ctrl-C` 가 cancel signal (그 외에는 quit). vim-style mode (header 우측 `-- NORMAL --` / `-- INSERT --`) — Library/Inspect 는 자동 NORMAL, Search/Ask 는 자동 INSERT. `i` 로 Normal→Insert (모든 pane — p9-fb-21), `Esc` 로 Insert→Normal 어디서나. mode-authoritative dispatch — Search 의 `j/k/o/g`, Ask 의 `e/j/k` 는 NORMAL 모드에서만 명령으로 동작, INSERT 에서는 입력 문자로 typing. (Search 의 chunk inspect 키는 `i`→`o` 로 rebind — `i` 가 universal Insert toggle.) **`F1` 로 cheatsheet popup** (현재 pane 의 키 매핑 + global 토글 표) — `Esc` / `F1` 로 닫기. Search 패널은 200ms debounce 후 background worker 가 검색 — 키 입력으로 UI freeze 안 됨, 사용자가 계속 타이핑하면 stale 결과 자동 폐기 (generation counter). Ask 패널은 multi-turn — 같은 conversation 안에서 Q1/A1, Q2/A2 transcript 누적, 다음 질문이 이전 턴을 history 로 받아 답변. 답변 본문은 markdown 렌더 (bold/italic/inline code/heading/list/code fence/table/blockquote, raw `**bold**` 가 실제 굵게 표시). `Ctrl-L` 로 새 conversation 시작. Search 의 `g` 키가 `$EDITOR` (기본 `vi`) 로 hit 의 citation 위치 열기 — 종료 후 TUI 화면이 자동으로 깨끗이 redraw. CLI `kebab ask` 는 raw markdown 그대로 (terminal 호환성 위해). Library 의 doc-list 가 한글 / 일본어 / 중국어 (CJK) 제목을 wide-char 정확한 column width 로 truncate — 한글 제목이 한 줄을 넘기지 않음 (CJK 1 자 = 2 col). Search/Ask/Filter 입력의 cursor 가 wide char 위에서 column 단위로 정렬 — 한글 입력 시 caret 이 글자 옆에 정확히 놓임. `← / →` 로 입력 문자열 중간 cursor 이동 (한글 한 글자 = 2 column 이라도 한 번에 이동), `Home / End` 로 양 끝 점프, `Delete` 로 cursor 위치 char 삭제 — 모든 input pane (Ask / Search / Library filter overlay) 동일 (p9-fb-22). Ask 트랜스크립트는 새 답변이 viewport 아래로 누적될 때 자동으로 tail 을 따라감 (auto-scroll); `j` / `k` 로 위로 스크롤하면 freeze, `Shift-G` 로 다시 bottom + auto-tail 재개. 화면 하단 hint line 은 한국어 동사구로 (`"위로"` / `"아래로"` / `"필터"` / `"타이핑 검색어"` / `"Esc 로 NORMAL 모드"` / `"i 입력모드"` 등) + 현재 (pane, mode) 조합에 맞춰 자동 분기, **첫 fragment 가 항상 `F1 도움말`** (cheatsheet 발견성 보장). 모든 모드에서 항상 떠 있는 상태바 — `kebab v<version> │ <pane> │ <docs> docs │ <state>` (state: streaming/searching/indexing/idle, ingest 진행 중에는 progress 가 같은 자리에 흡수됨). Ask 진입 시 conversation id 8 자 prefix 도 함께 표시. Ask 트랜스크립트와 Inspect 양쪽에서 `PgUp / PgDn` 으로 10 줄씩 페이지 스크롤. Library 의 doc list 위에는 `TITLE / TAGS / UPDATED / CHUNKS` 컬럼 헤더 행 표시 (display-width 정렬, Hangul / CJK 안전). |
| `kebab reset [--all / --data-only / --vector-only / --config-only] [--yes]` | XDG 데이터 wipe. **Irreversible.** TTY 면 confirm prompt, 아니면 `--yes` 필수. `--vector-only` 는 SQLite `embedding_records` 도 함께 truncate (orphan 방지) |
@@ -132,7 +146,7 @@ flowchart TB
subgraph Pipeline["도메인 + 파이프라인"]
parse["parse-md / parse-pdf / parse-image / parse-code"]
chunker["chunker (md-heading-v1, pdf-page-v1, code-{rust,python,ts,js,go,java,kotlin}-ast-v1, k8s-manifest-resource-v1, dockerfile-file-v1, manifest-file-v1, code-text-paragraph-v1)"]
chunker["chunker (md-heading-v1, pdf-page-v1, code-{rust,python,ts,js,go,java,kotlin,c,cpp}-ast-v1, k8s-manifest-resource-v1, dockerfile-file-v1, manifest-file-v1, code-text-paragraph-v1)"]
embedder["embedder (fastembed multilingual-e5-large)"]
retriever["retriever (lexical / vector / hybrid RRF)"]
rag["RAG pipeline"]

View File

@@ -12,8 +12,6 @@ kebab-core = { path = "../kebab-core" }
kebab-config = { path = "../kebab-config" }
kebab-source-fs = { path = "../kebab-source-fs" }
kebab-parse-md = { path = "../kebab-parse-md" }
kebab-parse-types = { path = "../kebab-parse-types" }
kebab-normalize = { path = "../kebab-normalize" }
kebab-chunk = { path = "../kebab-chunk" }
kebab-store-sqlite = { path = "../kebab-store-sqlite" }
kebab-store-vector = { path = "../kebab-store-vector" }
@@ -23,6 +21,11 @@ kebab-embed-local = { path = "../kebab-embed-local" }
kebab-llm = { path = "../kebab-llm" }
kebab-llm-local = { path = "../kebab-llm-local" }
kebab-rag = { path = "../kebab-rag" }
# p9-fb-41 PR-9c-2: facade construction of OnnxNliVerifier when
# `[rag] nli_threshold > 0`. Trait-only consumption via kebab-rag's
# `with_verifier`; no kebab-nli internals leak into kebab-app code
# beyond the construction site in `open_with_config`.
kebab-nli = { path = "../kebab-nli" }
# P6-4: image extractor + OCR + caption adapters live here. App
# threads them into the per-asset dispatch (see `ingest_one_asset`
# image branch). Trait-only consumption — no `kebab-parse-image`
@@ -76,3 +79,6 @@ lopdf = "0.32"
# error_wire::tests::llm_unreachable_classifies_to_model_unreachable needs a real
# reqwest::Error (private constructor) — built from a connect-refused call.
reqwest = { version = "0.12", default-features = false, features = ["blocking", "rustls-tls"] }
[lints]
workspace = true

View File

@@ -40,11 +40,18 @@ use anyhow::{Context, Result, anyhow};
use lru::LruCache;
use kebab_core::{
Answer, DocumentStore, Embedder, IndexVersion, LanguageModel, Retriever, SearchHit,
SearchMode, SearchOpts, SearchQuery, VectorStore,
Answer, DocumentStore, Embedder, ExtractContext, Extractor, IndexVersion, LanguageModel,
MediaType, Retriever, SearchHit, SearchMode, SearchOpts, SearchQuery, VectorStore,
};
use kebab_embed_local::FastembedEmbedder;
use kebab_llm_local::OllamaLanguageModel;
use kebab_parse_code::{
CAstExtractor, CppAstExtractor, GoAstExtractor, JavaAstExtractor,
JavascriptAstExtractor, KotlinAstExtractor, PythonAstExtractor, RustAstExtractor,
TypescriptAstExtractor,
};
use kebab_parse_image::ImageExtractor;
use kebab_parse_pdf::PdfTextExtractor;
use kebab_rag::{AskOpts, RagPipeline};
use kebab_search::{HybridRetriever, LexicalRetriever, VectorRetriever};
use kebab_store_sqlite::SqliteStore;
@@ -73,6 +80,37 @@ pub struct SearchResponse {
/// p9-fb-37: present when caller passed `SearchOpts.trace = true`.
/// Consumers that ignore trace should leave this `None`.
pub trace: Option<kebab_core::SearchTrace>,
/// v0.17.0 A5 Step 4b: human / agent-readable advisory string set
/// when the empty hit list is likely due to a query shorter than the
/// FTS5 trigram tokenizer's 3-char minimum. `None` otherwise. CLI
/// surfaces it on stderr (text mode); MCP / `--json` consumers
/// surface it however they prefer. See
/// `docs/superpowers/specs/2026-05-22-korean-trigram-tokenizer-design.md`
/// §3.3.
pub hint: Option<String>,
}
/// v0.17.0 A5 Step 4b: decide whether to attach a "3자 이상 키워드 권장"
/// hint to a `SearchResponse`. Fires only when the result set is empty
/// *and* the trimmed query is shorter than the trigram tokenizer can
/// resolve. Raw FTS5 mode (`'...'`) opts out — the user explicitly
/// invoked FTS5 syntax. Identical condition powers the CLI stderr line
/// and (separately) the TUI status bar.
pub fn short_query_hint(query_text: &str, hits_empty: bool) -> Option<String> {
if !hits_empty {
return None;
}
let trimmed = query_text.trim();
let bytes = trimmed.as_bytes();
// Raw single-quote mode: user opted into FTS5 syntax, no advisory.
if bytes.len() >= 2 && bytes[0] == b'\'' && bytes[bytes.len() - 1] == b'\'' {
return None;
}
if trimmed.chars().count() < 3 {
Some("3자 이상 키워드 권장 (trigram tokenizer 제약)".to_string())
} else {
None
}
}
/// Facade state — see module docs for lifetime rules.
@@ -84,6 +122,12 @@ pub struct SearchResponse {
pub struct App {
pub(crate) config: kebab_config::Config,
pub(crate) sqlite: Arc<SqliteStore>,
/// post-v0.18.0 extractor-dispatch-unification: polymorphic Extractor
/// registry. App init 시 1회 등록되어 `extract_for(...)` 가 lookup
/// 한다. 현재 11 entry (ImageExtractor + PdfTextExtractor + 9 AST).
/// MarkdownExtractor 는 별 PR 에서 추가 — markdown ingest path 는
/// 본 PR 에서 free-function 그대로 유지.
pub(crate) extractors: Vec<Box<dyn Extractor + Send + Sync>>,
/// Memoized embedder — built lazily on first `embedder()` call when
/// embeddings are enabled. `OnceLock` keeps the struct `Sync` and
/// the build path cold-only-once.
@@ -102,6 +146,17 @@ pub struct App {
/// `corpus_revision` snapshot embedded in `SearchCacheKey`
/// invalidates every entry the moment a new ingest commit lands.
search_cache: Option<Mutex<LruCache<SearchCacheKey, Vec<SearchHit>>>>,
/// p9-fb-41 PR-9c-2: NLI verifier built eagerly at
/// `open_with_config` time when `config.rag.nli_threshold > 0`,
/// consumed by `RagPipeline::with_verifier` on every `ask` /
/// `ask_with_session` call. `None` when the gate is disabled
/// (default, threshold = 0) — multi-hop skips step 8.5 entirely
/// and single-pass never touches the verifier.
///
/// Built eagerly (not lazy) so the `open_with_config` `?`
/// propagation surfaces NLI model construction errors at App
/// boot time, before any user query runs.
pipeline_verifier: Option<Arc<dyn kebab_nli::NliVerifier>>,
}
/// p9-fb-19: cache key for `App::search`. Includes every field that
@@ -162,16 +217,76 @@ impl App {
// `None` (cache disabled — every search hits the retrievers).
let search_cache = NonZeroUsize::new(config.search.cache_capacity)
.map(|cap| Mutex::new(LruCache::new(cap)));
// post-v0.18.0 extractor-dispatch-unification: build the 11-entry
// Extractor registry. All entries are state-less unit structs with
// zero-cost `new()`, so init cost is effectively 0 and side effects
// are 0 — `pipeline_verifier` fallible `?` below may bail but the
// already-constructed `extractors` Vec drops without cost. Markdown
// is NOT registered (see field doc).
let extractors: Vec<Box<dyn Extractor + Send + Sync>> = vec![
Box::new(ImageExtractor::new()),
Box::new(PdfTextExtractor::new()),
Box::new(RustAstExtractor::new()),
Box::new(PythonAstExtractor::new()),
Box::new(TypescriptAstExtractor::new()),
Box::new(JavascriptAstExtractor::new()),
Box::new(GoAstExtractor::new()),
Box::new(JavaAstExtractor::new()),
Box::new(KotlinAstExtractor::new()),
Box::new(CAstExtractor::new()),
Box::new(CppAstExtractor::new()),
];
// p9-fb-41 PR-9c-2: build the NLI verifier when the gate is
// enabled. App carries it on `RagPipeline` via
// `with_verifier` so the rag crate doesn't have to know about
// kebab-nli construction. Failure (`?`) surfaces as a user-
// facing error at App boot — never a panic in the pipeline's
// `expect("verifier must be Some when nli_threshold > 0.0")`.
let pipeline_verifier: Option<Arc<dyn kebab_nli::NliVerifier>> =
if config.rag.nli_threshold > 0.0 {
let v = kebab_nli::OnnxNliVerifier::new(&config).context(
"kebab-app: construct OnnxNliVerifier (config.rag.nli_threshold > 0)",
)?;
Some(Arc::new(v))
} else {
None
};
Ok(Self {
config,
sqlite: Arc::new(sqlite),
extractors,
embedder: OnceLock::new(),
vector: OnceLock::new(),
llm: OnceLock::new(),
search_cache,
pipeline_verifier,
})
}
/// Polymorphic dispatcher for the [`Extractor`] trait. Looks up the
/// first Extractor whose `supports(media)` returns true and invokes
/// `extract(ctx, bytes)` on it.
///
/// Errors with `anyhow!("no Extractor for media_type {media:?}")`
/// when no matching Extractor is registered. Callers in
/// `ingest_one_*_asset` reach this only after the outer 4-arm
/// dispatch (`MediaType::Markdown` / `Image` / `Pdf` / `Code(lang)`)
/// has matched, so a miss is a programming error — NOT a user-
/// facing skip.
pub(crate) fn extract_for(
&self,
media: &MediaType,
ctx: &ExtractContext<'_>,
bytes: &[u8],
) -> Result<kebab_core::CanonicalDocument> {
let extractor = self
.extractors
.iter()
.find(|e| e.supports(media))
.ok_or_else(|| anyhow!("no Extractor for media_type {media:?}"))?;
extractor.extract(ctx, bytes)
}
/// Run a [`SearchQuery`] through the configured retriever stack and
/// return the top-k hits. p9-fb-19: result is served from the
/// in-process LRU cache when the same `(query_norm, mode, k,
@@ -235,7 +350,7 @@ impl App {
// so other in-flight searches can use the cache concurrently.
drop(guard);
let hits = self.search_uncached(query)?;
let mut guard = cache.lock().unwrap_or_else(|e| e.into_inner());
let mut guard = cache.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
guard.put(key, hits.clone());
Ok(hits)
}
@@ -409,7 +524,7 @@ impl App {
// Snippet truncation if opts.snippet_chars set (mirror non-trace path).
if opts.snippet_chars.is_some() {
for h in hits.iter_mut() {
for h in &mut hits {
if h.snippet.chars().count() > snippet_chars {
h.snippet = trim_to_chars(&h.snippet, snippet_chars);
}
@@ -418,11 +533,13 @@ impl App {
// Trace path skips the budget loop. Caller will inspect
// `hits.len()` and `trace.timing` rather than paginate.
let hint = short_query_hint(&query.text, hits.is_empty());
return Ok(SearchResponse {
hits,
next_cursor: None,
truncated: false,
trace: Some(trace),
hint,
});
}
@@ -442,7 +559,7 @@ impl App {
// `config.search.snippet_chars`; this only kicks in when the
// caller asked for *less*).
if opts.snippet_chars.is_some() {
for h in hits.iter_mut() {
for h in &mut hits {
if h.snippet.chars().count() > snippet_chars {
h.snippet = trim_to_chars(&h.snippet, snippet_chars);
}
@@ -461,7 +578,7 @@ impl App {
{
current_snippet_cap =
(current_snippet_cap / 2).max(SNIPPET_FLOOR);
for h in hits.iter_mut() {
for h in &mut hits {
if h.snippet.chars().count() > current_snippet_cap {
h.snippet =
trim_to_chars(&h.snippet, current_snippet_cap);
@@ -505,11 +622,13 @@ impl App {
None
};
let hint = short_query_hint(&query.text, hits.is_empty());
Ok(SearchResponse {
hits,
next_cursor,
truncated,
trace: None,
hint,
})
}
@@ -518,9 +637,26 @@ impl App {
pub fn ask(&self, query: &str, opts: AskOpts) -> Result<Answer> {
let retriever = self.build_retriever(opts.mode)?;
let llm = self.llm()?;
let pipeline = self.build_pipeline(retriever, llm);
pipeline.ask(query, opts)
}
/// p9-fb-41 PR-9c-2: shared pipeline builder used by [`Self::ask`]
/// and [`Self::ask_with_session`]. Attaches the App-built NLI
/// verifier (when `cfg.rag.nli_threshold > 0`) via
/// `RagPipeline::with_verifier`, keeping the construction site in
/// a single place so the two call paths can't drift.
fn build_pipeline(
&self,
retriever: Arc<dyn Retriever>,
llm: Arc<dyn LanguageModel>,
) -> RagPipeline {
let pipeline =
RagPipeline::new(self.config.clone(), retriever, llm, self.sqlite.clone());
pipeline.ask(query, opts)
match &self.pipeline_verifier {
Some(v) => pipeline.with_verifier(v.clone()),
None => pipeline,
}
}
/// p9-fb-18: shared retriever-stack builder used by [`Self::ask`]
@@ -625,10 +761,11 @@ impl App {
// p9-fb-18 R1: shared retriever builder removes the prior
// copy of `ask`'s 35-line stack — see [`Self::build_retriever`].
// p9-fb-41 PR-9c-2: shared `build_pipeline` attaches the NLI
// verifier when the gate is enabled.
let retriever = self.build_retriever(opts.mode)?;
let llm = self.llm()?;
let pipeline =
RagPipeline::new(self.config.clone(), retriever, llm, self.sqlite.clone());
let pipeline = self.build_pipeline(retriever, llm);
let answer = pipeline.ask_with_history(
query,
history,
@@ -788,7 +925,7 @@ impl App {
/// clear` admin command). No-op when the cache is disabled.
pub fn clear_search_cache(&self) {
if let Some(cache) = self.search_cache.as_ref() {
let mut guard = cache.lock().unwrap_or_else(|e| e.into_inner());
let mut guard = cache.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
guard.clear();
}
}
@@ -1077,3 +1214,128 @@ mod tests_trace {
assert!(resp.trace.is_some(), "trace populated when opts.trace=true");
}
}
/// post-v0.18.0 extractor-dispatch-unification: in-crate unit tests for
/// the `App.extractors` registry + `App::extract_for` polymorphic
/// dispatch. In-crate (not `tests/`) because `extractors` + `extract_for`
/// are `pub(crate)` — integration tests cannot reach them.
///
/// Spec §5.1 + plan §2 Step 10 — 3 test class:
/// 1. registry length = 11 (image + pdf + 9 AST).
/// 2. mutually-exclusive `supports()` grid over 16 sample MediaTypes.
/// 3. `extract_for` returns `Err("no Extractor ...")` for registry-NOT-cover
/// MediaType (Audio).
#[cfg(test)]
mod tests_extractor_dispatch {
use super::*;
use kebab_core::{AudioType, ExtractConfig, ImageType};
/// helper: tempdir-isolated App for tests (mirrors `tests_trace`'s
/// `open_app_with_temp_dir` pattern).
fn open_app_with_temp_dir() -> (tempfile::TempDir, App) {
let dir = tempfile::tempdir().unwrap();
let mut cfg = kebab_config::Config::defaults();
cfg.storage.data_dir = dir.path().to_string_lossy().into_owned();
// Bring up migrations.
let store = kebab_store_sqlite::SqliteStore::open(&cfg).unwrap();
store.run_migrations().unwrap();
drop(store);
let app = App::open_with_config(cfg).unwrap();
(dir, app)
}
/// Registry length invariant: 11 Extractor (image + pdf + 9 AST).
/// Markdown is NOT registered (free-function path — defer to a
/// separate PR per spec §3.4).
#[test]
fn registry_has_eleven_extractors() {
let (_dir, app) = open_app_with_temp_dir();
assert_eq!(
app.extractors.len(),
11,
"registry must hold 11 Extractors (image + pdf + 9 AST). \
markdown 은 별 PR."
);
}
/// 11 Extractor 의 `supports()` 가 16 sample MediaType 에 대해
/// mutually exclusive — 어떤 두 Extractor 도 동일 MediaType 에
/// 대해 true 반환 안 됨.
#[test]
fn supports_grid_is_mutually_exclusive() {
let (_dir, app) = open_app_with_temp_dir();
let samples = vec![
MediaType::Markdown,
MediaType::Pdf,
MediaType::Image(ImageType::Png),
MediaType::Image(ImageType::Jpeg),
MediaType::Code("rust".into()),
MediaType::Code("python".into()),
MediaType::Code("typescript".into()),
MediaType::Code("javascript".into()),
MediaType::Code("go".into()),
MediaType::Code("java".into()),
MediaType::Code("kotlin".into()),
MediaType::Code("c".into()),
MediaType::Code("cpp".into()),
MediaType::Code("yaml".into()), // registry NOT cover
MediaType::Code("shell".into()), // registry NOT cover
MediaType::Audio(AudioType::Wav), // registry NOT cover
];
for sample in &samples {
let hits: Vec<_> = app
.extractors
.iter()
.filter(|e| e.supports(sample))
.collect();
assert!(
hits.len() <= 1,
"mutually exclusive violated for {sample:?}: {} hits",
hits.len()
);
}
}
/// `extract_for` 가 registry NOT cover MediaType (Audio) 에 대해
/// `Err("no Extractor for media_type ...")` 반환. Audio MediaType
/// 사용으로 RawAsset 의 actual content 의존 회피 — registry NOT
/// cover → 즉시 Err.
#[test]
fn extract_for_unsupported_media_errors() {
let (_dir, app) = open_app_with_temp_dir();
// Minimal RawAsset. Actual content never read — Audio MediaType
// 는 registry NOT cover → `extract_for` 가 dispatch loop 안에서
// 바로 Err 반환. RawAsset field set 은 `crates/kebab-core/src/
// asset.rs:62-73` 와 정합 (8 field).
let asset = kebab_core::RawAsset {
asset_id: kebab_core::AssetId("00".repeat(16)),
source_uri: kebab_core::SourceUri::File("/tmp/dummy.wav".into()),
workspace_path: kebab_core::WorkspacePath("dummy.wav".to_string()),
media_type: MediaType::Audio(AudioType::Wav),
byte_len: 0,
checksum: kebab_core::Checksum("00".repeat(32)),
discovered_at: time::OffsetDateTime::now_utc(),
// AssetStorage::Inline 미존재 — actual variant `Copied { path }`
// 사용 (kebab-core/src/asset.rs:55-60).
stored: kebab_core::AssetStorage::Copied {
path: std::path::PathBuf::from("/tmp/dummy.wav"),
},
};
let workspace_root: std::path::PathBuf = std::path::PathBuf::from("/tmp");
let cfg = ExtractConfig::default();
let ctx = ExtractContext {
asset: &asset,
workspace_root: &workspace_root,
config: &cfg,
};
let result = app.extract_for(&MediaType::Audio(AudioType::Wav), &ctx, &[]);
assert!(result.is_err(), "Audio 는 registry 미포함 → Err 기대");
let err_msg = format!("{:#}", result.unwrap_err());
assert!(
err_msg.contains("no Extractor"),
"unexpected err: {err_msg}"
);
}
}

View File

@@ -96,6 +96,11 @@ fn serialize_search_response(r: &SearchResponse) -> Value {
None => Value::Null,
};
map.insert("trace".to_string(), trace_v);
// v0.17.0 A5 Step 4b: only emit `hint` when set — matches
// the CLI wire wrapper's additive emit pattern.
if let Some(hint) = &r.hint {
map.insert("hint".to_string(), Value::String(hint.clone()));
}
}
v
}
@@ -134,9 +139,8 @@ fn parse_one(raw: &Value) -> Result<(SearchQuery, SearchOpts), String> {
let k = obj
.get("k")
.and_then(|v| v.as_u64())
.map(|n| n as usize)
.unwrap_or(0); // 0 → use config default in app
.and_then(serde_json::Value::as_u64)
.map_or(0, |n| n as usize); // 0 → use config default in app
let trust_min = match obj.get("trust_min").and_then(|v| v.as_str()) {
None => None,
@@ -204,14 +208,14 @@ fn parse_one(raw: &Value) -> Result<(SearchQuery, SearchOpts), String> {
let opts = SearchOpts {
max_tokens: obj
.get("max_tokens")
.and_then(|v| v.as_u64())
.and_then(serde_json::Value::as_u64)
.map(|n| n as usize),
snippet_chars: obj
.get("snippet_chars")
.and_then(|v| v.as_u64())
.and_then(serde_json::Value::as_u64)
.map(|n| n as usize),
cursor: obj.get("cursor").and_then(|v| v.as_str()).map(String::from),
trace: obj.get("trace").and_then(|v| v.as_bool()).unwrap_or(false),
trace: obj.get("trace").and_then(serde_json::Value::as_bool).unwrap_or(false),
};
Ok((

View File

@@ -91,7 +91,7 @@ pub fn classify(err: &anyhow::Error, verbose: bool) -> ErrorV1 {
}
let mut details = json!({});
if verbose {
let chain: Vec<String> = err.chain().map(|c| c.to_string()).collect();
let chain: Vec<String> = err.chain().map(std::string::ToString::to_string).collect();
details = json!({"chain": chain});
}
ErrorV1 {

View File

@@ -50,7 +50,7 @@ pub fn ensure_kebabignore_entry(workspace_root: &Path) -> Result<()> {
if !existing.is_empty() && !existing.ends_with('\n') {
file.write_all(b"\n")?;
}
writeln!(file, "{}", KEBABIGNORE_LINE)?;
writeln!(file, "{KEBABIGNORE_LINE}")?;
Ok(())
}

View File

@@ -166,8 +166,8 @@ mod tests {
};
let v = serde_json::to_value(&ev).unwrap();
assert_eq!(v.get("kind").and_then(|s| s.as_str()), Some("asset_started"));
assert_eq!(v.get("idx").and_then(|n| n.as_u64()), Some(1));
assert_eq!(v.get("total").and_then(|n| n.as_u64()), Some(10));
assert_eq!(v.get("idx").and_then(serde_json::Value::as_u64), Some(1));
assert_eq!(v.get("total").and_then(serde_json::Value::as_u64), Some(10));
assert_eq!(v.get("path").and_then(|s| s.as_str()), Some("notes/foo.md"));
assert_eq!(v.get("media").and_then(|s| s.as_str()), Some("markdown"));
}
@@ -184,8 +184,8 @@ mod tests {
let v = serde_json::to_value(&ev).unwrap();
assert_eq!(v.get("kind").and_then(|s| s.as_str()), Some("completed"));
let counts = v.get("counts").unwrap();
assert_eq!(counts.get("scanned").and_then(|n| n.as_u64()), Some(5));
assert_eq!(counts.get("new").and_then(|n| n.as_u64()), Some(2));
assert_eq!(counts.get("scanned").and_then(serde_json::Value::as_u64), Some(5));
assert_eq!(counts.get("new").and_then(serde_json::Value::as_u64), Some(2));
}
#[test]

View File

@@ -39,20 +39,17 @@ use std::sync::Arc;
use anyhow::{Context, anyhow};
use serde::{Deserialize, Serialize};
use kebab_chunk::{CodeGoAstV1Chunker, CodeJavaAstV1Chunker, CodeJsAstV1Chunker, CodeKotlinAstV1Chunker, CodePythonAstV1Chunker, CodeRustAstV1Chunker, CodeTextParagraphV1Chunker, CodeTsAstV1Chunker, DockerfileFileV1Chunker, K8sManifestResourceV1Chunker, ManifestFileV1Chunker, MdHeadingV1Chunker, PdfPageV1Chunker};
use kebab_chunk::{CodeCAstV1Chunker, CodeCppAstV1Chunker, CodeGoAstV1Chunker, CodeJavaAstV1Chunker, CodeJsAstV1Chunker, CodeKotlinAstV1Chunker, CodePythonAstV1Chunker, CodeRustAstV1Chunker, CodeTextParagraphV1Chunker, CodeTsAstV1Chunker, DockerfileFileV1Chunker, K8sManifestResourceV1Chunker, ManifestFileV1Chunker, MdHeadingV1Chunker, PdfPageV1Chunker};
use kebab_core::{
Answer, Block, CanonicalDocument, Chunk, ChunkId, ChunkPolicy, ChunkerVersion, Chunker,
DocFilter, DocSummary, DocumentId, DocumentStore, Embedder, EmbeddingInput,
EmbeddingKind, ExtractContext, Extractor, IngestReport, Lang, LanguageModel, MediaType,
EmbeddingKind, ExtractContext, IngestReport, Lang, LanguageModel, MediaType,
ParserVersion, RawAsset, SearchHit, SearchQuery, SourceScope,
SourceUri, VectorRecord, VectorStore,
};
use kebab_llm_local::OllamaLanguageModel;
use kebab_normalize::build_canonical_document;
use kebab_parse_image::{ImageExtractor, OllamaVisionOcr, apply_caption, apply_ocr};
use kebab_parse_code::{GoAstExtractor, JavaAstExtractor, JavascriptAstExtractor, KotlinAstExtractor, PythonAstExtractor, RustAstExtractor, TypescriptAstExtractor};
use kebab_parse_pdf::PdfTextExtractor;
use kebab_parse_md::{BodyHints, parse_blocks, parse_frontmatter};
use kebab_parse_image::{OllamaVisionOcr, apply_caption, apply_ocr};
use kebab_parse_md::{BodyHints, build_canonical_document, parse_blocks, parse_frontmatter};
use kebab_source_fs::FsSourceConnector;
mod app;
@@ -69,7 +66,7 @@ pub mod reset;
pub mod schema;
mod staleness;
pub use app::{App, SearchResponse};
pub use app::{App, SearchResponse, short_query_hint};
pub use ingest_progress::{AggregateCounts, IngestEvent, render_skipped_breakdown};
pub use reset::{ResetReport, ResetScope, enumerate_orphans};
pub use error_wire::{ERROR_V1_ID, ErrorV1, StructuredError, classify};
@@ -289,8 +286,7 @@ pub fn ingest_with_config_opts(
let cancelled = || {
opts.cancel
.as_ref()
.map(|c| c.load(std::sync::atomic::Ordering::Relaxed))
.unwrap_or(false)
.is_some_and(|c| c.load(std::sync::atomic::Ordering::Relaxed))
};
let force_reingest = opts.force_reingest;
let started_instant = std::time::Instant::now();
@@ -355,9 +351,7 @@ pub fn ingest_with_config_opts(
} else {
None
};
let image_extractor = ImageExtractor::new();
let image_pipeline = ImagePipeline {
extractor: &image_extractor,
ocr_engine: ocr_engine.as_ref(),
caption_llm: caption_llm.as_deref(),
};
@@ -394,7 +388,7 @@ pub fn ingest_with_config_opts(
let purged_deleted_files = sweep_deleted_files(
&app,
&scanned_paths,
vector_store.as_ref().map(|v| v.as_ref()),
vector_store.as_ref().map(std::convert::AsRef::as_ref),
)?;
let started_at = time::OffsetDateTime::now_utc();
@@ -509,10 +503,10 @@ pub fn ingest_with_config_opts(
*skipped_by_extension.entry(ext).or_insert(0) += 1;
}
kebab_core::IngestItemKind::Unchanged => {
unchanged_count = unchanged_count.saturating_add(1)
unchanged_count = unchanged_count.saturating_add(1);
}
kebab_core::IngestItemKind::Error => {
error_count = error_count.saturating_add(1)
error_count = error_count.saturating_add(1);
}
}
crate::ingest_progress::emit(
@@ -760,7 +754,6 @@ type SqliteStoreAlias = kebab_store_sqlite::SqliteStore;
/// once per ingest invocation. Threaded through `ingest_one_asset` so
/// the dispatch does not need ten separate parameters.
struct ImagePipeline<'a> {
extractor: &'a ImageExtractor,
ocr_engine: Option<&'a OllamaVisionOcr>,
caption_llm: Option<&'a dyn LanguageModel>,
}
@@ -795,6 +788,7 @@ fn try_skip_unchanged(
current_chunker_version: &ChunkerVersion,
current_embedding_version: Option<&kebab_core::EmbeddingVersion>,
force_reingest: bool,
fallback_chunker_version: Option<&ChunkerVersion>, // p10-3 fix
) -> anyhow::Result<Option<kebab_core::IngestItem>> {
if force_reingest {
return Ok(None);
@@ -829,12 +823,72 @@ fn try_skip_unchanged(
if existing_doc.source_asset_id != asset.asset_id {
return Ok(None);
}
// p10-3 fix: detect "stored doc was previously Tier 3 fallback".
// When a Tier 1/2 extractor emits empty chunks, the fallback wrapper
// retries with CodeTextParagraphV1Chunker and stores
// last_chunker_version = "code-text-paragraph-v1" + parser_version = "none-v1".
// On the next ingest the caller computes current_parser_version /
// current_chunker_version from the Tier 1/2 dispatch (e.g.
// "k8s-manifest-resource-v1"), which can never match the stored
// fallback values, causing spurious re-ingests. Detect this case
// and bypass the parser/chunker equality checks — only the embedder
// version still must match.
let stored_is_tier3_fallback = fallback_chunker_version.is_some_and(|fbv| {
existing_doc.last_chunker_version.as_ref() == Some(fbv)
&& existing_doc.parser_version.0 == "none-v1"
});
if stored_is_tier3_fallback {
// Embedder version still must match.
let embedder_match = existing_doc.last_embedding_version.as_ref()
== current_embedding_version;
if !embedder_match {
return Ok(None);
}
let candidate_doc_id = existing_doc.doc_id.clone();
tracing::debug!(
target: "kebab-app::ingest",
path = %asset.workspace_path.0,
doc_id = %candidate_doc_id.0,
"skip-unchanged: tier 3 fallback state detected; bypassing parser/chunker equality"
);
return Ok(Some(kebab_core::IngestItem {
kind: kebab_core::IngestItemKind::Unchanged,
doc_id: Some(candidate_doc_id),
doc_path: asset.workspace_path.clone(),
asset_id: Some(asset.asset_id.clone()),
byte_len: Some(asset.byte_len),
block_count: u32::try_from(existing_doc.blocks.len()).ok(),
chunk_count: None,
parser_version: Some(existing_doc.parser_version.clone()),
chunker_version: existing_doc.last_chunker_version.clone(),
warnings: Vec::new(),
error: None,
}));
}
// 2. Parser unchanged: parser_version is baked into id_for_doc so
// a version bump yields a different doc_id and the row above
// would have been missing. Checking here explicitly keeps the
// logic self-documenting and guards against future id_for_doc
// changes.
if existing_doc.parser_version != *current_parser_version {
// v0.17.0 PR-B: parser_version bump cascade. Same bytes (same
// asset_id) → asset-keyed `stale_chunk_ids_at` is a no-op, but
// the stale `documents` row at this workspace_path still
// collides with `idx_docs_workspace_path` on the next INSERT
// and the LanceDB rows under the old chunk_ids orphan. Sweep
// both stores here, before returning Ok(None), so the caller's
// full-ingest path lands a clean slate. The `keep_doc_id = ""`
// sentinel removes every doc at this path (the new doc_id is
// not yet known here — it's computed downstream from the new
// PARSER_VERSION).
purge_workspace_path_for_parser_bump(app, asset).with_context(|| {
format!(
"parser-bump orphan purge at {}",
asset.workspace_path.0
)
})?;
return Ok(None);
}
// 3. Chunker unchanged.
@@ -879,9 +933,7 @@ fn try_skip_unchanged(
fn ext_for_skip_warning(path: &str) -> String {
std::path::Path::new(path)
.extension()
.and_then(|s| s.to_str())
.map(|s| s.to_ascii_lowercase())
.unwrap_or_else(|| NO_EXT_SENTINEL.to_string())
.and_then(|s| s.to_str()).map_or_else(|| NO_EXT_SENTINEL.to_string(), str::to_ascii_lowercase)
}
/// p9-fb-25: render the `IngestItem.warnings` line for a Skipped
@@ -948,12 +1000,12 @@ fn ingest_one_asset(
force_reingest,
);
}
// p10-1A-2 / 1B: code ingest dispatch. p10-2: Tier 2 langs added. p10-3: shell added.
// p10-1A-2 / 1B: code ingest dispatch. p10-2: Tier 2 langs added. p10-3: shell added. p10-1D: c/cpp added.
MediaType::Code(lang)
if matches!(lang.as_str(),
"rust" | "python" | "typescript" | "javascript" | "go" | "java" | "kotlin"
| "yaml" | "dockerfile" | "toml" | "json" | "xml" | "groovy" | "go-mod"
| "shell") =>
| "shell" | "c" | "cpp") =>
{
return ingest_one_code_asset(
app,
@@ -1017,6 +1069,7 @@ fn ingest_one_asset(
&MdHeadingV1Chunker.chunker_version(),
embedder.map(|e| e.model_version()).as_ref(),
force_reingest,
None,
)? {
return Ok(item);
}
@@ -1057,7 +1110,7 @@ fn ingest_one_asset(
parser_version,
all_warnings,
)
.context("kb-normalize::build_canonical_document")?;
.context("kb-parse-md::build_canonical_document")?;
let chunks = MdHeadingV1Chunker
.chunk(&canonical, chunk_policy)
@@ -1174,7 +1227,6 @@ fn ingest_one_image_asset(
image_pipeline: &ImagePipeline<'_>,
force_reingest: bool,
) -> anyhow::Result<kebab_core::IngestItem> {
let image_extractor = image_pipeline.extractor;
let ocr_engine = image_pipeline.ocr_engine;
let caption_llm = image_pipeline.caption_llm;
let path = match &asset.source_uri {
@@ -1211,6 +1263,7 @@ fn ingest_one_image_asset(
&MdHeadingV1Chunker.chunker_version(),
embedder.map(|e| e.model_version()).as_ref(),
force_reingest,
None,
)? {
return Ok(item);
}
@@ -1232,9 +1285,9 @@ fn ingest_one_image_asset(
workspace_root: &workspace_root,
config: &extract_config,
};
let mut canonical = image_extractor
.extract(&ctx, &bytes)
.context("kb-parse-image::ImageExtractor::extract")?;
let mut canonical = app
.extract_for(&asset.media_type, &ctx, &bytes)
.context("kb-app::extract_for (image)")?;
// 2 + 3. Apply OCR / caption when their adapters exist. Both are
// Lenient — failure is captured into Provenance Warning,
@@ -1439,6 +1492,53 @@ fn record_image_analysis_failure(
warning_notes.push(note);
}
/// v0.17.0 PR-B: parser-bump cascade. When a code extractor ships a
/// new `PARSER_VERSION` (e.g. `code-c-v1` → `code-c-v2`), the same
/// (workspace_path, asset_id) pair re-emerges with a fresh `doc_id`.
/// The existing asset-keyed [`purge_vector_orphans_for_workspace_path`]
/// only fires on asset_id changes (file bytes edited) and is a no-op
/// here. Without an explicit doc-keyed sweep the next INSERT raises
/// `idx_docs_workspace_path` UNIQUE and the LanceDB rows under the
/// stale chunk_ids orphan. This helper:
///
/// 1. Fetches every stale chunk_id at `workspace_path` from SQLite
/// (`keep_doc_id = ""` means "all existing docs are stale" —
/// `try_skip_unchanged` calls this before the new doc_id is
/// computed).
/// 2. Deletes the matching vectors from every Lance table (no-op if
/// embeddings are disabled).
/// 3. Sweeps the SQLite `documents` row (CASCADE drops `blocks` /
/// `chunks` / `embedding_records`). The `assets` row stays — same
/// bytes, same asset_id, only the derived `doc_id` changed.
fn purge_workspace_path_for_parser_bump(
app: &App,
asset: &RawAsset,
) -> anyhow::Result<()> {
let path = &asset.workspace_path.0;
let stale = app
.sqlite
.stale_chunk_ids_for_workspace_path_except_doc_id(path, "")
.context("SqliteStore::stale_chunk_ids_for_workspace_path_except_doc_id")?;
if !stale.is_empty() {
if let Some(vec_store) = app.vector().context("App::vector")? {
use kebab_core::VectorStore as _;
vec_store
.delete_by_chunk_ids(&stale)
.context("VectorStore::delete_by_chunk_ids (parser-bump orphans)")?;
}
}
app.sqlite
.purge_document_at_workspace_path_except_doc_id(path, "")
.context("SqliteStore::purge_document_at_workspace_path_except_doc_id")?;
tracing::debug!(
target: "kebab-app",
path = %path,
count = stale.len(),
"purged orphan vectors + document for parser_version bump"
);
Ok(())
}
/// HOTFIXES 2026-05-02 P7-3 follow-up: when a tracked file's bytes
/// change, `purge_orphan_at_workspace_path` (in `kebab-store-sqlite`)
/// sweeps the SQLite chain (documents → blocks / chunks / embedding_records)
@@ -1657,6 +1757,7 @@ fn ingest_one_pdf_asset(
&PdfPageV1Chunker.chunker_version(),
embedder.map(|e| e.model_version()).as_ref(),
force_reingest,
None,
)? {
return Ok(item);
}
@@ -1673,9 +1774,9 @@ fn ingest_one_pdf_asset(
workspace_root: &workspace_root,
config: &extract_config,
};
let mut canonical = PdfTextExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-pdf::PdfTextExtractor::extract")?;
let mut canonical = app
.extract_for(&asset.media_type, &ctx, &bytes)
.context("kb-app::extract_for (pdf)")?;
// Per-medium chunker selection: PDF docs always use pdf-page-v1
// regardless of `config.chunking.chunker_version`. The chunker
@@ -1838,6 +1939,9 @@ fn ingest_one_code_asset(
=> ParserVersion("none-v1".to_string()),
// p10-3: shell direct routes to Tier 3 (no parse step).
"shell" => ParserVersion("none-v1".to_string()),
// p10-1D: C + C++ AST extractors.
"c" => ParserVersion(kebab_parse_code::C_PARSER_VERSION.to_string()),
"cpp" => ParserVersion(kebab_parse_code::CPP_PARSER_VERSION.to_string()),
other => anyhow::bail!("unsupported code_lang: {other}"),
};
@@ -1857,9 +1961,24 @@ fn ingest_one_code_asset(
=> ManifestFileV1Chunker.chunker_version(),
// p10-3:
"shell" => CodeTextParagraphV1Chunker.chunker_version(),
// p10-1D: C + C++ AST chunkers.
"c" => CodeCAstV1Chunker.chunker_version(),
"cpp" => CodeCppAstV1Chunker.chunker_version(),
other => anyhow::bail!("unreachable chunker_version: {other}"),
};
// p10-3 fix: if this lang can fall back to Tier 3, compute the fallback
// chunker_version so try_skip_unchanged can detect the stored-as-Tier-3
// state and skip parser/chunker equality checks.
let tier3_fallback_cv: Option<ChunkerVersion> = match code_lang {
"rust" | "python" | "typescript" | "javascript"
| "go" | "java" | "kotlin"
| "yaml" | "dockerfile" | "toml" | "json" | "xml" | "groovy" | "go-mod"
| "c" | "cpp" // p10-1D
=> Some(CodeTextParagraphV1Chunker.chunker_version()),
_ => None,
};
if let Some(item) = try_skip_unchanged(
app,
asset,
@@ -1867,6 +1986,7 @@ fn ingest_one_code_asset(
&chunker_version,
embedder.map(|e| e.model_version()).as_ref(),
force_reingest,
tier3_fallback_cv.as_ref(),
)? {
return Ok(item);
}
@@ -1881,30 +2001,18 @@ fn ingest_one_code_asset(
config: &extract_config,
};
// p10-1b Task D/G/J/L: extractor per-lang.
// post-v0.18.0 extractor-dispatch-unification:
// 9 AST lang 의 dispatch 가 polymorphic — App.extractors registry 의
// `*AstExtractor` entry 가 lang string 으로 disjoint `supports()` 비교
// 후 단일 hit. Tier 2 (manifest) + Tier 3 (shell) 은 free-function
// `synthesize_tier2_document` 유지 (Extractor impl 아님 — 별 PR).
// p10-3: capture Result so Tier 1 extractor errors can fall back to Tier 3.
let canonical_result: anyhow::Result<kebab_core::CanonicalDocument> = match code_lang {
"rust" => RustAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::RustAstExtractor::extract (code:rust)"),
"python" => PythonAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::PythonAstExtractor::extract (code:python)"),
"typescript" => TypescriptAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::TypescriptAstExtractor::extract (code:typescript)"),
"javascript" => JavascriptAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::JavascriptAstExtractor::extract (code:javascript)"),
"go" => GoAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::GoAstExtractor::extract (code:go)"),
"java" => JavaAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::JavaAstExtractor::extract (code:java)"),
"kotlin" => KotlinAstExtractor::new()
.extract(&ctx, &bytes)
.context("kb-parse-code::KotlinAstExtractor::extract (code:kotlin)"),
// 9 AST lang: rust / python / typescript / javascript / go / java / kotlin / c / cpp
"rust" | "python" | "typescript" | "javascript" | "go" | "java" | "kotlin" | "c"
| "cpp" => app
.extract_for(&asset.media_type, &ctx, &bytes)
.with_context(|| format!("kb-app::extract_for (code:{code_lang})")),
// p10-2 Tier 2: no extractor — synthesize Document directly from raw bytes.
"yaml" | "dockerfile" | "toml" | "json" | "xml" | "groovy" | "go-mod" => {
synthesize_tier2_document(asset, &bytes, code_lang, &parser_version)
@@ -1987,6 +2095,13 @@ fn ingest_one_code_asset(
"shell" => CodeTextParagraphV1Chunker
.chunk(&canonical, chunk_policy)
.context("kb-chunk::CodeTextParagraphV1Chunker::chunk (code:shell)"),
// p10-1D: C + C++ AST chunkers.
"c" => CodeCAstV1Chunker
.chunk(&canonical, chunk_policy)
.context("kebab-chunk::CodeCAstV1Chunker::chunk (code:c)"),
"cpp" => CodeCppAstV1Chunker
.chunk(&canonical, chunk_policy)
.context("kebab-chunk::CodeCppAstV1Chunker::chunk (code:cpp)"),
other => anyhow::bail!("unreachable (chunk): {other}"),
}
};
@@ -2263,7 +2378,7 @@ fn lang_hint_from_doc(doc: &CanonicalDocument) -> Option<Lang> {
/// Convenience: end byte of the frontmatter region (or 0 when absent).
fn fm_span_end(span: Option<kebab_parse_md::FrontmatterSpan>) -> usize {
span.map(|s| s.end).unwrap_or(0)
span.map_or(0, |s| s.end)
}
/// Count `\n` in a byte prefix to convert frontmatter byte span to
@@ -2566,8 +2681,7 @@ pub fn ingest_file_with_config(
const SUPPORTED_EXTS: &[&str] = &["md", "pdf", "png", "jpg", "jpeg"];
if !SUPPORTED_EXTS.contains(&ext.as_str()) {
anyhow::bail!(
"ingest-file: unsupported extension `.{}` (supported: {:?})",
ext, SUPPORTED_EXTS
"ingest-file: unsupported extension `.{ext}` (supported: {SUPPORTED_EXTS:?})"
);
}

View File

@@ -63,14 +63,26 @@ pub struct Stats {
/// p9-fb-37: docs whose `updated_at` exceeds the staleness threshold.
#[serde(default)]
pub stale_doc_count: u64,
/// p10-1A-1: code language breakdown (chunk counts by canonical lowercase
/// language identifier). Empty until 1A-2 produces code chunks.
/// p10-1A-1: code language breakdown (**doc** counts by canonical
/// lowercase language identifier). Empty until 1A-2 produces code
/// docs. v0.17.0 PR-C: doc-count semantics corrected here (the
/// previous "chunk counts" wording was a longstanding mis-label —
/// implementation has always been `COUNT(*) FROM documents
/// GROUP BY code_lang`). Use `code_lang_chunk_breakdown` for the
/// chunk-level companion.
#[serde(default)]
pub code_lang_breakdown: std::collections::BTreeMap<String, u32>,
/// p10-1A-1: repo breakdown (chunk counts by `metadata.repo` value).
/// Empty until 1A-2 produces code chunks.
/// p10-1A-1: repo breakdown (**doc** counts by `metadata.repo`
/// value). Empty until 1A-2 produces code docs. v0.17.0 PR-C:
/// doc-count wording corrected (mirror of code_lang_breakdown).
#[serde(default)]
pub repo_breakdown: std::collections::BTreeMap<String, u32>,
/// v0.17.0 PR-C: sister of [`Self::code_lang_breakdown`] returning
/// chunk counts instead of doc counts. Indexing-pressure metric —
/// one PDF spec → 200 chunks vs one Rust file → 5 chunks shows up
/// here in a way `code_lang_breakdown` (doc count) hides.
#[serde(default)]
pub code_lang_chunk_breakdown: std::collections::BTreeMap<String, u32>,
}
const KEBAB_VERSION: &str = env!("CARGO_PKG_VERSION");
@@ -153,7 +165,7 @@ fn collect_stats(
store: &kebab_store_sqlite::SqliteStore,
) -> anyhow::Result<Stats> {
let counts = store
.count_summary_with_threshold(cfg.search.stale_threshold_days as u64)?;
.count_summary_with_threshold(u64::from(cfg.search.stale_threshold_days))?;
let data_dir = kebab_config::expand_path(&cfg.storage.data_dir, "");
let index_bytes = kebab_store_sqlite::stats_ext::index_bytes(&data_dir)
.map_err(|e| anyhow::anyhow!("index_bytes: {e}"))?;
@@ -171,6 +183,9 @@ fn collect_stats(
// p10-1A-2 follow-up: dogfooding (2026-05-20) revealed this was a
// placeholder — mirror of code_lang_breakdown for the repo field.
repo_breakdown: store.repo_breakdown()?,
// v0.17.0 PR-C: chunk-level companion (closes HOTFIXES
// 2026-05-22 "code_lang_breakdown chunk granularity" LOW).
code_lang_chunk_breakdown: store.code_lang_chunk_breakdown()?,
})
}
@@ -210,6 +225,11 @@ mod tests_stats_ext {
v.get("repo_breakdown").is_some(),
"Stats JSON must include repo_breakdown: {v}"
);
// v0.17.0 PR-C: chunk-level companion field.
assert!(
v.get("code_lang_chunk_breakdown").is_some(),
"Stats JSON must include code_lang_chunk_breakdown (v0.17.0 PR-C): {v}"
);
// Empty BTreeMap serializes as `{}` — confirm it's an object, not null.
assert!(
v["code_lang_breakdown"].is_object(),
@@ -219,6 +239,10 @@ mod tests_stats_ext {
v["repo_breakdown"].is_object(),
"repo_breakdown must be an object: {v}"
);
assert!(
v["code_lang_chunk_breakdown"].is_object(),
"code_lang_chunk_breakdown must be an object: {v}"
);
}
#[test]

View File

@@ -33,6 +33,7 @@ fn ask_lexical_smoke() {
history: Vec::new(),
conversation_id: None,
turn_index: None,
multi_hop: false,
};
// The fixture workspace contains "ownership" content; the model's
// citation behavior depends on its training, so we don't assert on

View File

@@ -1064,3 +1064,328 @@ fn rust_file_re_ingest_is_unchanged() {
);
assert_eq!(item2.doc_id, item1.doc_id);
}
/// p10-3 fix regression: a docker-compose YAML that falls back to Tier 3
/// (k8s chunker returns empty, CodeTextParagraphV1Chunker retries) must
/// report Unchanged on the second ingest rather than re-processing.
/// Before the fix, try_skip_unchanged returned None because the stored
/// last_chunker_version ("code-text-paragraph-v1" / parser_version
/// "none-v1") never matched the caller's dispatch values.
#[test]
fn tier3_yaml_fallback_reingest_is_unchanged() {
let env = TestEnv::lexical_only();
std::fs::write(
env.workspace_root.join("docker-compose.yml"),
"version: '3'\nservices:\n api:\n image: nginx:latest\n",
)
.unwrap();
let report1 =
kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("first ingest");
let item1 = report1
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("docker-compose.yml"))
.expect("docker-compose.yml in first report");
assert!(
matches!(item1.kind, IngestItemKind::New),
"first ingest must be New, got {:?}", item1.kind
);
assert_eq!(
item1.chunker_version.as_ref().map(|c| c.0.as_str()),
Some("code-text-paragraph-v1"),
"first ingest must use Tier 3 fallback chunker"
);
let report2 =
kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("second ingest");
let item2 = report2
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("docker-compose.yml"))
.expect("docker-compose.yml in second report");
assert!(
matches!(item2.kind, IngestItemKind::Unchanged),
"second ingest must be Unchanged, got {:?}", item2.kind
);
}
/// p10-1d Task G: a `.c` file with a single top-level function is ingested
/// and the resulting `Citation::Code` hit must carry `lang="c"`,
/// `symbol="parse_record"` (function name only — no nesting in C), and
/// `chunker_version = "code-c-ast-v1"`.
#[test]
fn tier1_c_ingest_searchable() {
let env = TestEnv::lexical_only();
std::fs::write(
env.workspace_root.join("parser.c"),
"#include <stdio.h>\n\nint parse_record(const char *line) {\n if (line == NULL) return -1;\n return 0;\n}\n",
)
.unwrap();
let report = kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("ingest must succeed");
assert_eq!(report.errors, 0, "no ingest errors: {report:?}");
assert!(report.new >= 1, "c file ingested: {report:?}");
let c_item = report
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("parser.c"))
.expect("parser.c item present");
assert_eq!(
c_item.parser_version.as_ref().map(|p| p.0.as_str()),
Some("code-c-v2"),
"parser_version must be code-c-v2 (v0.17.0 PR-B: typedef-wrapped struct/enum/union 이 typedef alias unit 으로 방출)"
);
assert_eq!(
c_item.chunker_version.as_ref().map(|c| c.0.as_str()),
Some("code-c-ast-v1"),
"chunker_version must be code-c-ast-v1"
);
let query = kebab_core::SearchQuery {
text: "parse_record".to_string(),
mode: kebab_core::SearchMode::Lexical,
k: 10,
filters: kebab_core::SearchFilters {
code_lang: vec!["c".to_string()],
..Default::default()
},
};
let hits = kebab_app::search_with_config(env.config.clone(), query)
.expect("search must succeed");
let h = hits
.iter()
.find(|h| matches!(&h.citation, Citation::Code { .. }))
.expect("at least one Citation::Code hit for 'parse_record'");
match &h.citation {
Citation::Code {
lang,
symbol,
line_start,
..
} => {
assert_eq!(lang.as_deref(), Some("c"), "citation.lang must be 'c'");
assert_eq!(
symbol.as_deref(),
Some("parse_record"),
"C symbol must be function name only (no nesting)"
);
assert!(*line_start >= 1, "line_start must be >=1");
}
_ => unreachable!(),
}
assert_eq!(
h.code_lang.as_deref(),
Some("c"),
"SearchHit.code_lang must be 'c'"
);
assert_eq!(
h.chunker_version.0.as_str(),
"code-c-ast-v1",
"C chunks must be stamped with code-c-ast-v1"
);
}
/// p10-1d Task G: a `.cpp` file with nested namespace + class is ingested
/// and the resulting `Citation::Code` hit must carry `lang="cpp"`, a
/// `symbol` that starts with `"kebab::chunk::Foo"` (namespace::Class or
/// namespace::Class::method), and `chunker_version = "code-cpp-ast-v1"`.
#[test]
fn tier1_cpp_ingest_searchable() {
let env = TestEnv::lexical_only();
std::fs::write(
env.workspace_root.join("chunker.cpp"),
"namespace kebab {\nnamespace chunk {\nclass Foo {\npublic:\n void bar() { /* impl */ }\n};\n}\n}\n",
)
.unwrap();
let report = kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("ingest must succeed");
assert_eq!(report.errors, 0, "no ingest errors: {report:?}");
assert!(report.new >= 1, "cpp file ingested: {report:?}");
let cpp_item = report
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("chunker.cpp"))
.expect("chunker.cpp item present");
assert_eq!(
cpp_item.parser_version.as_ref().map(|p| p.0.as_str()),
Some("code-cpp-v1"),
"parser_version must be code-cpp-v1"
);
assert_eq!(
cpp_item.chunker_version.as_ref().map(|c| c.0.as_str()),
Some("code-cpp-ast-v1"),
"chunker_version must be code-cpp-ast-v1"
);
let query = kebab_core::SearchQuery {
text: "bar".to_string(),
mode: kebab_core::SearchMode::Lexical,
k: 10,
filters: kebab_core::SearchFilters {
code_lang: vec!["cpp".to_string()],
..Default::default()
},
};
let hits = kebab_app::search_with_config(env.config.clone(), query)
.expect("search must succeed");
let h = hits
.iter()
.find(|h| matches!(&h.citation, Citation::Code { .. }))
.expect("at least one Citation::Code hit for 'bar'");
match &h.citation {
Citation::Code {
lang,
symbol,
line_start,
..
} => {
assert_eq!(lang.as_deref(), Some("cpp"), "citation.lang must be 'cpp'");
// Symbol could be "kebab::chunk::Foo" (class) or "kebab::chunk::Foo::bar"
// (method) depending on which chunk ranks first.
assert!(
symbol.as_deref().is_some_and(|s| s.starts_with("kebab::chunk::Foo")),
"C++ symbol must start with namespace::Class prefix, got {symbol:?}"
);
assert!(*line_start >= 1, "line_start must be >=1");
}
_ => unreachable!(),
}
assert_eq!(
h.code_lang.as_deref(),
Some("cpp"),
"SearchHit.code_lang must be 'cpp'"
);
assert_eq!(
h.chunker_version.0.as_str(),
"code-cpp-ast-v1",
"C++ chunks must be stamped with code-cpp-ast-v1"
);
}
/// P10 dogfood regression: a k8s YAML with 2 documents (Deployment + Service
/// separated by `---`) must ingest without a UNIQUE constraint violation.
/// Before the fix, push_chunks_with_oversize emitted split_key=None for each
/// resource, giving every resource chunk the same id_hash → identical chunk_id
/// → SQLite UNIQUE constraint failure on the second resource.
#[test]
fn tier2_k8s_multi_resource_yaml_ingests_without_collision() {
let env = TestEnv::lexical_only();
let k8s_dir = env.workspace_root.join("k8s");
std::fs::create_dir_all(&k8s_dir).unwrap();
std::fs::write(
k8s_dir.join("k8s-multi.yaml"),
"apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: api\n namespace: prod\nspec:\n replicas: 2\n---\napiVersion: v1\nkind: Service\nmetadata:\n name: api\n namespace: prod\nspec:\n selector:\n app: api\n",
)
.unwrap();
let report = kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("ingest must succeed");
// The bug: this would land in report with an error + UNIQUE constraint message.
let item = report
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("k8s-multi.yaml"))
.expect("k8s-multi.yaml in report");
assert!(
item.error.is_none(),
"multi-resource k8s yaml must ingest without error, got: {:?}",
item.error
);
assert!(
matches!(item.kind, IngestItemKind::New),
"expected New, got {:?}",
item.kind
);
// Both resources must be searchable (≥2 hits: Deployment/prod/api + Service/prod/api).
let query = kebab_core::SearchQuery {
text: "api".to_string(),
mode: kebab_core::SearchMode::Lexical,
k: 10,
filters: kebab_core::SearchFilters {
code_lang: vec!["yaml".to_string()],
..Default::default()
},
};
let hits = kebab_app::search_with_config(env.config.clone(), query)
.expect("search must succeed");
assert!(
hits.len() >= 2,
"expected ≥2 hits (Deployment + Service), got {}",
hits.len()
);
}
/// p10-3 fix regression: a shell file (direct Tier 3, not a fallback)
/// must also report Unchanged on re-ingest. Shell goes straight to
/// CodeTextParagraphV1Chunker so `stored_is_tier3_fallback` is false
/// (parser_version is "none-v1" and chunker matches the current dispatch),
/// but the normal equality path should pass regardless.
#[test]
fn tier3_shell_reingest_is_unchanged() {
let env = TestEnv::lexical_only();
std::fs::write(
env.workspace_root.join("deploy.sh"),
"#!/usr/bin/env bash\nset -e\necho hello\n",
)
.unwrap();
let report1 =
kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("first ingest");
let item1 = report1
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("deploy.sh"))
.expect("deploy.sh in first report");
assert!(
matches!(item1.kind, IngestItemKind::New),
"first ingest must be New, got {:?}", item1.kind
);
let report2 =
kebab_app::ingest_with_config(env.config.clone(), env.scope(), false)
.expect("second ingest");
let item2 = report2
.items
.as_ref()
.expect("items present")
.iter()
.find(|i| i.doc_path.0.ends_with("deploy.sh"))
.expect("deploy.sh in second report");
assert!(
matches!(item2.kind, IngestItemKind::Unchanged),
"shell reingest must be Unchanged, got {:?}", item2.kind
);
}

View File

@@ -38,12 +38,16 @@ fn fetch_chunk_returns_target_only_when_no_context() {
#[test]
fn fetch_chunk_with_context_returns_neighbors() {
let env = common::TestEnv::new();
let body = "# H1\n\nA1\n\n# H2\n\nA2\n\n# H3\n\nA3\n\n# H4\n\nA4\n\n# H5\n\nA5\n";
// v0.17.0 trigram tokenizer: terms must be ≥3 Unicode chars to
// match. The earlier fixture used 2-char tokens like `A1`/`A3` for
// section bodies — those zero-hit under trigram. Use 5-char unique
// words per section so the query can pin one chunk deterministically.
let body = "# H1\n\napples\n\n# H2\n\nbanana\n\n# H3\n\ncherry\n\n# H4\n\ndurian\n\n# H5\n\nelder\n";
common::ingest_md(&env, "multi.md", body);
let app = env.app();
let q = kebab_core::SearchQuery {
text: "A3".to_string(),
text: "cherry".to_string(),
mode: kebab_core::SearchMode::Lexical,
k: 1,
filters: kebab_core::SearchFilters::default(),

View File

@@ -33,7 +33,7 @@ fn ingest_file_copies_external_md_and_reports_new() {
assert!(ext_dir.is_dir());
let entries: Vec<_> = fs::read_dir(&ext_dir)
.unwrap()
.filter_map(|e| e.ok())
.filter_map(std::result::Result::ok)
.collect();
assert_eq!(entries.len(), 1, "exactly one file in _external/");
let name = entries[0].file_name().to_string_lossy().into_owned();

View File

@@ -35,7 +35,7 @@ fn ingest_stdin_writes_frontmatter_and_reports_new() {
// _external/ contains exactly one .md file with frontmatter.
let ext_dir = std::path::PathBuf::from(&cfg.workspace.root).join("_external");
let entries: Vec<_> = fs::read_dir(&ext_dir).unwrap()
.filter_map(|e| e.ok())
.filter_map(std::result::Result::ok)
.collect();
assert_eq!(entries.len(), 1);
let content = fs::read_to_string(entries[0].path()).unwrap();
@@ -60,7 +60,7 @@ fn ingest_stdin_without_source_uri() {
let ext_dir = std::path::PathBuf::from(&cfg.workspace.root).join("_external");
let entries: Vec<_> = fs::read_dir(&ext_dir).unwrap()
.filter_map(|e| e.ok())
.filter_map(std::result::Result::ok)
.collect();
let content = fs::read_to_string(entries[0].path()).unwrap();
assert!(content.contains("title: \"Title\""));

View File

@@ -0,0 +1,81 @@
//! Tests for `App::open_with_config`'s NLI verifier construction path.
//!
//! Coverage:
//! 1. `open_with_config_nli_fails_when_model_dir_unwritable_and_threshold_positive` —
//! when `rag.nli_threshold > 0` and `storage.model_dir` is unwritable,
//! `open_with_config` returns `Err` with "OnnxNliVerifier" in the
//! error chain.
//! 2. `open_with_config_nli_skipped_when_threshold_zero` —
//! same bad `model_dir`, but `rag.nli_threshold = 0.0` (gate disabled),
//! so `OnnxNliVerifier::new` is never called and `open_with_config`
//! succeeds.
//!
//! `/proc/1/root` is the init process's filesystem root; on Linux it is
//! owned by root and not traversable by unprivileged users, making
//! `create_dir_all` fail with `EACCES` — a reliable "unwritable path"
//! that requires no test setup beyond the path literal.
use kebab_config::Config;
/// Return a `Config` whose `data_dir` lives in a fresh `TempDir`
/// (so `SqliteStore::open` succeeds) and whose `model_dir` is set to
/// `/proc/1/root` (unwritable by non-root processes on Linux).
///
/// The `TempDir` is returned alongside the config so the caller keeps
/// it alive until the test completes — dropping it early would delete
/// the data directory before any assertions run.
fn config_with_unwritable_model_dir() -> (tempfile::TempDir, Config) {
let tmp = tempfile::tempdir().expect("tempdir");
let mut cfg = Config::defaults();
// Valid data_dir → SqliteStore::open + run_migrations succeed.
cfg.storage.data_dir = tmp.path().to_string_lossy().into_owned();
// /proc/1/root is only accessible to root; create_dir_all will
// return EACCES for any unprivileged user, which is exactly the
// failure mode we want to exercise.
cfg.storage.model_dir = "/proc/1/root".to_string();
(tmp, cfg)
}
// ── 1. Failure path: threshold > 0 + unwritable model_dir ─────────────────
#[test]
fn open_with_config_nli_fails_when_model_dir_unwritable_and_threshold_positive() {
let (_tmp, mut cfg) = config_with_unwritable_model_dir();
cfg.rag.nli_threshold = 0.5; // gate enabled → OnnxNliVerifier::new runs
let result = kebab_app::App::open_with_config(cfg);
let Err(err) = result else {
panic!(
"App::open_with_config must fail when model_dir is unwritable and nli_threshold > 0"
);
};
// The error chain must identify the OnnxNliVerifier as the source so
// an operator reading logs can trace the failure to the NLI config.
let err_chain = format!("{err:?}");
assert!(
err_chain.contains("OnnxNliVerifier"),
"error chain must mention OnnxNliVerifier; full chain: {err_chain}"
);
}
// ── 2. Success path: threshold = 0.0 → NLI verifier never constructed ──────
#[test]
fn open_with_config_nli_skipped_when_threshold_zero() {
let (_tmp, cfg) = config_with_unwritable_model_dir();
// Default nli_threshold is 0.0 — gate disabled, verifier skipped.
assert!(
(cfg.rag.nli_threshold - 0.0).abs() < f32::EPSILON,
"precondition: default nli_threshold must be 0.0 (gate disabled)"
);
// A bad model_dir must NOT cause a failure when the NLI gate is off.
let result = kebab_app::App::open_with_config(cfg);
assert!(
result.is_ok(),
"App::open_with_config must succeed when nli_threshold = 0.0 \
(OnnxNliVerifier is never constructed); err: {:?}",
result.err()
);
}

View File

@@ -46,3 +46,88 @@ fn korean_lexical_query_returns_korean_document() {
hits.iter().map(|h| &h.doc_path.0).collect::<Vec<_>>()
);
}
/// A4 Step 1c — multi-token Korean query (`해시 충돌`) must hit when
/// the lexical builder routes it through a whole-phrase MATCH candidate.
///
/// Expected: FAIL until A5 (`build_match_string` redesign) lands — the
/// current builder emits `"해시" "충돌"` AND, but FTS5 trigram tokenizer
/// has no 2-char terms so each side is 0-hit. A5 introduces a whole-
/// phrase candidate (`"해시 충돌"`) OR'd with the token AND, restoring
/// hits for the dominant Korean usage pattern.
#[test]
fn lexical_multi_token_korean_query_hits() {
let env = TestEnv::lexical_only();
// Copy the synthetic Korean fixture (introduced in A4 Step 0) into
// the test workspace. The fixture contains the exact phrase
// "해시 충돌" multiple times.
let dest = env.workspace_root.join("hash-table.md");
let src = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("..")
.join("..")
.join("fixtures")
.join("search")
.join("korean")
.join("hash-table.md");
std::fs::copy(&src, &dest).expect("copy korean fixture");
kebab_app::ingest_with_config(env.config.clone(), env.scope(), true)
.expect("ingest must succeed");
let hits = kebab_app::search_with_config(
env.config.clone(),
common::lexical_query("해시 충돌"),
)
.expect("search must succeed");
assert!(
!hits.is_empty(),
"multi-token Korean query '해시 충돌' must hit the hash-table fixture; got {:?}",
hits.iter().map(|h| &h.doc_path.0).collect::<Vec<_>>()
);
let any_hash_table = hits.iter().any(|h| h.doc_path.0.contains("hash-table"));
assert!(
any_hash_table,
"expected at least one hit on the hash-table fixture, got: {:?}",
hits.iter().map(|h| &h.doc_path.0).collect::<Vec<_>>()
);
}
/// A4 Step 1c — mixed Korean+English multi-token query (`Rust 충돌은`).
/// Both tokens are ≥3 chars, so the redesigned builder (A5) emits
/// `("Rust 충돌은") OR ("Rust" AND "충돌은")`. With trigram tokenizer
/// each side has substring coverage in the document, so the AND branch
/// alone is enough. Expected: FAIL pre-A5, PASS post-A5.
#[test]
fn lexical_mixed_korean_english_multi_token_query_hits() {
let env = TestEnv::lexical_only();
let doc_path = env.workspace_root.join("rust-hash.md");
std::fs::write(
&doc_path,
"# Rust 해시 테이블\n\nRust 의 std::collections::HashMap 에서 \
해시 충돌은 SipHash 로 완화한다.\n",
)
.expect("write rust-hash fixture");
kebab_app::ingest_with_config(env.config.clone(), env.scope(), true)
.expect("ingest must succeed");
let hits = kebab_app::search_with_config(
env.config.clone(),
common::lexical_query("Rust 충돌은"),
)
.expect("search must succeed");
assert!(
!hits.is_empty(),
"mixed Korean+English multi-token query 'Rust 충돌은' must hit the rust-hash fixture; got {:?}",
hits.iter().map(|h| &h.doc_path.0).collect::<Vec<_>>()
);
let any_rust_hash = hits.iter().any(|h| h.doc_path.0.contains("rust-hash"));
assert!(
any_rust_hash,
"expected at least one hit on the rust-hash fixture, got: {:?}",
hits.iter().map(|h| &h.doc_path.0).collect::<Vec<_>>()
);
}

View File

@@ -14,12 +14,10 @@ use common::TestEnv;
fn require_avx_or_panic() {
#[cfg(target_arch = "x86_64")]
{
if !std::is_x86_feature_detected!("avx") {
panic!(
"kb-app vector integration test requires AVX-capable hardware; \
host CPU lacks AVX. Run on an AVX-capable machine."
);
}
assert!(std::is_x86_feature_detected!("avx"),
"kb-app vector integration test requires AVX-capable hardware; \
host CPU lacks AVX. Run on an AVX-capable machine."
);
}
}

View File

@@ -16,12 +16,16 @@ tracing = { workspace = true }
serde_yaml = { workspace = true }
[dev-dependencies]
# kb-parse-md / kb-normalize are dev-only — used by the snapshot integration
# test to build a CanonicalDocument from a fixture Markdown file. Forbidden as
# regular deps per design §8 (chunker consumes CanonicalDocument from kb-core
# only); `cargo tree -p kb-chunk --depth 1` (default scope, excludes dev-deps)
# kb-parse-md / kb-parse-code are dev-only — used by the snapshot integration
# tests to build a CanonicalDocument from fixture files. kb-parse-md absorbed
# kb-normalize in v0.19.0 (HOTFIXES.md 2026-05-26). Forbidden as regular deps
# per design §8 (chunker consumes CanonicalDocument from kb-core only);
# `cargo tree -p kb-chunk --depth 1` (default scope, excludes dev-deps)
# confirms this.
kebab-parse-md = { path = "../kebab-parse-md" }
kebab-normalize = { path = "../kebab-normalize" }
serde_json = { workspace = true }
time = { workspace = true }
kebab-parse-md = { path = "../kebab-parse-md" }
kebab-parse-code = { path = "../kebab-parse-code" }
serde_json = { workspace = true }
time = { workspace = true }
[lints]
workspace = true

View File

@@ -0,0 +1,322 @@
//! `code-c-ast-v1` — maps a tree-sitter-derived C AST
//! `CanonicalDocument` (one `Block::Code` per semantic unit, each with
//! `SourceSpan::Code`) to chunks 1:1. A unit longer than
//! `AST_CHUNK_MAX_LINES` is split into `<symbol> [part i/N]` sub-chunks
//! at blank-line paragraph boundaries (design §9.1 oversize fallback).
//!
//! tree-sitter is intentionally NOT a dependency here: AST work is
//! parser-side (`kebab-parse-code`, design §6.3). This chunker only
//! consumes the `CanonicalDocument`.
//!
//! `AST_CHUNK_MAX_LINES` is a constant matching
//! `IngestCodeCfg::default().ast_chunk_max_lines` (200). Per-medium
//! config threading needs a chunker registry (P+); same deviation
//! pattern as `pdf-page-v1`'s pinned `chunker_version`
//! (`tasks/HOTFIXES.md`).
use kebab_core::{
Block, BlockId, CanonicalDocument, Chunk, ChunkPolicy, Chunker, ChunkerVersion, DocumentId,
SourceSpan, id_for_chunk,
};
const VERSION_LABEL: &str = "code-c-ast-v1";
const BYTES_PER_TOKEN: usize = 3;
const POLICY_HASH_HEX_LEN: usize = 16;
const AST_CHUNK_MAX_LINES: u32 = 200;
#[derive(Clone, Copy, Debug, Default)]
pub struct CodeCAstV1Chunker;
impl Chunker for CodeCAstV1Chunker {
fn chunker_version(&self) -> ChunkerVersion {
ChunkerVersion(VERSION_LABEL.to_string())
}
fn policy_hash(&self, policy: &ChunkPolicy) -> String {
let bytes = serde_json_canonicalizer::to_vec(policy)
.expect("canonical JSON serialization of ChunkPolicy must not fail");
let hex = blake3::hash(&bytes).to_hex().to_string();
hex[..POLICY_HASH_HEX_LEN].to_string()
}
fn chunk(
&self,
doc: &CanonicalDocument,
policy: &ChunkPolicy,
) -> anyhow::Result<Vec<Chunk>> {
for b in &doc.blocks {
let c = match b {
Block::Code(c) => c,
_ => anyhow::bail!(
"CodeCAstV1Chunker only handles code docs (got non-Code block)"
),
};
if !matches!(c.common.source_span, SourceSpan::Code { .. }) {
anyhow::bail!(
"CodeCAstV1Chunker only handles code docs (got non-Code source_span)"
);
}
}
let base_policy_hash = self.policy_hash(policy);
let chunker_version = self.chunker_version();
let mut out: Vec<Chunk> = Vec::new();
for b in &doc.blocks {
let cb = match b {
Block::Code(c) => c,
_ => unreachable!("validated above"),
};
let (ls, le, symbol, lang) = match &cb.common.source_span {
SourceSpan::Code { line_start, line_end, symbol, lang } => {
(*line_start, *line_end, symbol.clone(), lang.clone())
}
_ => unreachable!("validated above"),
};
let block_ids: Vec<BlockId> = vec![cb.common.block_id.clone()];
let span_lines = le.saturating_sub(ls) + 1;
if span_lines <= AST_CHUNK_MAX_LINES {
let span = SourceSpan::Code {
line_start: ls,
line_end: le,
symbol: symbol.clone(),
lang: lang.clone(),
};
out.push(make_chunk(
doc, &chunker_version, &block_ids, &base_policy_hash,
None, span, cb.code.clone(),
));
} else {
let parts = split_oversize(&cb.code);
let n = parts.len();
for (i, (off_start, off_end, text)) in parts.into_iter().enumerate() {
let part_ls = ls + off_start;
let part_le = ls + off_end;
let part_sym = symbol
.as_ref()
.map(|s| format!("{s} [part {}/{n}]", i + 1));
let span = SourceSpan::Code {
line_start: part_ls,
line_end: part_le,
symbol: part_sym,
lang: lang.clone(),
};
out.push(make_chunk(
doc, &chunker_version, &block_ids, &base_policy_hash,
Some(part_ls), span, text,
));
}
}
}
tracing::debug!(
target: "kebab-chunk",
doc_id = %doc.doc_id,
chunks = out.len(),
"code-c-ast-v1 chunked",
);
Ok(out)
}
}
#[allow(clippy::too_many_arguments)]
fn make_chunk(
doc: &CanonicalDocument,
chunker_version: &ChunkerVersion,
block_ids: &[BlockId],
base_policy_hash: &str,
split_key: Option<u32>,
span: SourceSpan,
text: String,
) -> Chunk {
let id_hash = match split_key {
Some(k) => format!("{base_policy_hash}#L{k}"),
None => base_policy_hash.to_string(),
};
let chunk_id = id_for_chunk(&doc.doc_id, chunker_version, block_ids, &id_hash);
let token_estimate = text.len().div_ceil(BYTES_PER_TOKEN);
Chunk {
chunk_id,
doc_id: DocumentId(doc.doc_id.0.clone()),
block_ids: block_ids.to_vec(),
text,
heading_path: Vec::new(),
source_spans: vec![span],
token_estimate,
chunker_version: chunker_version.clone(),
policy_hash: base_policy_hash.to_string(),
}
}
/// Split an oversize unit at blank-line paragraph boundaries, greedily
/// gluing paragraphs until ~`AST_CHUNK_MAX_LINES` lines accumulate.
/// Returns `(line_offset_start, line_offset_end, text)` where offsets are
/// 0-based within the unit (caller adds the unit's absolute `line_start`).
fn split_oversize(code: &str) -> Vec<(u32, u32, String)> {
let lines: Vec<&str> = code.split('\n').collect();
let total = lines.len() as u32;
let mut out: Vec<(u32, u32, String)> = Vec::new();
let mut start: u32 = 0;
while start < total {
let mut end = (start + AST_CHUNK_MAX_LINES).min(total);
let floor = start + (AST_CHUNK_MAX_LINES * 4 / 5);
if end < total {
if let Some(b) = (floor.min(end)..end)
.rev()
.find(|&i| lines[i as usize].trim().is_empty())
{
end = b + 1;
}
}
let text = lines[start as usize..end as usize].join("\n");
out.push((start, end.saturating_sub(1), text));
start = end;
}
if out.is_empty() {
out.push((0, total.saturating_sub(1), code.to_string()));
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use kebab_core::{
Block, CanonicalDocument, ChunkPolicy, Chunker, ChunkerVersion, CodeBlock, CommonBlock,
SourceSpan, id_for_block, id_for_doc, AssetId, Lang, Metadata, ParserVersion, Provenance,
SourceType, TrustLevel, WorkspacePath,
};
use time::OffsetDateTime;
fn code_doc(units: &[(&str, u32, u32, &str)]) -> CanonicalDocument {
let wp = WorkspacePath("crates/x/src/a.c".into());
let aid = AssetId("a".repeat(64));
let pv = ParserVersion("code-c-v1".into());
let doc_id = id_for_doc(&wp, &aid, &pv);
let blocks = units
.iter()
.enumerate()
.map(|(i, (sym, ls, le, code))| {
let span = SourceSpan::Code {
line_start: *ls,
line_end: *le,
symbol: Some((*sym).to_string()),
lang: Some("c".into()),
};
let bid = id_for_block(&doc_id, "code", &[], i as u32, &span);
Block::Code(CodeBlock {
common: CommonBlock { block_id: bid, heading_path: vec![], source_span: span },
lang: Some("c".into()),
code: (*code).to_string(),
})
})
.collect();
CanonicalDocument {
doc_id, source_asset_id: aid, workspace_path: wp, title: "a".into(),
lang: Lang("und".into()), blocks,
metadata: Metadata {
aliases: vec![], tags: vec![],
created_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
updated_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
source_type: SourceType::Note, trust_level: TrustLevel::Primary,
user_id_alias: None, user: Default::default(),
repo: Some("kebab".into()), git_branch: Some("main".into()),
git_commit: Some("0".repeat(40)), code_lang: Some("c".into()),
},
provenance: Provenance { events: vec![] },
parser_version: pv, schema_version: 1, doc_version: 1,
last_chunker_version: None, last_embedding_version: None,
}
}
fn policy() -> ChunkPolicy {
ChunkPolicy { target_tokens: 500, overlap_tokens: 80,
respect_markdown_headings: false,
chunker_version: ChunkerVersion(VERSION_LABEL.into()) }
}
#[test]
fn chunker_version_is_code_c_ast_v1() {
assert_eq!(CodeCAstV1Chunker.chunker_version(),
ChunkerVersion("code-c-ast-v1".into()));
}
#[test]
fn one_chunk_per_unit_preserves_code_span() {
let doc = code_doc(&[
("parse", 1, 3, "int parse() {\n\t// x\n}"),
("print", 5, 7, "void print() {\n\t//\n\treturn;\n}"),
]);
let chunks = CodeCAstV1Chunker.chunk(&doc, &policy()).unwrap();
assert_eq!(chunks.len(), 2);
for c in &chunks {
assert_eq!(c.source_spans.len(), 1);
assert!(matches!(c.source_spans[0], SourceSpan::Code { .. }));
assert_eq!(c.heading_path, Vec::<String>::new());
assert_eq!(c.chunker_version.0, "code-c-ast-v1");
}
match &chunks[0].source_spans[0] {
SourceSpan::Code { symbol, line_start, line_end, .. } => {
assert_eq!(symbol.as_deref(), Some("parse"));
assert_eq!((*line_start, *line_end), (1, 3));
}
_ => unreachable!(),
}
}
#[test]
fn oversize_unit_splits_into_parts_with_unique_ids() {
let body = (0..500).map(|i| format!("\tx{i} = {i};\n")).collect::<String>();
let code = format!("int big() {{\n{body}\n}}");
let doc = code_doc(&[("big", 1, 502, &code)]);
let chunks = CodeCAstV1Chunker.chunk(&doc, &policy()).unwrap();
assert!(chunks.len() >= 2, "oversize unit must split, got {}", chunks.len());
for c in &chunks {
match &c.source_spans[0] {
SourceSpan::Code { symbol, .. } => {
assert!(symbol.as_deref().unwrap().starts_with("big [part "),
"part-numbered symbol, got {symbol:?}");
}
_ => unreachable!(),
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}
#[test]
fn non_code_doc_errors() {
use kebab_core::TextBlock;
let mut doc = code_doc(&[("parse", 1, 1, "int parse() {}")]);
doc.blocks = vec![Block::Paragraph(TextBlock {
common: CommonBlock {
block_id: kebab_core::BlockId("b".into()),
heading_path: vec![],
source_span: SourceSpan::Line { start: 1, end: 1 },
},
text: "x".into(), inlines: vec![],
})];
let err = CodeCAstV1Chunker.chunk(&doc, &policy()).unwrap_err();
assert!(err.to_string().contains("CodeCAstV1Chunker"));
}
#[test]
fn deterministic_chunk_ids_1000() {
let doc = code_doc(&[("parse", 1, 2, "int parse() {}\n")]);
let base: Vec<String> = CodeCAstV1Chunker.chunk(&doc, &policy())
.unwrap().into_iter().map(|c| c.chunk_id.0).collect();
for _ in 0..1000 {
let again: Vec<String> = CodeCAstV1Chunker.chunk(&doc, &policy())
.unwrap().into_iter().map(|c| c.chunk_id.0).collect();
assert_eq!(again, base);
}
}
#[test]
fn policy_hash_matches_md_heading_v1() {
let p = policy();
assert_eq!(CodeCAstV1Chunker.policy_hash(&p),
crate::MdHeadingV1Chunker.policy_hash(&p));
}
}

View File

@@ -0,0 +1,322 @@
//! `code-cpp-ast-v1` — maps a tree-sitter-derived C++ AST
//! `CanonicalDocument` (one `Block::Code` per semantic unit, each with
//! `SourceSpan::Code`) to chunks 1:1. A unit longer than
//! `AST_CHUNK_MAX_LINES` is split into `<symbol> [part i/N]` sub-chunks
//! at blank-line paragraph boundaries (design §9.1 oversize fallback).
//!
//! tree-sitter is intentionally NOT a dependency here: AST work is
//! parser-side (`kebab-parse-code`, design §6.3). This chunker only
//! consumes the `CanonicalDocument`.
//!
//! `AST_CHUNK_MAX_LINES` is a constant matching
//! `IngestCodeCfg::default().ast_chunk_max_lines` (200). Per-medium
//! config threading needs a chunker registry (P+); same deviation
//! pattern as `pdf-page-v1`'s pinned `chunker_version`
//! (`tasks/HOTFIXES.md`).
use kebab_core::{
Block, BlockId, CanonicalDocument, Chunk, ChunkPolicy, Chunker, ChunkerVersion, DocumentId,
SourceSpan, id_for_chunk,
};
const VERSION_LABEL: &str = "code-cpp-ast-v1";
const BYTES_PER_TOKEN: usize = 3;
const POLICY_HASH_HEX_LEN: usize = 16;
const AST_CHUNK_MAX_LINES: u32 = 200;
#[derive(Clone, Copy, Debug, Default)]
pub struct CodeCppAstV1Chunker;
impl Chunker for CodeCppAstV1Chunker {
fn chunker_version(&self) -> ChunkerVersion {
ChunkerVersion(VERSION_LABEL.to_string())
}
fn policy_hash(&self, policy: &ChunkPolicy) -> String {
let bytes = serde_json_canonicalizer::to_vec(policy)
.expect("canonical JSON serialization of ChunkPolicy must not fail");
let hex = blake3::hash(&bytes).to_hex().to_string();
hex[..POLICY_HASH_HEX_LEN].to_string()
}
fn chunk(
&self,
doc: &CanonicalDocument,
policy: &ChunkPolicy,
) -> anyhow::Result<Vec<Chunk>> {
for b in &doc.blocks {
let c = match b {
Block::Code(c) => c,
_ => anyhow::bail!(
"CodeCppAstV1Chunker only handles code docs (got non-Code block)"
),
};
if !matches!(c.common.source_span, SourceSpan::Code { .. }) {
anyhow::bail!(
"CodeCppAstV1Chunker only handles code docs (got non-Code source_span)"
);
}
}
let base_policy_hash = self.policy_hash(policy);
let chunker_version = self.chunker_version();
let mut out: Vec<Chunk> = Vec::new();
for b in &doc.blocks {
let cb = match b {
Block::Code(c) => c,
_ => unreachable!("validated above"),
};
let (ls, le, symbol, lang) = match &cb.common.source_span {
SourceSpan::Code { line_start, line_end, symbol, lang } => {
(*line_start, *line_end, symbol.clone(), lang.clone())
}
_ => unreachable!("validated above"),
};
let block_ids: Vec<BlockId> = vec![cb.common.block_id.clone()];
let span_lines = le.saturating_sub(ls) + 1;
if span_lines <= AST_CHUNK_MAX_LINES {
let span = SourceSpan::Code {
line_start: ls,
line_end: le,
symbol: symbol.clone(),
lang: lang.clone(),
};
out.push(make_chunk(
doc, &chunker_version, &block_ids, &base_policy_hash,
None, span, cb.code.clone(),
));
} else {
let parts = split_oversize(&cb.code);
let n = parts.len();
for (i, (off_start, off_end, text)) in parts.into_iter().enumerate() {
let part_ls = ls + off_start;
let part_le = ls + off_end;
let part_sym = symbol
.as_ref()
.map(|s| format!("{s} [part {}/{n}]", i + 1));
let span = SourceSpan::Code {
line_start: part_ls,
line_end: part_le,
symbol: part_sym,
lang: lang.clone(),
};
out.push(make_chunk(
doc, &chunker_version, &block_ids, &base_policy_hash,
Some(part_ls), span, text,
));
}
}
}
tracing::debug!(
target: "kebab-chunk",
doc_id = %doc.doc_id,
chunks = out.len(),
"code-cpp-ast-v1 chunked",
);
Ok(out)
}
}
#[allow(clippy::too_many_arguments)]
fn make_chunk(
doc: &CanonicalDocument,
chunker_version: &ChunkerVersion,
block_ids: &[BlockId],
base_policy_hash: &str,
split_key: Option<u32>,
span: SourceSpan,
text: String,
) -> Chunk {
let id_hash = match split_key {
Some(k) => format!("{base_policy_hash}#L{k}"),
None => base_policy_hash.to_string(),
};
let chunk_id = id_for_chunk(&doc.doc_id, chunker_version, block_ids, &id_hash);
let token_estimate = text.len().div_ceil(BYTES_PER_TOKEN);
Chunk {
chunk_id,
doc_id: DocumentId(doc.doc_id.0.clone()),
block_ids: block_ids.to_vec(),
text,
heading_path: Vec::new(),
source_spans: vec![span],
token_estimate,
chunker_version: chunker_version.clone(),
policy_hash: base_policy_hash.to_string(),
}
}
/// Split an oversize unit at blank-line paragraph boundaries, greedily
/// gluing paragraphs until ~`AST_CHUNK_MAX_LINES` lines accumulate.
/// Returns `(line_offset_start, line_offset_end, text)` where offsets are
/// 0-based within the unit (caller adds the unit's absolute `line_start`).
fn split_oversize(code: &str) -> Vec<(u32, u32, String)> {
let lines: Vec<&str> = code.split('\n').collect();
let total = lines.len() as u32;
let mut out: Vec<(u32, u32, String)> = Vec::new();
let mut start: u32 = 0;
while start < total {
let mut end = (start + AST_CHUNK_MAX_LINES).min(total);
let floor = start + (AST_CHUNK_MAX_LINES * 4 / 5);
if end < total {
if let Some(b) = (floor.min(end)..end)
.rev()
.find(|&i| lines[i as usize].trim().is_empty())
{
end = b + 1;
}
}
let text = lines[start as usize..end as usize].join("\n");
out.push((start, end.saturating_sub(1), text));
start = end;
}
if out.is_empty() {
out.push((0, total.saturating_sub(1), code.to_string()));
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use kebab_core::{
Block, CanonicalDocument, ChunkPolicy, Chunker, ChunkerVersion, CodeBlock, CommonBlock,
SourceSpan, id_for_block, id_for_doc, AssetId, Lang, Metadata, ParserVersion, Provenance,
SourceType, TrustLevel, WorkspacePath,
};
use time::OffsetDateTime;
fn code_doc(units: &[(&str, u32, u32, &str)]) -> CanonicalDocument {
let wp = WorkspacePath("crates/x/src/a.cpp".into());
let aid = AssetId("a".repeat(64));
let pv = ParserVersion("code-cpp-v1".into());
let doc_id = id_for_doc(&wp, &aid, &pv);
let blocks = units
.iter()
.enumerate()
.map(|(i, (sym, ls, le, code))| {
let span = SourceSpan::Code {
line_start: *ls,
line_end: *le,
symbol: Some((*sym).to_string()),
lang: Some("cpp".into()),
};
let bid = id_for_block(&doc_id, "code", &[], i as u32, &span);
Block::Code(CodeBlock {
common: CommonBlock { block_id: bid, heading_path: vec![], source_span: span },
lang: Some("cpp".into()),
code: (*code).to_string(),
})
})
.collect();
CanonicalDocument {
doc_id, source_asset_id: aid, workspace_path: wp, title: "a".into(),
lang: Lang("und".into()), blocks,
metadata: Metadata {
aliases: vec![], tags: vec![],
created_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
updated_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
source_type: SourceType::Note, trust_level: TrustLevel::Primary,
user_id_alias: None, user: Default::default(),
repo: Some("kebab".into()), git_branch: Some("main".into()),
git_commit: Some("0".repeat(40)), code_lang: Some("cpp".into()),
},
provenance: Provenance { events: vec![] },
parser_version: pv, schema_version: 1, doc_version: 1,
last_chunker_version: None, last_embedding_version: None,
}
}
fn policy() -> ChunkPolicy {
ChunkPolicy { target_tokens: 500, overlap_tokens: 80,
respect_markdown_headings: false,
chunker_version: ChunkerVersion(VERSION_LABEL.into()) }
}
#[test]
fn chunker_version_is_code_cpp_ast_v1() {
assert_eq!(CodeCppAstV1Chunker.chunker_version(),
ChunkerVersion("code-cpp-ast-v1".into()));
}
#[test]
fn one_chunk_per_unit_preserves_code_span() {
let doc = code_doc(&[
("parse", 1, 3, "int parse() {\n\t// x\n}"),
("print", 5, 7, "void print() {\n\t//\n\treturn;\n}"),
]);
let chunks = CodeCppAstV1Chunker.chunk(&doc, &policy()).unwrap();
assert_eq!(chunks.len(), 2);
for c in &chunks {
assert_eq!(c.source_spans.len(), 1);
assert!(matches!(c.source_spans[0], SourceSpan::Code { .. }));
assert_eq!(c.heading_path, Vec::<String>::new());
assert_eq!(c.chunker_version.0, "code-cpp-ast-v1");
}
match &chunks[0].source_spans[0] {
SourceSpan::Code { symbol, line_start, line_end, .. } => {
assert_eq!(symbol.as_deref(), Some("parse"));
assert_eq!((*line_start, *line_end), (1, 3));
}
_ => unreachable!(),
}
}
#[test]
fn oversize_unit_splits_into_parts_with_unique_ids() {
let body = (0..500).map(|i| format!("\tx{i} = {i};\n")).collect::<String>();
let code = format!("int big() {{\n{body}\n}}");
let doc = code_doc(&[("big", 1, 502, &code)]);
let chunks = CodeCppAstV1Chunker.chunk(&doc, &policy()).unwrap();
assert!(chunks.len() >= 2, "oversize unit must split, got {}", chunks.len());
for c in &chunks {
match &c.source_spans[0] {
SourceSpan::Code { symbol, .. } => {
assert!(symbol.as_deref().unwrap().starts_with("big [part "),
"part-numbered symbol, got {symbol:?}");
}
_ => unreachable!(),
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}
#[test]
fn non_code_doc_errors() {
use kebab_core::TextBlock;
let mut doc = code_doc(&[("parse", 1, 1, "int parse() {}")]);
doc.blocks = vec![Block::Paragraph(TextBlock {
common: CommonBlock {
block_id: kebab_core::BlockId("b".into()),
heading_path: vec![],
source_span: SourceSpan::Line { start: 1, end: 1 },
},
text: "x".into(), inlines: vec![],
})];
let err = CodeCppAstV1Chunker.chunk(&doc, &policy()).unwrap_err();
assert!(err.to_string().contains("CodeCppAstV1Chunker"));
}
#[test]
fn deterministic_chunk_ids_1000() {
let doc = code_doc(&[("parse", 1, 2, "int parse() {}\n")]);
let base: Vec<String> = CodeCppAstV1Chunker.chunk(&doc, &policy())
.unwrap().into_iter().map(|c| c.chunk_id.0).collect();
for _ in 0..1000 {
let again: Vec<String> = CodeCppAstV1Chunker.chunk(&doc, &policy())
.unwrap().into_iter().map(|c| c.chunk_id.0).collect();
assert_eq!(again, base);
}
}
#[test]
fn policy_hash_matches_md_heading_v1() {
let p = policy();
assert_eq!(CodeCppAstV1Chunker.policy_hash(&p),
crate::MdHeadingV1Chunker.policy_hash(&p));
}
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -281,7 +281,7 @@ mod tests {
}
}
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
let n = ids.len(); ids.sort(); ids.dedup();
let n = ids.len(); ids.sort_unstable(); ids.dedup();
assert_eq!(ids.len(), n, "chunk_ids unique across split parts");
}

View File

@@ -43,6 +43,7 @@ impl Chunker for DockerfileFileV1Chunker {
"<dockerfile>",
"dockerfile",
VERSION_LABEL,
None,
)?;
tracing::debug!(

View File

@@ -85,6 +85,7 @@ impl Chunker for K8sManifestResourceV1Chunker {
&symbol,
"yaml",
VERSION_LABEL,
Some(slice.line_start),
)?;
}

View File

@@ -15,6 +15,8 @@
//! embedder, the retriever, the LLM, the RAG layer, or the UI layers.
//! It consumes `CanonicalDocument` purely through `kb-core` types.
mod code_c_ast_v1;
mod code_cpp_ast_v1;
mod code_go_ast_v1;
mod code_java_ast_v1;
mod code_js_ast_v1;
@@ -30,6 +32,8 @@ pub mod dockerfile_file_v1;
pub mod manifest_file_v1;
pub mod code_text_paragraph_v1;
pub use code_c_ast_v1::CodeCAstV1Chunker;
pub use code_cpp_ast_v1::CodeCppAstV1Chunker;
pub use code_go_ast_v1::CodeGoAstV1Chunker;
pub use code_java_ast_v1::CodeJavaAstV1Chunker;
pub use code_js_ast_v1::CodeJsAstV1Chunker;

View File

@@ -44,6 +44,7 @@ impl Chunker for ManifestFileV1Chunker {
"<manifest>",
lang,
VERSION_LABEL,
None,
)?;
tracing::debug!(

View File

@@ -387,9 +387,7 @@ fn render_block_text(b: &Block) -> String {
// alt keeps lexical search hits on filenames working even when
// P6-1's filename auto-fill is bypassed.
Block::ImageRef(i) => {
let alt = if !i.alt.is_empty() {
i.alt.clone()
} else {
let alt = if i.alt.is_empty() {
// P6-1 falls back to filename so this branch is
// defensive — keep it lest a future test fixture or
// synthetic block path skip the auto-fill.
@@ -399,17 +397,17 @@ fn render_block_text(b: &Block) -> String {
.filter(|s| !s.is_empty())
.unwrap_or("[image]")
.to_string()
} else {
i.alt.clone()
};
let ocr = i
.ocr
.as_ref()
.map(|o| o.joined.as_str())
.unwrap_or("");
.map_or("", |o| o.joined.as_str());
let cap = i
.caption
.as_ref()
.map(|c| c.text.as_str())
.unwrap_or("");
.map_or("", |c| c.text.as_str());
[alt.as_str(), ocr, cap]
.iter()
.filter(|s| !s.is_empty())

View File

@@ -450,7 +450,7 @@ mod tests {
// chunk_ids stay distinct despite identical block_ids — the
// per-chunk policy_hash variant is doing its job.
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
ids.sort();
ids.sort_unstable();
let total = ids.len();
ids.dedup();
assert_eq!(ids.len(), total, "all chunk_ids must be unique");
@@ -668,7 +668,7 @@ mod tests {
// chunk_ids stay distinct (the per-chunk hash variant keys off
// char_start which is now strictly increasing).
let mut ids: Vec<&str> = chunks.iter().map(|c| c.chunk_id.0.as_str()).collect();
ids.sort();
ids.sort_unstable();
let total = ids.len();
ids.dedup();
assert_eq!(ids.len(), total, "chunk_ids must remain unique");

View File

@@ -25,6 +25,13 @@ pub(crate) fn policy_hash(policy: &ChunkPolicy) -> String {
/// Emit one chunk for `(text, line_start..=line_end, symbol, lang)`, splitting
/// into line-windows of at most `AST_CHUNK_MAX_LINES` if the slice is oversize.
/// Mirrors the oversize path in `code_rust_ast_v1`'s `chunk` impl.
///
/// `base_split_key` is used as the `split_key` for the non-oversize single-chunk
/// case. Callers that emit multiple chunks from the same document (e.g.
/// `K8sManifestResourceV1Chunker` — one call per k8s resource) MUST pass
/// `Some(line_start)` so that each call produces a distinct `chunk_id`.
/// Single-chunk callers (dockerfile-file-v1, manifest-file-v1) pass `None` to
/// keep chunk_ids stable (no sibling can collide when there's only one chunk).
#[allow(clippy::too_many_arguments)]
pub(crate) fn push_chunks_with_oversize(
out: &mut Vec<Chunk>,
@@ -36,6 +43,7 @@ pub(crate) fn push_chunks_with_oversize(
symbol: &str,
lang: &str,
chunker_version: &str,
base_split_key: Option<u32>,
) -> Result<()> {
let n_lines = (line_end - line_start + 1).max(1);
let cv = ChunkerVersion(chunker_version.to_string());
@@ -51,7 +59,7 @@ pub(crate) fn push_chunks_with_oversize(
line_end,
symbol,
lang,
None,
base_split_key,
));
return Ok(());
}

View File

@@ -0,0 +1,196 @@
//! Snapshot test pinning the `Vec<Chunk>` JSON for a
//! representative C code `CanonicalDocument`.
//!
//! This is an integration test. `kebab-parse-code` is intentionally NOT
//! a dev-dep (design §6.3 / §8 boundary: AST extraction is parser-side).
//! The `CanonicalDocument` is built inline from hand-crafted `Block::Code`
//! units, which is the same pattern used in `code_go_ast_v1.rs`'s
//! internal `code_doc` test helper.
//!
//! Set `UPDATE_SNAPSHOTS=1` to re-bake the baseline.
use std::path::PathBuf;
use kebab_chunk::CodeCAstV1Chunker;
use kebab_core::{
AssetId, Block, CanonicalDocument, ChunkPolicy, Chunker, ChunkerVersion, CodeBlock, CommonBlock,
Lang, Metadata, ParserVersion, Provenance, SourceSpan, SourceType, TrustLevel, WorkspacePath,
id_for_block, id_for_doc,
};
use serde_json::Value;
use time::OffsetDateTime;
fn fixtures_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests")
.join("fixtures")
}
fn fixed_doc() -> CanonicalDocument {
let wp = WorkspacePath("projects/record.c".into());
let aid = AssetId("c".repeat(64));
// Pin parser_version so doc_id / block_ids are reproducible.
let pv = ParserVersion("code-c-v1".into());
let doc_id = id_for_doc(&wp, &aid, &pv);
// Representative units:
// 0. imports + defines (lines 14, ≤200)
// 1. status_t enum typedef (lines 69, ≤200)
// 2. record_t struct typedef (lines 1116, ≤200)
// 3. static counter decl glue (line 18, ≤200)
// 4. parse_record fn (lines 2023, ≤200)
// 5. print_record fn (lines 2527, ≤200)
// 6. main fn (lines 2933, ≤200)
let raw_units: Vec<(&str, u32, u32, String)> = vec![
(
"<top-level>",
1,
18,
"#include <stdio.h>\n#include <stdlib.h>\n\n#define MAX_BUF 4096\n\ntypedef enum {\n OK = 0,\n ERR_PARSE,\n ERR_IO,\n} status_t;\n\ntypedef struct {\n int id;\n char name[64];\n status_t status;\n} record_t;\n\nstatic int counter = 0;".to_string(),
),
(
"parse_record",
20,
23,
"int parse_record(const char *line, record_t *out) {\n if (line == NULL || out == NULL) return ERR_PARSE;\n return OK;\n}".to_string(),
),
(
"print_record",
25,
27,
"void print_record(const record_t *r) {\n printf(\"[%d] %s (status=%d)\\n\", r->id, r->name, r->status);\n}".to_string(),
),
(
"main",
29,
33,
"int main(void) {\n record_t r = { .id = 1, .name = \"foo\", .status = OK };\n print_record(&r);\n return 0;\n}".to_string(),
),
];
let blocks: Vec<Block> = raw_units
.iter()
.enumerate()
.map(|(i, (sym, ls, le, code))| {
let span = SourceSpan::Code {
line_start: *ls,
line_end: *le,
symbol: Some((*sym).to_string()),
lang: Some("c".into()),
};
let bid = id_for_block(&doc_id, "code", &[], i as u32, &span);
Block::Code(CodeBlock {
common: CommonBlock {
block_id: bid,
heading_path: vec![],
source_span: span,
},
lang: Some("c".into()),
code: code.clone(),
})
})
.collect();
CanonicalDocument {
doc_id,
source_asset_id: aid,
workspace_path: wp,
title: "record.c".into(),
lang: Lang("und".into()),
blocks,
metadata: Metadata {
aliases: vec![],
tags: vec![],
created_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
updated_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
source_type: SourceType::Note,
trust_level: TrustLevel::Primary,
user_id_alias: None,
user: Default::default(),
repo: Some("kebab".into()),
git_branch: Some("main".into()),
git_commit: Some("0".repeat(40)),
code_lang: Some("c".into()),
},
provenance: Provenance { events: vec![] },
parser_version: pv,
schema_version: 1,
doc_version: 1,
last_chunker_version: None,
last_embedding_version: None,
}
}
fn fixed_policy() -> ChunkPolicy {
ChunkPolicy {
target_tokens: 500,
overlap_tokens: 80,
respect_markdown_headings: false,
chunker_version: ChunkerVersion("code-c-ast-v1".into()),
}
}
#[test]
fn code_c_ast_chunks_snapshot() {
let doc = fixed_doc();
let policy = fixed_policy();
let chunks = CodeCAstV1Chunker.chunk(&doc, &policy).expect("chunk");
let actual = serde_json::to_value(&chunks).unwrap();
let dir = fixtures_dir();
let baseline_path = dir.join("code-sample.c.chunks.snapshot.json");
let baseline_text = match std::fs::read_to_string(&baseline_path) {
Ok(s) => s,
Err(_) if std::env::var("UPDATE_SNAPSHOTS").is_ok() => {
std::fs::create_dir_all(&dir).unwrap();
let pretty = serde_json::to_string_pretty(&actual).unwrap();
std::fs::write(&baseline_path, format!("{pretty}\n")).unwrap();
return;
}
Err(e) => panic!(
"missing baseline {}; run with UPDATE_SNAPSHOTS=1 to create: {e}",
baseline_path.display()
),
};
let expected: Value = serde_json::from_str(&baseline_text).expect("baseline parses as json");
if actual != expected {
if std::env::var("UPDATE_SNAPSHOTS").is_ok() {
let pretty = serde_json::to_string_pretty(&actual).unwrap();
std::fs::write(&baseline_path, format!("{pretty}\n")).unwrap();
eprintln!("updated baseline {}", baseline_path.display());
return;
}
let pretty = serde_json::to_string_pretty(&actual).unwrap();
panic!(
"code-c-ast-v1 chunks snapshot drift\n\
--- expected ({}) ---\n{baseline_text}\n\
--- actual ---\n{pretty}\n\
If intentional, re-run with UPDATE_SNAPSHOTS=1.",
baseline_path.display()
);
}
}
/// Determinism cross-check: re-running the same pipeline yields the same
/// chunk_ids byte-for-byte.
#[test]
fn code_c_ast_chunks_are_deterministic() {
let policy = fixed_policy();
let baseline: Vec<String> = CodeCAstV1Chunker
.chunk(&fixed_doc(), &policy)
.unwrap()
.into_iter()
.map(|c| c.chunk_id.0)
.collect();
for _ in 0..5 {
let again: Vec<String> = CodeCAstV1Chunker
.chunk(&fixed_doc(), &policy)
.unwrap()
.into_iter()
.map(|c| c.chunk_id.0)
.collect();
assert_eq!(again, baseline);
}
}

View File

@@ -0,0 +1,325 @@
//! Snapshot test pinning the `Vec<Chunk>` JSON for a
//! representative C++ code `CanonicalDocument`.
//!
//! Two complementary tests:
//! 1. `code_cpp_ast_chunks_snapshot` — hand-built `fixed_doc()` validates the
//! chunker's 1:1 mapping (design §6.3 / §8 boundary: no parse-code dep needed).
//! 2. `code_cpp_ast_extractor_snapshot` — invokes `CppAstExtractor` against the
//! real `tests/fixtures/sample.cpp` fixture, validating the extractor → chunker
//! end-to-end pipeline. `kebab-parse-code` is a dev-dep (same pattern as
//! `kebab-parse-md` in Markdown snapshot tests).
//!
//! Set `UPDATE_SNAPSHOTS=1` to re-bake the baseline.
use std::path::PathBuf;
use kebab_chunk::CodeCppAstV1Chunker;
use kebab_core::{
AssetId, Block, CanonicalDocument, ChunkPolicy, Chunker, ChunkerVersion, CodeBlock, CommonBlock,
Lang, Metadata, ParserVersion, Provenance, SourceSpan, SourceType, TrustLevel, WorkspacePath,
id_for_block, id_for_doc,
};
use kebab_parse_code::CppAstExtractor;
use serde_json::Value;
use time::OffsetDateTime;
fn fixtures_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests")
.join("fixtures")
}
fn fixed_doc() -> CanonicalDocument {
let wp = WorkspacePath("projects/record.cpp".into());
let aid = AssetId("c".repeat(64));
// Pin parser_version so doc_id / block_ids are reproducible.
let pv = ParserVersion("code-cpp-v1".into());
let doc_id = id_for_doc(&wp, &aid, &pv);
// Representative units (C++ specific):
// 0. includes + namespace opening (lines 14, ≤200)
// 1. class definition (lines 620, ≤200)
// 2. template function (lines 2225, ≤200)
// 3. namespace closing + free fn (lines 2729, ≤200)
// 4. main fn (lines 3134, ≤200)
let raw_units: Vec<(&str, u32, u32, String)> = vec![
(
"<top-level>",
1,
4,
"#include <string>\n#include <vector>\n\nnamespace kebab {".to_string(),
),
(
"kebab::chunk::MdHeadingV1Chunker",
6,
20,
"class MdHeadingV1Chunker {\npublic:\n MdHeadingV1Chunker() = default;\n ~MdHeadingV1Chunker() = default;\n\n std::string chunk_doc(const std::string& doc) {\n return doc;\n }\n\n int operator()(int x) const {\n return x * 2;\n }\n\nprivate:\n int counter_ = 0;\n};".to_string(),
),
(
"kebab::identity",
22,
25,
"template <typename T>\nT identity(T value) {\n return value;\n}".to_string(),
),
(
"kebab::global_helper",
27,
29,
"void global_helper() {\n // free function in kebab namespace\n}".to_string(),
),
(
"main",
31,
34,
"int main() {\n kebab::chunk::MdHeadingV1Chunker c;\n return 0;\n}".to_string(),
),
];
let blocks: Vec<Block> = raw_units
.iter()
.enumerate()
.map(|(i, (sym, ls, le, code))| {
let span = SourceSpan::Code {
line_start: *ls,
line_end: *le,
symbol: Some((*sym).to_string()),
lang: Some("cpp".into()),
};
let bid = id_for_block(&doc_id, "code", &[], i as u32, &span);
Block::Code(CodeBlock {
common: CommonBlock {
block_id: bid,
heading_path: vec![],
source_span: span,
},
lang: Some("cpp".into()),
code: code.clone(),
})
})
.collect();
CanonicalDocument {
doc_id,
source_asset_id: aid,
workspace_path: wp,
title: "record.cpp".into(),
lang: Lang("und".into()),
blocks,
metadata: Metadata {
aliases: vec![],
tags: vec![],
created_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
updated_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
source_type: SourceType::Note,
trust_level: TrustLevel::Primary,
user_id_alias: None,
user: Default::default(),
repo: Some("kebab".into()),
git_branch: Some("main".into()),
git_commit: Some("0".repeat(40)),
code_lang: Some("cpp".into()),
},
provenance: Provenance { events: vec![] },
parser_version: pv,
schema_version: 1,
doc_version: 1,
last_chunker_version: None,
last_embedding_version: None,
}
}
fn fixed_policy() -> ChunkPolicy {
ChunkPolicy {
target_tokens: 500,
overlap_tokens: 80,
respect_markdown_headings: false,
chunker_version: ChunkerVersion("code-cpp-ast-v1".into()),
}
}
// ---------------------------------------------------------------------------
// Helper: run the real CppAstExtractor against tests/fixtures/sample.cpp
// ---------------------------------------------------------------------------
fn extract_cpp_fixture() -> CanonicalDocument {
use kebab_core::{
AssetId, AssetStorage, Checksum, ExtractConfig, ExtractContext, Extractor, RawAsset,
SourceUri, WorkspacePath,
};
use std::path::PathBuf;
let bytes = std::fs::read(fixtures_dir().join("sample.cpp")).expect("read sample.cpp fixture");
let src = String::from_utf8(bytes).expect("fixture is valid UTF-8");
let wp = WorkspacePath("tests/fixtures/sample.cpp".to_string());
let asset = RawAsset {
asset_id: AssetId("e".repeat(64)),
source_uri: SourceUri::File(PathBuf::from("tests/fixtures/sample.cpp")),
workspace_path: wp,
media_type: kebab_core::MediaType::Code("cpp".to_string()),
byte_len: src.len() as u64,
checksum: Checksum("f".repeat(64)),
discovered_at: time::OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
stored: AssetStorage::Reference {
path: PathBuf::from("tests/fixtures/sample.cpp"),
sha: Checksum("f".repeat(64)),
},
};
let cfg = ExtractConfig::default();
let root = PathBuf::from("/tmp");
let ctx = ExtractContext {
asset: &asset,
workspace_root: &root,
config: &cfg,
};
CppAstExtractor::new().extract(&ctx, src.as_bytes()).unwrap()
}
// ---------------------------------------------------------------------------
// Test 1 (hand-built): chunker-only 1:1 mapping validation
// ---------------------------------------------------------------------------
#[test]
fn code_cpp_ast_chunks_snapshot() {
let doc = fixed_doc();
let policy = fixed_policy();
let chunks = CodeCppAstV1Chunker.chunk(&doc, &policy).expect("chunk");
let actual = serde_json::to_value(&chunks).unwrap();
let dir = fixtures_dir();
let baseline_path = dir.join("code-sample.cpp.chunks.snapshot.json");
let baseline_text = match std::fs::read_to_string(&baseline_path) {
Ok(s) => s,
Err(_) if std::env::var("UPDATE_SNAPSHOTS").is_ok() => {
std::fs::create_dir_all(&dir).unwrap();
let pretty = serde_json::to_string_pretty(&actual).unwrap();
std::fs::write(&baseline_path, format!("{pretty}\n")).unwrap();
return;
}
Err(e) => panic!(
"missing baseline {}; run with UPDATE_SNAPSHOTS=1 to create: {e}",
baseline_path.display()
),
};
let expected: Value = serde_json::from_str(&baseline_text).expect("baseline parses as json");
if actual != expected {
if std::env::var("UPDATE_SNAPSHOTS").is_ok() {
let pretty = serde_json::to_string_pretty(&actual).unwrap();
std::fs::write(&baseline_path, format!("{pretty}\n")).unwrap();
eprintln!("updated baseline {}", baseline_path.display());
return;
}
let pretty = serde_json::to_string_pretty(&actual).unwrap();
panic!(
"code-cpp-ast-v1 chunks snapshot drift\n\
--- expected ({}) ---\n{baseline_text}\n\
--- actual ---\n{pretty}\n\
If intentional, re-run with UPDATE_SNAPSHOTS=1.",
baseline_path.display()
);
}
}
/// Determinism cross-check: re-running the same pipeline yields the same
/// chunk_ids byte-for-byte.
#[test]
fn code_cpp_ast_chunks_are_deterministic() {
let policy = fixed_policy();
let baseline: Vec<String> = CodeCppAstV1Chunker
.chunk(&fixed_doc(), &policy)
.unwrap()
.into_iter()
.map(|c| c.chunk_id.0)
.collect();
for _ in 0..5 {
let again: Vec<String> = CodeCppAstV1Chunker
.chunk(&fixed_doc(), &policy)
.unwrap()
.into_iter()
.map(|c| c.chunk_id.0)
.collect();
assert_eq!(again, baseline);
}
}
// ---------------------------------------------------------------------------
// Test 2 (real extractor): end-to-end extractor → chunker pipeline
// ---------------------------------------------------------------------------
/// Validates that the real `CppAstExtractor` processes `sample.cpp` and
/// emits the expected set of symbols through the full chunker pipeline.
///
/// `sample.cpp` contains:
/// - `#include` directives + nested namespace `kebab::chunk` → glue + struct unit
/// - `class MdHeadingV1Chunker` with methods (ctor, dtor, chunk_doc, operator())
/// - `template <typename T> T identity(T value)` (template fn)
/// - `void kebab::global_helper()` (free fn in namespace)
/// - `int main()` (global free fn)
#[test]
fn code_cpp_ast_extractor_snapshot() {
let doc = extract_cpp_fixture();
// Verify the extractor emits all expected named units.
let block_syms: Vec<Option<String>> = doc.blocks.iter().filter_map(|b| match b {
Block::Code(c) => match &c.common.source_span {
SourceSpan::Code { symbol, .. } => Some(symbol.clone()),
_ => None,
},
_ => None,
}).collect();
// Must include namespace-qualified class and its methods
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::chunk::MdHeadingV1Chunker")),
"class unit missing: {block_syms:?}"
);
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::chunk::MdHeadingV1Chunker::MdHeadingV1Chunker")),
"ctor unit missing: {block_syms:?}"
);
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::chunk::MdHeadingV1Chunker::~MdHeadingV1Chunker")),
"dtor unit missing: {block_syms:?}"
);
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::chunk::MdHeadingV1Chunker::chunk_doc")),
"chunk_doc unit missing: {block_syms:?}"
);
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::chunk::MdHeadingV1Chunker::operator()")),
"operator() unit missing: {block_syms:?}"
);
// Template function (inside kebab::chunk namespace in the fixture)
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::chunk::identity")),
"identity template fn unit missing: {block_syms:?}"
);
// Free function in outer namespace
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("kebab::global_helper")),
"global_helper unit missing: {block_syms:?}"
);
// Global main
assert!(
block_syms.iter().any(|s| s.as_deref() == Some("main")),
"main unit missing: {block_syms:?}"
);
}
/// End-to-end chunker output from real extractor is deterministic.
#[test]
fn code_cpp_ast_extractor_chunks_deterministic() {
let doc1 = extract_cpp_fixture();
let doc2 = extract_cpp_fixture();
assert_eq!(doc1.blocks, doc2.blocks, "extractor output non-deterministic");
let policy = fixed_policy();
let chunks1 = CodeCppAstV1Chunker.chunk(&doc1, &policy).unwrap();
let chunks2 = CodeCppAstV1Chunker.chunk(&doc2, &policy).unwrap();
assert_eq!(
chunks1.iter().map(|c| c.chunk_id.0.clone()).collect::<Vec<_>>(),
chunks2.iter().map(|c| c.chunk_id.0.clone()).collect::<Vec<_>>(),
"chunker output non-deterministic"
);
}

View File

@@ -0,0 +1,86 @@
[
{
"block_ids": [
"8149e12ca002489acb4a0f74c97a061a"
],
"chunk_id": "ec3cf06ae56c8e9796bbc9196438b7c5",
"chunker_version": "code-c-ast-v1",
"doc_id": "6bec42dd593920a060541db16c4e8e45",
"heading_path": [],
"policy_hash": "ecfad2ec1223662d",
"source_spans": [
{
"kind": "code",
"lang": "c",
"line_end": 18,
"line_start": 1,
"symbol": "<top-level>"
}
],
"text": "#include <stdio.h>\n#include <stdlib.h>\n\n#define MAX_BUF 4096\n\ntypedef enum {\n OK = 0,\n ERR_PARSE,\n ERR_IO,\n} status_t;\n\ntypedef struct {\n int id;\n char name[64];\n status_t status;\n} record_t;\n\nstatic int counter = 0;",
"token_estimate": 78
},
{
"block_ids": [
"1baaa89f21a47b2f32d6396a24a85454"
],
"chunk_id": "c2d7a81c898106733ef2e703774a6a4a",
"chunker_version": "code-c-ast-v1",
"doc_id": "6bec42dd593920a060541db16c4e8e45",
"heading_path": [],
"policy_hash": "ecfad2ec1223662d",
"source_spans": [
{
"kind": "code",
"lang": "c",
"line_end": 23,
"line_start": 20,
"symbol": "parse_record"
}
],
"text": "int parse_record(const char *line, record_t *out) {\n if (line == NULL || out == NULL) return ERR_PARSE;\n return OK;\n}",
"token_estimate": 41
},
{
"block_ids": [
"8d0e14cbcc6d1e92d7878ab796ea68b8"
],
"chunk_id": "0e4d7b131ab64eba03b51903b5d8f96d",
"chunker_version": "code-c-ast-v1",
"doc_id": "6bec42dd593920a060541db16c4e8e45",
"heading_path": [],
"policy_hash": "ecfad2ec1223662d",
"source_spans": [
{
"kind": "code",
"lang": "c",
"line_end": 27,
"line_start": 25,
"symbol": "print_record"
}
],
"text": "void print_record(const record_t *r) {\n printf(\"[%d] %s (status=%d)\\n\", r->id, r->name, r->status);\n}",
"token_estimate": 35
},
{
"block_ids": [
"9c2ede84423871b615d48c38fefb1853"
],
"chunk_id": "e076f8edb2ff141d7e99b4106bb95157",
"chunker_version": "code-c-ast-v1",
"doc_id": "6bec42dd593920a060541db16c4e8e45",
"heading_path": [],
"policy_hash": "ecfad2ec1223662d",
"source_spans": [
{
"kind": "code",
"lang": "c",
"line_end": 33,
"line_start": 29,
"symbol": "main"
}
],
"text": "int main(void) {\n record_t r = { .id = 1, .name = \"foo\", .status = OK };\n print_record(&r);\n return 0;\n}",
"token_estimate": 38
}
]

View File

@@ -0,0 +1,107 @@
[
{
"block_ids": [
"53292605459065d170cd36c118e20546"
],
"chunk_id": "50a5b324300d9082eac4ce2a422810e1",
"chunker_version": "code-cpp-ast-v1",
"doc_id": "fff1e1f0a7ff70ef682937470e5d1d28",
"heading_path": [],
"policy_hash": "71f3c07bb9ec1d09",
"source_spans": [
{
"kind": "code",
"lang": "cpp",
"line_end": 4,
"line_start": 1,
"symbol": "<top-level>"
}
],
"text": "#include <string>\n#include <vector>\n\nnamespace kebab {",
"token_estimate": 18
},
{
"block_ids": [
"f349acad94c9fa4cf9ad1c0a93e83610"
],
"chunk_id": "0e6bc7c522665af8a4b0f66afb9d29c8",
"chunker_version": "code-cpp-ast-v1",
"doc_id": "fff1e1f0a7ff70ef682937470e5d1d28",
"heading_path": [],
"policy_hash": "71f3c07bb9ec1d09",
"source_spans": [
{
"kind": "code",
"lang": "cpp",
"line_end": 20,
"line_start": 6,
"symbol": "kebab::chunk::MdHeadingV1Chunker"
}
],
"text": "class MdHeadingV1Chunker {\npublic:\n MdHeadingV1Chunker() = default;\n ~MdHeadingV1Chunker() = default;\n\n std::string chunk_doc(const std::string& doc) {\n return doc;\n }\n\n int operator()(int x) const {\n return x * 2;\n }\n\nprivate:\n int counter_ = 0;\n};",
"token_estimate": 95
},
{
"block_ids": [
"8b9811387717d0bd4abf84abcc35b8b1"
],
"chunk_id": "d9326d252905b665b2adb9a416c20451",
"chunker_version": "code-cpp-ast-v1",
"doc_id": "fff1e1f0a7ff70ef682937470e5d1d28",
"heading_path": [],
"policy_hash": "71f3c07bb9ec1d09",
"source_spans": [
{
"kind": "code",
"lang": "cpp",
"line_end": 25,
"line_start": 22,
"symbol": "kebab::identity"
}
],
"text": "template <typename T>\nT identity(T value) {\n return value;\n}",
"token_estimate": 21
},
{
"block_ids": [
"1754cb6b971f6a4cb292f144a4f0570b"
],
"chunk_id": "56ee5f991de4a413c016da8dc4acfc35",
"chunker_version": "code-cpp-ast-v1",
"doc_id": "fff1e1f0a7ff70ef682937470e5d1d28",
"heading_path": [],
"policy_hash": "71f3c07bb9ec1d09",
"source_spans": [
{
"kind": "code",
"lang": "cpp",
"line_end": 29,
"line_start": 27,
"symbol": "kebab::global_helper"
}
],
"text": "void global_helper() {\n // free function in kebab namespace\n}",
"token_estimate": 22
},
{
"block_ids": [
"14b5f3393d6d25f822f5b70763d24acd"
],
"chunk_id": "c0d7c043cdd575c530db3909b54cc906",
"chunker_version": "code-cpp-ast-v1",
"doc_id": "fff1e1f0a7ff70ef682937470e5d1d28",
"heading_path": [],
"policy_hash": "71f3c07bb9ec1d09",
"source_spans": [
{
"kind": "code",
"lang": "cpp",
"line_end": 34,
"line_start": 31,
"symbol": "main"
}
],
"text": "int main() {\n kebab::chunk::MdHeadingV1Chunker c;\n return 0;\n}",
"token_estimate": 23
}
]

View File

@@ -0,0 +1,33 @@
#include <stdio.h>
#include <stdlib.h>
#define MAX_BUF 4096
typedef enum {
OK = 0,
ERR_PARSE,
ERR_IO,
} status_t;
typedef struct {
int id;
char name[64];
status_t status;
} record_t;
static int counter = 0;
int parse_record(const char *line, record_t *out) {
if (line == NULL || out == NULL) return ERR_PARSE;
return OK;
}
void print_record(const record_t *r) {
printf("[%d] %s (status=%d)\n", r->id, r->name, r->status);
}
int main(void) {
record_t r = { .id = 1, .name = "foo", .status = OK };
print_record(&r);
return 0;
}

View File

@@ -0,0 +1,40 @@
#include <string>
#include <vector>
namespace kebab {
namespace chunk {
class MdHeadingV1Chunker {
public:
MdHeadingV1Chunker() = default;
~MdHeadingV1Chunker() = default;
std::string chunk_doc(const std::string& doc) {
return doc;
}
int operator()(int x) const {
return x * 2;
}
private:
int counter_ = 0;
};
template <typename T>
T identity(T value) {
return value;
}
} // namespace chunk
void global_helper() {
// free function in kebab namespace
}
} // namespace kebab
int main() {
kebab::chunk::MdHeadingV1Chunker c;
return 0;
}

View File

@@ -140,6 +140,17 @@ fn k8s_multi_doc_emits_one_chunk_per_resource() {
for chunk in &chunks {
assert_eq!(chunk.chunker_version.0, "k8s-manifest-resource-v1");
}
// Every chunk from a multi-resource file must have a distinct chunk_id.
// Without the fix, all non-oversize resources get split_key=None which
// collapses to the same id_hash (= base_policy_hash) → UNIQUE constraint
// violation on the second resource.
let ids: std::collections::HashSet<_> = chunks.iter().map(|c| c.chunk_id.clone()).collect();
assert_eq!(
ids.len(),
chunks.len(),
"every k8s resource chunk must have a distinct chunk_id (multi-resource collision regression)"
);
}
/// A YAML document with an indentation error (tab in a space-indented context)
@@ -269,9 +280,7 @@ fn k8s_oversize_splits_into_line_windows_sharing_symbol() {
assert_eq!(
prev_end + 1,
next_start,
"line ranges must be contiguous: {} → {} (got gap or overlap)",
prev_end,
next_start
"line ranges must be contiguous: {prev_end} → {next_start} (got gap or overlap)"
);
}
}

View File

@@ -18,8 +18,7 @@ use kebab_core::{
AssetId, AssetStorage, Checksum, ChunkPolicy, ChunkerVersion, Chunker, MediaType,
ParserVersion, RawAsset, SourceUri, WorkspacePath,
};
use kebab_normalize::build_canonical_document;
use kebab_parse_md::{BodyHints, parse_blocks, parse_frontmatter};
use kebab_parse_md::{BodyHints, build_canonical_document, parse_blocks, parse_frontmatter};
use serde_json::Value;
use time::OffsetDateTime;

View File

@@ -51,7 +51,7 @@ fn manifest_doc(lang: &str, manifest_text: &str) -> CanonicalDocument {
doc_id,
source_asset_id: aid,
workspace_path: wp,
title: format!("Manifest ({})", lang),
title: format!("Manifest ({lang})"),
lang: Lang("und".into()),
blocks: vec![block],
metadata: Metadata {

View File

@@ -50,3 +50,6 @@ tempfile = { workspace = true }
# to simulate stale docs. `time` is the formatter used by the helper.
rusqlite = { workspace = true }
time = { workspace = true }
[lints]
workspace = true

View File

@@ -250,6 +250,18 @@ enum Cmd {
/// `answer.v1`. Off by default to preserve final-only behavior.
#[arg(long)]
stream: bool,
/// p9-fb-41: route this ask through the multi-hop pipeline
/// — the query is decomposed into sub-questions, each
/// retrieved independently, then synthesized over the
/// merged chunk pool. Cost trade-off: 25× LLM calls
/// (decompose + 0..N decide + synthesize) vs. single-pass.
/// Worth it for compound questions (X 와 Y 의 관계, prereq
/// chain, cross-doc reasoning); single-pass is faster for
/// simple fact lookups. The full per-hop trace is exposed
/// on `Answer.hops` in `--json` mode.
#[arg(long)]
multi_hop: bool,
},
/// Wipe XDG data dirs (and optionally the Lance vector store) so the
@@ -785,7 +797,7 @@ fn run(cli: &Cli) -> anyhow::Result<()> {
serde_json::to_string(&item.query)?,
)?;
if let Some(err) = &item.error {
writeln!(stdout, "error: {}", err)?;
writeln!(stdout, "error: {err}")?;
} else if let Some(resp) = &item.response {
writeln!(
stdout,
@@ -933,6 +945,15 @@ fn run(cli: &Cli) -> anyhow::Result<()> {
let next = resp.next_cursor.as_deref().unwrap_or("(none)");
eprintln!("[truncated; use --cursor {next} for the next page]");
}
// v0.17.0 A5 Step 4: short-query advisory. `resp.hint`
// is `Some` only when the result list is empty and the
// trimmed query is shorter than the trigram tokenizer
// can resolve (raw FTS5 mode opts out). stderr so it
// doesn't pollute the stdout hit list. `--json` skips
// this branch entirely; the field rides the wire.
if let Some(hint) = &resp.hint {
eprintln!("[hint] {hint}");
}
if *trace {
if let Some(t) = &resp.trace {
eprintln!();
@@ -964,6 +985,7 @@ fn run(cli: &Cli) -> anyhow::Result<()> {
hide_citations,
session,
stream,
multi_hop,
} => {
let cfg = kebab_config::Config::load(cli.config.as_deref())?;
if *stream {
@@ -990,6 +1012,7 @@ fn run(cli: &Cli) -> anyhow::Result<()> {
history: Vec::new(),
conversation_id: None,
turn_index: None,
multi_hop: *multi_hop,
};
let cfg2 = cfg.clone();
let q = query.clone();
@@ -1065,6 +1088,7 @@ fn run(cli: &Cli) -> anyhow::Result<()> {
history: Vec::new(),
conversation_id: None,
turn_index: None,
multi_hop: *multi_hop,
};
let ans = match session.as_deref() {
Some(sid) => kebab_app::ask_with_session_with_config(cfg, sid, query, opts)?,
@@ -1147,15 +1171,13 @@ fn run(cli: &Cli) -> anyhow::Result<()> {
let report = kebab_app::reset::execute(scope, &cfg)?;
if cli.json {
println!("{}", serde_json::to_string(&wire::wire_reset(&report))?);
} else {
if report.orphans_purged > 0 {
println!("orphans purged: {}", report.orphans_purged);
for p in &report.purged_paths {
println!(" - {}", p.0);
}
} else {
println!("no orphaned docs found — store is already in sync with walker scope");
} else if report.orphans_purged > 0 {
println!("orphans purged: {}", report.orphans_purged);
for p in &report.purged_paths {
println!(" - {}", p.0);
}
} else {
println!("no orphaned docs found — store is already in sync with walker scope");
}
return Ok(());
}
@@ -1484,11 +1506,11 @@ fn confirm_destructive(
) -> anyhow::Result<bool> {
use std::io::Write;
let mut out = std::io::stderr().lock();
writeln!(out, "kebab reset ({:?}): about to remove", scope)?;
writeln!(out, "kebab reset ({scope:?}): about to remove")?;
for p in paths {
writeln!(out, " - {}", p.display())?;
}
writeln!(out, "estimated total: {} bytes", bytes)?;
writeln!(out, "estimated total: {bytes} bytes")?;
write!(out, "Proceed? [y/N] ")?;
out.flush()?;
@@ -1549,19 +1571,19 @@ fn render_fetch_plain(r: &kebab_core::FetchResult) {
if !r.context_before.is_empty() {
println!("\n=== before ===");
for c in &r.context_before {
let heading = c.heading_path.last().map(|s| s.as_str()).unwrap_or("");
let heading = c.heading_path.last().map_or("", std::string::String::as_str);
println!("[{} § {}]\n{}\n", c.chunk_id.0, heading, c.text);
}
}
if let Some(c) = &r.chunk {
println!("\n=== target ===");
let heading = c.heading_path.last().map(|s| s.as_str()).unwrap_or("");
let heading = c.heading_path.last().map_or("", std::string::String::as_str);
println!("[{} § {}]\n{}\n", c.chunk_id.0, heading, c.text);
}
if !r.context_after.is_empty() {
println!("\n=== after ===");
for c in &r.context_after {
let heading = c.heading_path.last().map(|s| s.as_str()).unwrap_or("");
let heading = c.heading_path.last().map_or("", std::string::String::as_str);
println!("[{} § {}]\n{}\n", c.chunk_id.0, heading, c.text);
}
}
@@ -1628,6 +1650,8 @@ mod tests {
created_at: OffsetDateTime::now_utc(),
conversation_id: None,
turn_index: None,
hops: None,
verification: None,
}
}

View File

@@ -92,6 +92,14 @@ pub fn wire_search_response(r: &kebab_app::SearchResponse) -> Value {
map.insert("trace".to_string(), trace_v);
}
}
// v0.17.0 A5 Step 4b: emit `hint` only when set. Keeps responses
// that don't carry a hint backward-compatible with v0 consumers
// that don't know the field.
if let Some(hint) = &r.hint {
if let Value::Object(ref mut map) = v {
map.insert("hint".to_string(), Value::String(hint.clone()));
}
}
tag_object(v, "search_response.v1")
}
@@ -292,6 +300,7 @@ mod tests {
next_cursor: Some("opaque-cursor-abc".to_string()),
truncated: true,
trace: None,
hint: None,
};
let v = wire_search_response(&r);
assert_eq!(schema_of(&v), Some("search_response.v1"));
@@ -304,7 +313,7 @@ mod tests {
v.get("next_cursor").and_then(|c| c.as_str()),
Some("opaque-cursor-abc")
);
assert_eq!(v.get("truncated").and_then(|t| t.as_bool()), Some(true));
assert_eq!(v.get("truncated").and_then(serde_json::Value::as_bool), Some(true));
}
#[test]
@@ -405,6 +414,7 @@ mod tests {
}],
timing: TraceTiming { lexical_ms: 5, vector_ms: 0, fusion_ms: 1, total_ms: 7 },
}),
hint: None,
};
let v = wire_search_response(&r);
assert_eq!(schema_of(&v), Some("search_response.v1"));
@@ -420,6 +430,7 @@ mod tests {
next_cursor: None,
truncated: false,
trace: None,
hint: None,
};
let v = wire_search_response(&r);
assert!(v.get("trace").is_none(), "trace field absent when None");

View File

@@ -88,5 +88,5 @@ max_context_tokens = 8000
let stdout = String::from_utf8_lossy(&out.stdout);
let v: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap();
assert_eq!(v.get("schema_version").and_then(|s| s.as_str()), Some("ingest_report.v1"));
assert_eq!(v.get("new").and_then(|n| n.as_u64()), Some(1));
assert_eq!(v.get("new").and_then(serde_json::Value::as_u64), Some(1));
}

View File

@@ -96,5 +96,5 @@ max_context_tokens = 8000
let stdout = String::from_utf8_lossy(&out.stdout);
let v: serde_json::Value = serde_json::from_str(stdout.trim()).unwrap();
assert_eq!(v.get("schema_version").and_then(|s| s.as_str()), Some("ingest_report.v1"));
assert_eq!(v.get("new").and_then(|n| n.as_u64()), Some(1));
assert_eq!(v.get("new").and_then(serde_json::Value::as_u64), Some(1));
}

View File

@@ -43,7 +43,7 @@ fn cli_mcp_initialize_then_tools_list() {
reader.read_line(&mut line).unwrap();
let init: serde_json::Value = serde_json::from_str(line.trim()).unwrap();
assert_eq!(
init.get("id").and_then(|i| i.as_i64()),
init.get("id").and_then(serde_json::Value::as_i64),
Some(1),
"unexpected id in initialize response: {init}"
);
@@ -57,7 +57,7 @@ fn cli_mcp_initialize_then_tools_list() {
reader.read_line(&mut line).unwrap();
let list: serde_json::Value = serde_json::from_str(line.trim()).unwrap();
assert_eq!(
list.get("id").and_then(|i| i.as_i64()),
list.get("id").and_then(serde_json::Value::as_i64),
Some(2),
"unexpected id in tools/list response: {list}"
);

View File

@@ -76,8 +76,7 @@ fn cli_schema_json_emits_schema_v1() {
assert!(
v.get("kebab_version")
.and_then(|s| s.as_str())
.map(|s| !s.is_empty())
.unwrap_or(false),
.is_some_and(|s| !s.is_empty()),
"kebab_version must be a non-empty string"
);
@@ -86,12 +85,12 @@ fn cli_schema_json_emits_schema_v1() {
.and_then(|c| c.as_object())
.expect("capabilities must be a JSON object");
assert_eq!(
caps.get("json_mode").and_then(|b| b.as_bool()),
caps.get("json_mode").and_then(serde_json::Value::as_bool),
Some(true),
"capabilities.json_mode must be true"
);
assert_eq!(
caps.get("mcp_server").and_then(|b| b.as_bool()),
caps.get("mcp_server").and_then(serde_json::Value::as_bool),
Some(true),
"capabilities.mcp_server must be true (fb-30)"
);

View File

@@ -155,8 +155,8 @@ fn ingest_json_progress_lines_carry_kind_and_ts() {
saw_completed = true;
// Counts mirror the report.
let counts = v.get("counts").unwrap();
assert_eq!(counts.get("scanned").and_then(|n| n.as_u64()), Some(2));
assert_eq!(counts.get("new").and_then(|n| n.as_u64()), Some(2));
assert_eq!(counts.get("scanned").and_then(serde_json::Value::as_u64), Some(2));
assert_eq!(counts.get("new").and_then(serde_json::Value::as_u64), Some(2));
}
}
assert!(saw_scan_started, "missing scan_started event");

View File

@@ -0,0 +1,254 @@
//! p9-fb-41 PR-4: CLI `--multi-hop` flag wiring + answer.v1 / error.v1
//! schema additivity.
//!
//! Four Ollama-free pins:
//!
//! 1. `--multi-hop` is exposed on `kebab ask --help` so users can
//! discover the flag at the CLI surface (clap-level smoke).
//! 2. `answer.schema.json` parses as valid JSON and declares a
//! `hops` property with a `HopRecord` `$defs` entry — guards
//! against accidental schema deletion / typo in future edits.
//! 3. `answer.schema.json`'s `refusal_reason` enum lists
//! `multi_hop_decompose_failed` — agents validating against
//! the schema accept the new variant on refusal answers.
//! 4. `error.schema.json`'s `code` enum lists
//! `multi_hop_decompose_failed` — forward-looking enum extension
//! documented in PR-4.
//!
//! End-to-end multi-hop ask against a live Ollama lands in a
//! follow-up `#[ignore]` test (same pattern as `wire_ask_stale.rs`).
use std::path::PathBuf;
use std::process::Command;
fn schema_path(name: &str) -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("..")
.join("..")
.join("docs")
.join("wire-schema")
.join("v1")
.join(name)
}
fn parse_schema(name: &str) -> serde_json::Value {
let text = std::fs::read_to_string(schema_path(name))
.unwrap_or_else(|e| panic!("read {name}: {e}"));
serde_json::from_str(&text)
.unwrap_or_else(|e| panic!("{name} must parse as valid JSON: {e}"))
}
#[test]
fn cli_ask_help_advertises_multi_hop_flag() {
let bin = env!("CARGO_BIN_EXE_kebab");
let out = Command::new(bin).args(["ask", "--help"]).output().unwrap();
let stdout = String::from_utf8_lossy(&out.stdout);
assert!(
stdout.contains("--multi-hop"),
"`kebab ask --help` must advertise --multi-hop so users can discover it:\n{stdout}"
);
}
#[test]
fn answer_schema_declares_hops_property_with_hop_record_defs() {
let schema = parse_schema("answer.schema.json");
assert!(
schema["properties"]["hops"].is_object(),
"`hops` property must be declared on answer.v1"
);
// `hops` allows array-or-null (single-pass omits the field;
// multi-hop emits a non-empty array).
let hops_any_of = schema["properties"]["hops"]["anyOf"]
.as_array()
.expect("hops must declare anyOf (array | null)");
assert!(
hops_any_of.iter().any(|v| v["type"] == "array"),
"hops anyOf must include array shape"
);
assert!(
hops_any_of.iter().any(|v| v["type"] == "null"),
"hops anyOf must include null (single-pass omits the field)"
);
// HopRecord $defs entry — guards against accidental deletion or
// structural drift in future schema edits.
let hop_record = &schema["$defs"]["HopRecord"];
assert!(
hop_record.is_object(),
"$defs.HopRecord must be declared so `hops.items` can $ref it"
);
let kind_enum = hop_record["properties"]["kind"]["enum"]
.as_array()
.expect("HopRecord.kind must be an enum");
let kinds: Vec<&str> = kind_enum.iter().filter_map(|v| v.as_str()).collect();
for needed in ["decompose", "decide", "synthesize"] {
assert!(
kinds.contains(&needed),
"HopRecord.kind enum must include {needed:?}, got {kinds:?}"
);
}
}
#[test]
fn answer_schema_refusal_reason_enum_includes_multi_hop_decompose_failed() {
let schema = parse_schema("answer.schema.json");
let refusal_any_of = schema["properties"]["refusal_reason"]["anyOf"]
.as_array()
.expect("refusal_reason must declare anyOf");
let enum_arr = refusal_any_of
.iter()
.find_map(|v| v["enum"].as_array())
.expect("one of refusal_reason.anyOf entries must declare an enum");
let values: Vec<&str> = enum_arr.iter().filter_map(|v| v.as_str()).collect();
assert!(
values.contains(&"multi_hop_decompose_failed"),
"refusal_reason enum must include `multi_hop_decompose_failed`, got {values:?}"
);
// All earlier RefusalReason wire values remain on the enum —
// guards against an accidental rewrite dropping old variants.
for needed in [
"score_gate",
"llm_self_judge",
"no_index",
"no_chunks",
"llm_stream_aborted",
] {
assert!(
values.contains(&needed),
"refusal_reason enum must keep prior variant {needed:?}, got {values:?}"
);
}
}
#[test]
fn error_schema_code_enum_includes_multi_hop_decompose_failed() {
let schema = parse_schema("error.schema.json");
let code_enum = schema["properties"]["code"]["enum"]
.as_array()
.expect("error.v1 must declare code.enum");
let values: Vec<&str> = code_enum.iter().filter_map(|v| v.as_str()).collect();
assert!(
values.contains(&"multi_hop_decompose_failed"),
"error.v1 code enum must include forward-looking `multi_hop_decompose_failed`, got {values:?}"
);
// Existing codes remain — guards against accidental deletion.
for needed in [
"config_invalid",
"not_indexed",
"model_unreachable",
"generic",
] {
assert!(
values.contains(&needed),
"error.v1 code enum must keep prior code {needed:?}, got {values:?}"
);
}
}
// ── p9-fb-41 PR-9c-1: NLI verification surface pins ─────────────────────
/// answer.v1 must declare a `verification` property AND a
/// `$defs.VerificationSummary` entry with all three required fields.
/// Guards against accidental schema deletion / typo in future edits.
#[test]
fn answer_schema_declares_verification_field_and_defs() {
let schema = parse_schema("answer.schema.json");
assert!(
schema["properties"]["verification"].is_object(),
"`verification` property must be declared on answer.v1"
);
// `verification` allows object-or-null (multi-hop with threshold>0
// emits an object; everything else omits the field).
let v_any_of = schema["properties"]["verification"]["anyOf"]
.as_array()
.expect("verification must declare anyOf (object | null)");
assert!(
v_any_of.iter().any(|v| v["type"] == "null"),
"verification anyOf must include null (single-pass / disabled gate omits the field)"
);
assert!(
v_any_of
.iter()
.any(|v| v["$ref"].as_str() == Some("#/$defs/VerificationSummary")),
"verification anyOf must $ref VerificationSummary"
);
// VerificationSummary $defs entry + required fields.
let vs = &schema["$defs"]["VerificationSummary"];
assert!(
vs.is_object(),
"$defs.VerificationSummary must be declared so verification.anyOf can $ref it"
);
let required: Vec<&str> = vs["required"]
.as_array()
.expect("VerificationSummary.required must be an array")
.iter()
.filter_map(|v| v.as_str())
.collect();
for needed in ["nli_score", "nli_threshold", "nli_passed"] {
assert!(
required.contains(&needed),
"VerificationSummary.required must include {needed:?}, got {required:?}"
);
}
}
#[test]
fn answer_schema_refusal_reason_enum_includes_nli_verification_failed() {
let schema = parse_schema("answer.schema.json");
let refusal_any_of = schema["properties"]["refusal_reason"]["anyOf"]
.as_array()
.expect("refusal_reason must declare anyOf");
let enum_arr = refusal_any_of
.iter()
.find_map(|v| v["enum"].as_array())
.expect("one of refusal_reason.anyOf entries must declare an enum");
let values: Vec<&str> = enum_arr.iter().filter_map(|v| v.as_str()).collect();
assert!(
values.contains(&"nli_verification_failed"),
"refusal_reason enum must include `nli_verification_failed`, got {values:?}"
);
}
#[test]
fn answer_schema_refusal_reason_enum_includes_nli_model_unavailable() {
let schema = parse_schema("answer.schema.json");
let refusal_any_of = schema["properties"]["refusal_reason"]["anyOf"]
.as_array()
.expect("refusal_reason must declare anyOf");
let enum_arr = refusal_any_of
.iter()
.find_map(|v| v["enum"].as_array())
.expect("one of refusal_reason.anyOf entries must declare an enum");
let values: Vec<&str> = enum_arr.iter().filter_map(|v| v.as_str()).collect();
assert!(
values.contains(&"nli_model_unavailable"),
"refusal_reason enum must include `nli_model_unavailable`, got {values:?}"
);
}
#[test]
fn error_schema_code_enum_includes_nli_verification_failed() {
let schema = parse_schema("error.schema.json");
let code_enum = schema["properties"]["code"]["enum"]
.as_array()
.expect("error.v1 must declare code.enum");
let values: Vec<&str> = code_enum.iter().filter_map(|v| v.as_str()).collect();
assert!(
values.contains(&"nli_verification_failed"),
"error.v1 code enum must include forward-looking `nli_verification_failed`, got {values:?}"
);
}
#[test]
fn error_schema_code_enum_includes_nli_model_unavailable() {
let schema = parse_schema("error.schema.json");
let code_enum = schema["properties"]["code"]["enum"]
.as_array()
.expect("error.v1 must declare code.enum");
let values: Vec<&str> = code_enum.iter().filter_map(|v| v.as_str()).collect();
assert!(
values.contains(&"nli_model_unavailable"),
"error.v1 code enum must include forward-looking `nli_model_unavailable`, got {values:?}"
);
}

View File

@@ -47,8 +47,20 @@ fn search_json_emits_search_response_v1_wrapper() {
fn search_json_truncates_with_max_tokens() {
let dir = tempfile::tempdir().unwrap();
let (cfg, workspace, _data) = common::write_config(dir.path(), 30);
let body: String = "rust ownership is a memory model. ".repeat(10);
fs::write(workspace.join("a.md"), format!("# T\n\n{body}\n")).unwrap();
// v0.17.0 trigram tokenizer makes FTS5 snippet() tokens 3-char wide
// (was full words under unicode61), so an individual snippet stays
// around ~60 chars — too short to ever exceed the snippet-shorten
// budget cap on a single-hit fixture. To still exercise the budget
// loop deterministically, we ingest multiple hits and pick a budget
// small enough that the loop has to *pop* hits, which flips
// truncated=true regardless of snippet length.
for i in 0..5 {
fs::write(
workspace.join(format!("d{i}.md")),
format!("# T{i}\n\nrust ownership is a memory model.\n"),
)
.unwrap();
}
common::ingest(&cfg, &workspace);
let (stdout, _stderr) = common::run_search_with_args(
@@ -211,8 +223,15 @@ fn search_stale_cursor_returns_error_v1_with_stale_cursor_code() {
fn search_plain_emits_truncated_hint_to_stderr() {
let dir = tempfile::tempdir().unwrap();
let (cfg, workspace, _data) = common::write_config(dir.path(), 30);
let body: String = "rust ownership is a memory model. ".repeat(10);
fs::write(workspace.join("a.md"), format!("# T\n\n{body}\n")).unwrap();
// v0.17.0 trigram tokenizer — same multi-doc rationale as
// `search_json_truncates_with_max_tokens` above.
for i in 0..5 {
fs::write(
workspace.join(format!("d{i}.md")),
format!("# T{i}\n\nrust ownership is a memory model.\n"),
)
.unwrap();
}
common::ingest(&cfg, &workspace);
let (_stdout, stderr) = common::run_search_with_args(
@@ -224,3 +243,76 @@ fn search_plain_emits_truncated_hint_to_stderr() {
"stderr must carry truncated hint: {stderr:?}"
);
}
#[test]
fn search_plain_emits_short_query_hint_to_stderr() {
// v0.17.0 A5 Step 6: 2-char query under trigram tokenizer emits
// empty hits + stderr `[hint]` advisory. Empty workspace is enough
// — hits are always empty so the hint condition depends only on
// query length (<3 chars trimmed) + non-raw mode + hits.is_empty.
let dir = tempfile::tempdir().unwrap();
let (cfg, workspace, _data) = common::write_config(dir.path(), 30);
common::ingest(&cfg, &workspace);
let (_stdout, stderr) = common::run_search_with_args(
&cfg,
&["--mode", "lexical", "ab"],
);
assert!(
stderr.contains("[hint]"),
"stderr must carry short-query hint: {stderr:?}"
);
assert!(
stderr.contains("3자 이상"),
"hint message must mention '3자 이상' (Korean advisory): {stderr:?}"
);
}
#[test]
fn search_json_emits_hint_field_for_short_query() {
// v0.17.0 A5 Step 6: --json mode carries the same advisory on the
// `search_response.v1.hint` additive field. Empty hits + 2-char
// query + non-raw mode trips the helper. Verifies the MCP-visible
// surface (agents read the field instead of parsing stderr).
let dir = tempfile::tempdir().unwrap();
let (cfg, workspace, _data) = common::write_config(dir.path(), 30);
common::ingest(&cfg, &workspace);
let (stdout, _stderr) = common::run_search_with_args(
&cfg,
&["--json", "--mode", "lexical", "ab"],
);
let v: Value = serde_json::from_str(stdout.trim())
.unwrap_or_else(|e| panic!("not JSON: {stdout:?}: {e}"));
assert!(
v["hits"].as_array().unwrap().is_empty(),
"empty hits expected for short query in empty KB: {v}"
);
assert_eq!(
v["hint"].as_str().expect("hint field set on short empty result"),
"3자 이상 키워드 권장 (trigram tokenizer 제약)",
"hint must carry the standard advisory: {v}"
);
}
#[test]
fn search_json_omits_hint_field_when_query_is_long_enough() {
// v0.17.0 A5 Step 6 (negative case): 3+ char query never trips
// hint, even on an empty KB. Verifies `serialize_search_response`
// omits the additive `hint` field when `None` so existing wire
// consumers stay backward-compatible.
let dir = tempfile::tempdir().unwrap();
let (cfg, workspace, _data) = common::write_config(dir.path(), 30);
common::ingest(&cfg, &workspace);
let (stdout, _stderr) = common::run_search_with_args(
&cfg,
&["--json", "--mode", "lexical", "abc"],
);
let v: Value = serde_json::from_str(stdout.trim())
.unwrap_or_else(|e| panic!("not JSON: {stdout:?}: {e}"));
assert!(
v.get("hint").is_none(),
"hint must be absent for ≥3-char queries: {v}"
);
}

View File

@@ -22,3 +22,6 @@ tracing = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
[lints]
workspace = true

View File

@@ -103,6 +103,34 @@ pub struct ChunkingCfg {
pub struct ModelsCfg {
pub embedding: EmbeddingModelCfg,
pub llm: LlmCfg,
/// p9-fb-41 PR-9c-1: NLI verifier model + provider knob.
/// `#[serde(default)]` so pre-v0.18 config files that predate the
/// `[models.nli]` section still load with built-in defaults
/// (`Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7` / `onnx`).
/// The verifier itself is gated by `[rag].nli_threshold` — even
/// with a model configured here, threshold `0.0` (the default)
/// skips the verification step entirely.
#[serde(default = "NliCfg::defaults")]
pub nli: NliCfg,
}
/// p9-fb-41 PR-9c-1: NLI verifier configuration. The model id flows to
/// `OnnxNliVerifier::new` via `kebab-nli` (PR-9c-2 wiring); the provider
/// is reserved for future verifier swap-in (currently only `"onnx"` is
/// recognized — anything else falls back to the same path).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct NliCfg {
pub model: String,
pub provider: String,
}
impl NliCfg {
pub fn defaults() -> Self {
Self {
model: "Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7".to_string(),
provider: "onnx".to_string(),
}
}
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -122,6 +150,23 @@ pub struct LlmCfg {
pub endpoint: String,
pub temperature: f32,
pub seed: u64,
/// v0.17.0 post-dogfood: Hard ceiling on a single HTTP exchange to
/// the LLM endpoint (Ollama, etc.). Cold-loading an 8B+ model on
/// CPU-only hosts can spend 60-90s on model load + several minutes
/// on a first inference, blowing past the old hard-coded 300s cap
/// and surfacing as `error: kb-rag: llm.generate_stream` to the
/// user. Config-driven so 16-GB / CPU-only deployments using small
/// (≤4B) models can keep the original 300s and large-model dogfood
/// can dial it up (e.g. 1200s) without rebuilding.
///
/// **Edge case — `0` is NOT a disable sentinel.**
/// `reqwest::ClientBuilder::timeout(Duration::from_secs(0))` sets a
/// 0-second read timeout, so every request fails *immediately* with
/// `error: kb-rag: ollama timeout`. To approximate "no cap", use a
/// large finite value (e.g. `u64::MAX` ≈ 5.8 × 10¹¹ years, or
/// just a generous number like `86400`).
#[serde(default = "default_llm_request_timeout_secs")]
pub request_timeout_secs: u64,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -147,6 +192,13 @@ fn default_cache_capacity() -> usize {
256
}
/// v0.17.0 post-dogfood: matches the legacy hard-coded ceiling so
/// existing configs that omit the field keep behaving identically.
/// Overridable per config / `KEBAB_MODELS_LLM_REQUEST_TIMEOUT_SECS`.
fn default_llm_request_timeout_secs() -> u64 {
300
}
fn default_stale_threshold_days() -> u32 {
30
}
@@ -157,6 +209,73 @@ pub struct RagCfg {
pub score_gate: f32,
pub explain_default: bool,
pub max_context_tokens: usize,
/// p9-fb-41: hard ceiling on the number of multi-hop iterations
/// (decompose iter + decide iters). When the LLM keeps returning
/// `continue` past this depth the pipeline cuts to `synthesize`
/// with `HopRecord.forced_stop = true`. Default `3` — enough for
/// most cross-doc reasoning, low enough to bound LLM cost.
#[serde(default = "default_multi_hop_max_depth")]
pub multi_hop_max_depth: u32,
/// p9-fb-41: cap on how many sub-queries the LLM may emit in a
/// single decompose / decide call. This is the *prompt-side
/// soft hint* — the value the pipeline injects into the
/// decompose / decide prompts so the LLM knows what to aim for.
/// kebab-rag enforces a separate compile-time hard ceiling
/// (`MULTI_HOP_MAX_SUB_QUERIES_HARD_CAP`, currently 10) as a
/// safety net against misbehaving models — if you raise this
/// knob above the hard cap, bump the const in the same PR.
/// Default `5`.
#[serde(default = "default_multi_hop_max_sub_queries_per_iter")]
pub multi_hop_max_sub_queries_per_iter: u32,
/// p9-fb-41: hard ceiling on the deduped chunk pool. When the
/// accumulated pool would exceed this many chunks the pipeline
/// stops accepting new retrieval results and forces synthesize
/// with `forced_stop = true`.
///
/// Default `15` — tuned down from the original 30 in the v0.18
/// pre-cut dogfood (`tasks/HOTFIXES.md` 2026-05-25 fb-41 entry,
/// "post-PR-7 dogfood retest + PR-8 partial mitigation" sub-section).
/// With 30 chunks the synthesize prompt was large enough for
/// gemma3:4b to lose the citation rule + drift into unrelated
/// chunks; 15 keeps the prompt tight while still allowing 3-iter
/// cross-doc reasoning over ~5 chunks per iter.
#[serde(default = "default_multi_hop_max_pool_chunks")]
pub multi_hop_max_pool_chunks: u32,
/// p9-fb-41 PR-9c-1: minimum NLI entailment score required for the
/// multi-hop synthesize answer to be returned as `grounded=true`
/// (spec §2.6 single gate). When the post-synthesize NLI verifier
/// returns `NliScores::faithfulness() < nli_threshold` the
/// pipeline refuses with `RefusalReason::NliVerificationFailed`.
///
/// Default `0.0` = verification disabled — no NLI call, multi-hop
/// matches its PR-3b behavior exactly. Set to e.g. `0.5` to
/// activate the gate. Knob lives on `[rag]` (the gate is a RAG
/// policy, not a model property); the model itself comes from
/// `[models.nli].model`.
///
/// Single-pass `ask` ignores this knob entirely — only multi-hop
/// runs through the verification step (PR-9c-2 wires it).
#[serde(default = "default_nli_threshold")]
pub nli_threshold: f32,
}
fn default_multi_hop_max_depth() -> u32 {
3
}
fn default_multi_hop_max_sub_queries_per_iter() -> u32 {
5
}
fn default_multi_hop_max_pool_chunks() -> u32 {
15
}
/// p9-fb-41 PR-9c-1: NLI gate disabled by default per spec §2.6
/// (verification opt-in — users explicitly raise the threshold once
/// they're ready to trade refusal-rate for groundedness).
fn default_nli_threshold() -> f32 {
0.0
}
/// Settings for the image ingest pipeline (P6). `ocr` controls OCR
@@ -204,6 +323,22 @@ pub struct OcrCfg {
/// Cap the long edge of the image (in pixels) before sending. Larger
/// images bloat prompt cost. Default `1600`.
pub max_pixels: u32,
/// v0.17.2 post-dogfood: Hard ceiling on a single HTTP exchange to
/// the OCR endpoint. Sister knob to [`LlmCfg::request_timeout_secs`]
/// — kept separate because OCR latency is typically shorter than
/// chat-LLM cold start, and large vision models on CPU-only hosts
/// occasionally need a different budget. See HOTFIXES 2026-05-25
/// for the rationale.
///
/// **Edge case — `0` is NOT a disable sentinel.** Same semantics as
/// [`LlmCfg::request_timeout_secs`]: `Duration::from_secs(0)` means
/// "every request fails immediately" (reqwest 0.12.x — the read
/// timeout is applied as a 0-second deadline), not "no timeout".
/// To approximate "no cap", use a large finite value (e.g.
/// `u64::MAX` ≈ 5.8 × 10¹¹ years, or just a generous number like
/// `86400`).
#[serde(default = "default_ocr_request_timeout_secs")]
pub request_timeout_secs: u64,
}
impl OcrCfg {
@@ -215,10 +350,18 @@ impl OcrCfg {
endpoint: None,
languages: vec!["eng".to_string(), "kor".to_string()],
max_pixels: 1600,
request_timeout_secs: default_ocr_request_timeout_secs(),
}
}
}
/// v0.17.2 post-dogfood: matches the legacy hard-coded ceiling so
/// existing configs that omit the field keep behaving identically.
/// Overridable per config / `KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS`.
fn default_ocr_request_timeout_secs() -> u64 {
300
}
/// Caption settings (P6-3). Caption uses the same Ollama-vision /
/// `LanguageModel` pipeline as the rest of the workspace; the trait
/// abstraction is the part the spec demands. `enabled` defaults to
@@ -363,13 +506,16 @@ impl Config {
// gemma4 계열 통일 — OCR (P6-2) + caption (P6-3)
// 어댑터가 같은 family 사용. 사용자가 더 큰
// variant (gemma4:26b 등) 원하면 자기 config.toml
// 에서 override.
// 에서 override. CPU-only / ≤16 GB RAM 환경이면
// gemma3:4b 같은 ≤4B Q4 모델 권장 (README 참조).
model: "gemma4:e4b".to_string(),
context_tokens: 32768,
endpoint: "http://127.0.0.1:11434".to_string(),
temperature: 0.0,
seed: 0,
request_timeout_secs: default_llm_request_timeout_secs(),
},
nli: NliCfg::defaults(),
},
search: SearchCfg {
default_k: 10,
@@ -384,6 +530,11 @@ impl Config {
score_gate: 0.30,
explain_default: false,
max_context_tokens: 8000,
multi_hop_max_depth: default_multi_hop_max_depth(),
multi_hop_max_sub_queries_per_iter:
default_multi_hop_max_sub_queries_per_iter(),
multi_hop_max_pool_chunks: default_multi_hop_max_pool_chunks(),
nli_threshold: default_nli_threshold(),
},
image: ImageCfg::defaults(),
ui: UiCfg::defaults(),
@@ -621,6 +772,15 @@ impl Config {
self.models.llm.seed = n;
}
}
"KEBAB_MODELS_LLM_REQUEST_TIMEOUT_SECS" => {
if let Ok(n) = v.parse::<u64>() {
self.models.llm.request_timeout_secs = n;
}
}
// models.nli (p9-fb-41 PR-9c-1)
"KEBAB_MODELS_NLI_MODEL" => self.models.nli.model = v.clone(),
"KEBAB_MODELS_NLI_PROVIDER" => self.models.nli.provider = v.clone(),
// search
"KEBAB_SEARCH_DEFAULT_K" => {
@@ -662,6 +822,39 @@ impl Config {
self.rag.max_context_tokens = n;
}
}
"KEBAB_RAG_MULTI_HOP_MAX_DEPTH" => {
if let Ok(n) = v.parse::<u32>() {
self.rag.multi_hop_max_depth = n;
}
}
"KEBAB_RAG_MULTI_HOP_MAX_SUB_QUERIES_PER_ITER" => {
if let Ok(n) = v.parse::<u32>() {
self.rag.multi_hop_max_sub_queries_per_iter = n;
}
}
"KEBAB_RAG_MULTI_HOP_MAX_POOL_CHUNKS" => {
if let Ok(n) = v.parse::<u32>() {
self.rag.multi_hop_max_pool_chunks = n;
}
}
// p9-fb-41 PR-9c-1: NLI gate threshold. Parse failure
// emits a `tracing::warn!` (not silent like the other
// numeric env overrides) because this knob gates the
// NLI verification entirely — a malformed env value
// would silently disable a security-flavored gate the
// user thought they enabled, which is the failure mode
// most worth surfacing. The default (`0.0`) survives
// on parse failure so behaviour stays well-defined.
"KEBAB_RAG_NLI_THRESHOLD" => match v.parse::<f32>() {
Ok(f) => self.rag.nli_threshold = f,
Err(e) => tracing::warn!(
target: "kebab-config",
env_key = "KEBAB_RAG_NLI_THRESHOLD",
env_value = %v,
error = %e,
"invalid KEBAB_RAG_NLI_THRESHOLD; keeping prior value (0.0 = NLI gate disabled)"
),
},
// image.ocr
"KEBAB_IMAGE_OCR_ENABLED" => {
@@ -691,6 +884,11 @@ impl Config {
self.image.ocr.max_pixels = n;
}
}
"KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS" => {
if let Ok(n) = v.parse::<u64>() {
self.image.ocr.request_timeout_secs = n;
}
}
// image.caption (P6-3)
"KEBAB_IMAGE_CAPTION_ENABLED" => {
@@ -803,6 +1001,83 @@ fn parse_bool(s: &str) -> bool {
mod tests {
use super::*;
/// Legacy TOML fixture written before the `request_timeout_secs`
/// knobs (LLM in v0.17.1, OCR follow-up) existed. Shared by
/// `legacy_config_without_request_timeout_secs_uses_default`
/// (LLM-side) and `legacy_config_without_ocr_request_timeout_secs_uses_default`
/// (OCR-side) so both invariants pin against the same on-disk
/// shape — schema drift in the legacy form only needs one edit.
const LEGACY_PRE_TIMEOUT_TOML: &str = r#"
schema_version = 1
[workspace]
root = "/tmp/x"
exclude = []
[storage]
data_dir = "/tmp/x"
sqlite = "/tmp/x/kebab.sqlite"
vector_dir = "/tmp/x/lancedb"
asset_dir = "/tmp/x/assets"
artifact_dir = "/tmp/x/artifacts"
model_dir = "/tmp/x/models"
runs_dir = "/tmp/x/runs"
copy_threshold_mb = 100
[indexing]
max_parallel_extractors = 2
max_parallel_embeddings = 1
watch_filesystem = false
[chunking]
target_tokens = 500
overlap_tokens = 80
respect_markdown_headings = true
chunker_version = "md-heading-v1"
[models.embedding]
provider = "fastembed"
model = "multilingual-e5-large"
version = "v1"
dimensions = 1024
batch_size = 64
[models.llm]
provider = "ollama"
model = "gemma3:4b"
context_tokens = 4096
endpoint = "http://127.0.0.1:11434"
temperature = 0.0
seed = 0
[search]
default_k = 10
hybrid_fusion = "rrf"
rrf_k = 60
snippet_chars = 220
[rag]
prompt_template_version = "rag-v2"
score_gate = 0.3
explain_default = false
max_context_tokens = 8000
[image.ocr]
enabled = false
engine = "ollama-vision"
model = "gemma3:4b"
languages = ["eng"]
max_pixels = 1600
[image.caption]
enabled = false
max_pixels = 768
prompt_template_version = "caption-v1"
[ui]
theme = "dark"
"#;
#[test]
fn defaults_are_serde_roundtrip_stable() {
let c = Config::defaults();
@@ -873,6 +1148,35 @@ mod tests {
assert!((c.models.llm.temperature - 0.7).abs() < 1e-6);
}
/// v0.17.0 post-dogfood: matches the legacy hard-coded 300s cap so
/// existing configs that omit the new field are not affected.
#[test]
fn default_llm_request_timeout_secs_is_300() {
assert_eq!(Config::defaults().models.llm.request_timeout_secs, 300);
}
#[test]
fn env_overrides_models_llm_request_timeout_secs() {
let mut env = HashMap::new();
env.insert(
"KEBAB_MODELS_LLM_REQUEST_TIMEOUT_SECS".to_string(),
"1200".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(c.models.llm.request_timeout_secs, 1200);
}
/// v0.17.0 post-dogfood: a config file written before the field
/// existed (no `request_timeout_secs` key) must still parse and fall
/// back to the 300s default — backwards-compat invariant. Fixture
/// shared with the OCR-side invariant via [`LEGACY_PRE_TIMEOUT_TOML`].
#[test]
fn legacy_config_without_request_timeout_secs_uses_default() {
let c: Config = toml::from_str(LEGACY_PRE_TIMEOUT_TOML)
.expect("parse legacy config");
assert_eq!(c.models.llm.request_timeout_secs, 300);
}
#[test]
fn env_overrides_indexing_watch_filesystem_bool() {
let mut env = HashMap::new();
@@ -894,6 +1198,175 @@ mod tests {
assert_eq!(c.image.ocr.max_pixels, 1600);
}
/// v0.17.2 post-dogfood: matches the legacy hard-coded 300s cap so
/// existing configs that omit the new field keep behaving identically.
#[test]
fn default_ocr_request_timeout_secs_is_300() {
assert_eq!(
Config::defaults().image.ocr.request_timeout_secs,
300
);
}
#[test]
fn env_overrides_image_ocr_request_timeout_secs() {
let mut env = HashMap::new();
env.insert(
"KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS".to_string(),
"900".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(c.image.ocr.request_timeout_secs, 900);
}
/// post-v0.17.1 dogfood: a config file written before the OCR
/// timeout field existed must still parse and fall back to the
/// 300s default — backwards-compat invariant. Fixture shared
/// with the LLM-side invariant via [`LEGACY_PRE_TIMEOUT_TOML`].
#[test]
fn legacy_config_without_ocr_request_timeout_secs_uses_default() {
let c: Config = toml::from_str(LEGACY_PRE_TIMEOUT_TOML)
.expect("parse legacy config");
assert_eq!(c.image.ocr.request_timeout_secs, 300);
}
// ── p9-fb-41: multi-hop RAG knobs ────────────────────────────────────
#[test]
fn default_multi_hop_max_depth_is_3() {
assert_eq!(Config::defaults().rag.multi_hop_max_depth, 3);
}
#[test]
fn default_multi_hop_max_sub_queries_per_iter_is_5() {
assert_eq!(
Config::defaults().rag.multi_hop_max_sub_queries_per_iter,
5
);
}
#[test]
fn default_multi_hop_max_pool_chunks_is_15() {
// v0.18 dogfood (HOTFIXES 2026-05-25 fb-41 post-PR-7) tuned
// this down from 30 → 15 to keep the synthesize prompt tight
// enough for gemma3:4b to follow the citation rule.
assert_eq!(Config::defaults().rag.multi_hop_max_pool_chunks, 15);
}
#[test]
fn env_overrides_multi_hop_knobs() {
let mut env = HashMap::new();
env.insert(
"KEBAB_RAG_MULTI_HOP_MAX_DEPTH".to_string(),
"5".to_string(),
);
env.insert(
"KEBAB_RAG_MULTI_HOP_MAX_SUB_QUERIES_PER_ITER".to_string(),
"7".to_string(),
);
env.insert(
"KEBAB_RAG_MULTI_HOP_MAX_POOL_CHUNKS".to_string(),
"50".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(c.rag.multi_hop_max_depth, 5);
assert_eq!(c.rag.multi_hop_max_sub_queries_per_iter, 7);
assert_eq!(c.rag.multi_hop_max_pool_chunks, 50);
}
/// post-PR-3 fb-41: a config file written before the multi-hop
/// knobs existed must still parse and fall back to the documented
/// defaults — backwards-compat invariant. Fixture shared with the
/// LLM / OCR timeout invariants via [`LEGACY_PRE_TIMEOUT_TOML`]
/// (that fixture also predates the multi_hop_* fields).
#[test]
fn legacy_config_without_multi_hop_knobs_uses_defaults() {
let c: Config = toml::from_str(LEGACY_PRE_TIMEOUT_TOML)
.expect("parse legacy config");
assert_eq!(c.rag.multi_hop_max_depth, 3);
assert_eq!(c.rag.multi_hop_max_sub_queries_per_iter, 5);
// v0.18 dogfood (post-PR-7): pool default 30 → 15.
assert_eq!(c.rag.multi_hop_max_pool_chunks, 15);
}
// ── p9-fb-41 PR-9c-1: NLI verification knobs ─────────────────────────
#[test]
fn default_nli_threshold_is_zero() {
// Spec §2.6: NLI gate disabled by default — verification is
// opt-in. `0.0` keeps multi-hop behavior identical to PR-3b.
assert_eq!(Config::defaults().rag.nli_threshold, 0.0);
}
#[test]
fn default_nli_model_is_xenova_mdeberta() {
// Pin the default model id so a refactor that touches NliCfg
// can't silently flip to a different verifier model.
assert_eq!(
Config::defaults().models.nli.model,
"Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7"
);
assert_eq!(Config::defaults().models.nli.provider, "onnx");
}
/// A config file written before the `[models.nli]` / `nli_threshold`
/// keys existed must still parse and fall back to the documented
/// defaults. Fixture shared via [`LEGACY_PRE_TIMEOUT_TOML`] (predates
/// all PR-9c-1 fields).
#[test]
fn legacy_config_without_nli_uses_defaults() {
let c: Config = toml::from_str(LEGACY_PRE_TIMEOUT_TOML)
.expect("parse legacy config");
assert_eq!(c.rag.nli_threshold, 0.0);
assert_eq!(
c.models.nli.model,
"Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7"
);
assert_eq!(c.models.nli.provider, "onnx");
}
#[test]
fn env_override_nli_threshold() {
let mut env = HashMap::new();
env.insert("KEBAB_RAG_NLI_THRESHOLD".to_string(), "0.5".to_string());
let c = Config::defaults().apply_env(&env);
assert!((c.rag.nli_threshold - 0.5).abs() < 1e-6);
}
#[test]
fn env_override_nli_model_and_provider() {
let mut env = HashMap::new();
env.insert(
"KEBAB_MODELS_NLI_MODEL".to_string(),
"user/custom-nli-model".to_string(),
);
env.insert(
"KEBAB_MODELS_NLI_PROVIDER".to_string(),
"candle".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(c.models.nli.model, "user/custom-nli-model");
assert_eq!(c.models.nli.provider, "candle");
}
/// Malformed `KEBAB_RAG_NLI_THRESHOLD` keeps the prior value (does
/// NOT silently disable nor crash). The `tracing::warn!` surface
/// is observable only when the user has tracing wired; the
/// behavior contract is "default survives".
#[test]
fn env_malformed_nli_threshold_keeps_prior_value() {
let mut env = HashMap::new();
env.insert(
"KEBAB_RAG_NLI_THRESHOLD".to_string(),
"not-a-float".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(
c.rag.nli_threshold, 0.0,
"malformed env value must keep the default unchanged"
);
}
#[test]
fn image_ocr_env_overrides() {
let mut env = HashMap::new();

View File

@@ -157,7 +157,7 @@ mod tests {
#[test]
fn xdg_data_home_set_replaces_var() {
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let _guard = XdgGuard::capture();
// SAFETY: lock held for the duration of this test.
unsafe { std::env::set_var("XDG_DATA_HOME", "/custom/path") };
@@ -168,7 +168,7 @@ mod tests {
#[test]
fn xdg_data_home_unset_uses_default() {
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let _guard = XdgGuard::capture();
// SAFETY: lock held for the duration of this test.
unsafe { std::env::remove_var("XDG_DATA_HOME") };
@@ -181,7 +181,7 @@ mod tests {
#[test]
fn xdg_with_no_default_resolves_to_empty_when_unset() {
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let _guard = XdgGuard::capture();
// SAFETY: lock held for the duration of this test.
unsafe { std::env::remove_var("XDG_DATA_HOME") };
@@ -193,7 +193,7 @@ mod tests {
#[test]
fn leading_tilde_expands_to_home() {
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let home = std::env::var("HOME").expect("HOME must be set in tests");
let p = expand_path("~/runs", "");
assert_eq!(p, PathBuf::from(home).join("runs"));
@@ -229,7 +229,7 @@ mod tests {
#[test]
fn tilde_path_ignores_base_dir() {
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let home = std::env::var("HOME").expect("HOME must be set in tests");
let base = Path::new("/tmp/ignored-cfg");
let p = expand_path_with_base("~/x", "", base);
@@ -238,7 +238,7 @@ mod tests {
#[test]
fn xdg_var_path_ignores_base_dir() {
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let _guard = XdgGuard::capture();
// SAFETY: lock held for the duration of this test.
unsafe { std::env::set_var("XDG_DATA_HOME", "/xdg/data") };
@@ -255,7 +255,7 @@ mod tests {
// Order matters: substitute `{data_dir}` (which itself contains
// an unexpanded `${XDG_DATA_HOME}` and `~`), then the other two
// resolve the result.
let _lock = ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _lock = ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let _guard = XdgGuard::capture();
// SAFETY: lock held for the duration of this test.
unsafe { std::env::set_var("XDG_DATA_HOME", "/xdg/data") };

View File

@@ -16,3 +16,6 @@ time = { workspace = true }
blake3 = { workspace = true }
serde_json_canonicalizer = "0.3"
unicode-normalization = "0.1"
[lints]
workspace = true

View File

@@ -29,6 +29,37 @@ pub struct Answer {
/// 이면 single-shot.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub turn_index: Option<u32>,
/// p9-fb-41: multi-hop hop trace. `None` for single-pass asks.
/// Each entry records one hop (`decompose` / `decide` / `synthesize`)
/// — the LLM call category, the sub-queries emitted, retrieval
/// counts, and a `forced_stop` flag for cap-driven termination.
/// Wire-additive: `answer.v1` schema_version unchanged; consumers
/// reading older single-pass answers see `hops: None` (or absent).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub hops: Option<Vec<HopRecord>>,
/// p9-fb-41 PR-9c-1: NLI-based post-synthesis verification summary.
/// `None` for single-pass asks and for multi-hop runs with
/// `[rag].nli_threshold == 0` (verification disabled — the default).
/// Present only when the multi-hop pipeline reached the post-
/// synthesize verification step (PR-9c-2 wires step 8.5). Wire-
/// additive: `answer.v1` schema_version unchanged; consumers
/// reading pre-v0.18 answers see `verification: None` (or absent).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub verification: Option<VerificationSummary>,
}
/// p9-fb-41 PR-9c-1: post-synthesize NLI verification summary stamped
/// onto [`Answer::verification`] when multi-hop runs reach step 8.5
/// (NLI gate). Three required fields ride together on every wire emit:
/// `nli_score` is the entailment channel of the XNLI verifier,
/// `nli_threshold` mirrors `[rag].nli_threshold` for audit, and
/// `nli_passed` is `nli_score >= nli_threshold`. The whole struct is
/// omitted (serde skip) when no verification ran.
#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct VerificationSummary {
pub nli_score: f32,
pub nli_threshold: f32,
pub nli_passed: bool,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -55,6 +86,79 @@ pub struct Turn {
pub created_at: OffsetDateTime,
}
/// p9-fb-41: one entry in [`Answer::hops`] — the per-iteration trace
/// of a multi-hop ask. The pipeline appends a `HopRecord` per LLM
/// call (decompose / decide / synthesize) so a `--multi-hop` user
/// can see what sub-queries the LLM emitted, how many chunks each
/// hop contributed, whether the iter stopped on the model's own
/// signal or hit a cap, and the per-hop LLM latency.
///
/// Wire-additive — every field uses `#[serde(default)]` where it
/// could plausibly be omitted by a future schema reader.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct HopRecord {
/// 0-based hop index within this ask. `iter=0` is always the
/// initial decompose call; subsequent iters are decide calls;
/// the final iter is the synthesize call.
pub iter: u32,
pub kind: HopKind,
/// Sub-queries associated with this hop. The meaning depends on
/// `kind`:
///
/// - [`HopKind::Decompose`]: the initial sub-queries the LLM
/// broke the original user query into. These drive the
/// `iter=1` retrieval round.
/// - [`HopKind::Decide`]: the *new* sub-queries the LLM
/// emitted to drive the next retrieval round. Empty when the
/// LLM signalled stop OR when `forced_stop = true` (cap hit
/// or parse-degraded).
/// - [`HopKind::Synthesize`]: always empty — the final hop
/// produces the user-visible answer, not more sub-queries.
#[serde(default)]
pub sub_queries: Vec<String>,
/// Number of *new* chunks the retrieval round contributed to the
/// pool (dedup'd by `chunk_id` — repeated hits from a previous
/// iter do not count). `0` for the decompose hop (no retrieval
/// yet) and the synthesize hop.
pub context_chunks_added: u32,
/// `true` when the pipeline cut the iter loop short because a
/// safety cap fired (`max_depth` / `max_total_sub_queries` /
/// `max_pool_chunks`) rather than because the LLM signalled
/// stop. The user-visible answer still reflects all chunks
/// accumulated up to that point — `forced_stop` is a tracing
/// signal, not a refusal.
pub forced_stop: bool,
/// Wall-clock latency of the LLM call for this hop, in
/// milliseconds. Useful for cost / latency analysis when a
/// `kebab eval` run records `Answer.hops`.
///
/// `0` is overloaded: it means "no LLM call happened at this
/// hop" when (a) the hop was a Decide skipped due to
/// `forced_stop` (depth-cap or pool-cap fired before the LLM
/// was asked) or (b) the pool was empty before any decide
/// could run. Treat `0` as "absent or instantaneous" rather
/// than as a genuine measurement.
pub llm_call_ms: u32,
}
/// p9-fb-41: which stage of the multi-hop pipeline a [`HopRecord`]
/// describes. The serde tag matches the wire shape so agents /
/// CLIs can branch on the snake_case string without referencing
/// the Rust enum.
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum HopKind {
/// First hop — LLM decomposed the user query into sub-queries.
Decompose,
/// Subsequent hop — LLM was asked whether more retrieval is
/// needed and either emitted new sub-queries (`continue`) or
/// returned an empty array (`stop`).
Decide,
/// Terminal hop — LLM produced the final user-visible answer
/// over the accumulated chunk pool.
Synthesize,
}
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RefusalReason {
@@ -66,6 +170,24 @@ pub enum RefusalReason {
/// 가 채워져 있을 수 있음 (사용자가 본 부분까지). RAG retrieval
/// 자체는 정상 — 모델 generation 단계에서만 중단.
LlmStreamAborted,
/// p9-fb-41: multi-hop pipeline 의 decompose LLM call 이 JSON
/// parse 가능한 sub-question array 를 반환하지 못함 (parse
/// error, 빈 응답, 또는 잘못된 형식). retrieval / synthesize
/// 단계 진입 못 함. CLI / MCP / TUI 가 받는 wire error code
/// = `"multi_hop_decompose_failed"` (PR-4 의 error_wire 매핑).
MultiHopDecomposeFailed,
/// p9-fb-41 PR-9c-1: post-synthesize NLI verification gate fired —
/// `NliScores::faithfulness()` (entailment channel) fell below
/// `[rag].nli_threshold`. Wire string = `"nli_verification_failed"`
/// (single source of truth: also the matching `error.v1.code`).
/// Multi-hop only; behavior wiring lands in PR-9c-2.
NliVerificationFailed,
/// p9-fb-41 PR-9c-1: NLI verifier was configured (threshold > 0)
/// but the model / runtime is unavailable (download failure,
/// missing tokenizer, ONNX session init error). Treated as a soft
/// refusal — the user sees an unverified-answer outcome rather
/// than crashing the ask. Wire string = `"nli_model_unavailable"`.
NliModelUnavailable,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -103,6 +225,81 @@ mod tests {
use crate::citation::Citation;
use time::macros::datetime;
/// p9-fb-41 PR-9c-1: pin the wire-side spelling of the new
/// `RefusalReason` variants. The strings here must match
/// `answer.schema.json::refusal_reason.enum` AND
/// `error.schema.json::code.enum` byte-for-byte (single source of
/// truth per spec §2.4).
#[test]
fn refusal_reason_nli_variants_serialize_to_snake_case() {
assert_eq!(
serde_json::to_string(&RefusalReason::NliVerificationFailed).unwrap(),
"\"nli_verification_failed\""
);
assert_eq!(
serde_json::to_string(&RefusalReason::NliModelUnavailable).unwrap(),
"\"nli_model_unavailable\""
);
}
/// p9-fb-41 PR-9c-1: `Answer.verification` is `Option<...>` with
/// `skip_serializing_if = None`. A `verification: None` answer
/// must NOT emit a `"verification"` key on the wire — the field
/// is additive and pre-v0.18 readers see no new key.
#[test]
fn answer_omits_verification_field_when_none() {
let ans = Answer {
answer: "x".into(),
citations: vec![],
grounded: true,
refusal_reason: None,
model: ModelRef {
id: "m".into(),
provider: "p".into(),
dimensions: None,
},
embedding: None,
prompt_template_version: PromptTemplateVersion("rag-v2".into()),
retrieval: AnswerRetrievalSummary {
trace_id: TraceId("t".into()),
mode: crate::SearchMode::Lexical,
k: 1,
score_gate: 0.0,
top_score: 0.0,
chunks_returned: 0,
chunks_used: 0,
},
usage: TokenUsage {
prompt_tokens: 0,
completion_tokens: 0,
latency_ms: 0,
},
created_at: datetime!(2026-05-09 12:00:00 UTC),
conversation_id: None,
turn_index: None,
hops: None,
verification: None,
};
let v = serde_json::to_value(&ans).unwrap();
assert!(
v.get("verification").is_none(),
"verification: None must be omitted from wire output, got: {v}"
);
}
#[test]
fn verification_summary_serializes_all_three_required_fields() {
let vs = VerificationSummary {
nli_score: 0.87,
nli_threshold: 0.5,
nli_passed: true,
};
let v = serde_json::to_value(vs).unwrap();
assert!((v["nli_score"].as_f64().unwrap() - 0.87).abs() < 1e-5);
assert!((v["nli_threshold"].as_f64().unwrap() - 0.5).abs() < 1e-5);
assert_eq!(v["nli_passed"], true);
}
#[test]
fn answer_citation_serializes_indexed_at_and_stale() {
let ac = AnswerCitation {

View File

@@ -226,28 +226,25 @@ fn parse_hms_ms(s: &str) -> Result<u64> {
let m: u64 = parts[1]
.parse()
.map_err(|_| anyhow::anyhow!("bad minutes in {:?} (input {s:?})", parts[1]))?;
let (sec, ms) = match parts[2].split_once('.') {
Some((s_part, ms_part)) => {
let sec: u64 = s_part
.parse()
.map_err(|_| anyhow::anyhow!("bad seconds in {s_part:?} (input {s:?})"))?;
// Pad/truncate to exactly 3 digits.
let mut ms_str = ms_part.to_owned();
while ms_str.len() < 3 {
ms_str.push('0');
}
ms_str.truncate(3);
let ms: u64 = ms_str
.parse()
.map_err(|_| anyhow::anyhow!("bad milliseconds in {ms_part:?} (input {s:?})"))?;
(sec, ms)
}
None => {
let sec: u64 = parts[2]
.parse()
.map_err(|_| anyhow::anyhow!("bad seconds in {:?} (input {s:?})", parts[2]))?;
(sec, 0)
let (sec, ms) = if let Some((s_part, ms_part)) = parts[2].split_once('.') {
let sec: u64 = s_part
.parse()
.map_err(|_| anyhow::anyhow!("bad seconds in {s_part:?} (input {s:?})"))?;
// Pad/truncate to exactly 3 digits.
let mut ms_str = ms_part.to_owned();
while ms_str.len() < 3 {
ms_str.push('0');
}
ms_str.truncate(3);
let ms: u64 = ms_str
.parse()
.map_err(|_| anyhow::anyhow!("bad milliseconds in {ms_part:?} (input {s:?})"))?;
(sec, ms)
} else {
let sec: u64 = parts[2]
.parse()
.map_err(|_| anyhow::anyhow!("bad seconds in {:?} (input {s:?})", parts[2]))?;
(sec, 0)
};
Ok(h * 3_600_000 + m * 60_000 + sec * 1000 + ms)
}

View File

@@ -56,8 +56,8 @@ pub use search::{
TraceCandidate, TraceFusionInput, TraceTiming,
};
pub use answer::{
Answer, AnswerCitation, AnswerRetrievalSummary, ModelRef, RefusalReason, TokenUsage,
TraceId, Turn,
Answer, AnswerCitation, AnswerRetrievalSummary, HopKind, HopRecord, ModelRef,
RefusalReason, TokenUsage, TraceId, Turn, VerificationSummary,
};
pub use ingest::{IngestItem, IngestItemKind, IngestReport, SkipExamples};
pub use jobs::{JobFilter, JobId, JobKind, JobRow, JobStatus};

View File

@@ -33,7 +33,7 @@ pub struct Metadata {
pub git_commit: Option<String>,
/// p10-1A-1: programming language identifier (lowercase canonical). null
/// for markdown / pdf / image. Set by `kebab_parse_code::lang::code_lang_for_path`.
/// for markdown / pdf / image. Set by the local-filesystem source connector during ingest.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub code_lang: Option<String>,
}

View File

@@ -471,7 +471,7 @@ mod tests {
doc_path: WorkspacePath("a.md".into()),
heading_path: vec![],
section_label: None,
snippet: "".into(),
snippet: String::new(),
citation: Citation::Line {
path: WorkspacePath("a.md".into()),
start: 1,
@@ -502,7 +502,7 @@ mod tests {
doc_path: WorkspacePath("a.rs".into()),
heading_path: vec![],
section_label: None,
snippet: "".into(),
snippet: String::new(),
citation: Citation::Code {
path: WorkspacePath("a.rs".into()),
line_start: 1,

View File

@@ -20,3 +20,6 @@ anyhow = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
serde_json = { workspace = true }
[lints]
workspace = true

View File

@@ -158,7 +158,7 @@ impl Embedder for FastembedEmbedder {
let guard = self
.inner
.lock()
.unwrap_or_else(|p| p.into_inner());
.unwrap_or_else(std::sync::PoisonError::into_inner);
let batch: Vec<Vec<f32>> = guard
.embed(chunk_vec, Some(self.batch_size))
.context("fastembed: embed")?;

View File

@@ -28,3 +28,6 @@ mock = []
[dev-dependencies]
proptest = { workspace = true }
[lints]
workspace = true

View File

@@ -59,7 +59,7 @@ pub fn assert_vector_shape(vecs: &[Vec<f32>], expected_dims: usize) {
/// Panics on mismatch (test-only helper — callers are tests).
pub fn assert_unit_norm(vecs: &[Vec<f32>], tolerance: f32) {
for (i, v) in vecs.iter().enumerate() {
let norm_sq: f64 = v.iter().map(|&x| (x as f64) * (x as f64)).sum();
let norm_sq: f64 = v.iter().map(|&x| f64::from(x) * f64::from(x)).sum();
let norm = norm_sq.sqrt() as f32;
assert!(
(norm - 1.0).abs() <= tolerance,

View File

@@ -132,10 +132,10 @@ impl Embedder for MockEmbedder {
.collect();
// L2-normalize. Skip the rare all-zero case to avoid 0/0 = NaN.
let norm_sq: f64 = v.iter().map(|&x| (x as f64) * (x as f64)).sum();
let norm_sq: f64 = v.iter().map(|&x| f64::from(x) * f64::from(x)).sum();
if norm_sq > 0.0 {
let inv = (1.0 / norm_sq.sqrt()) as f32;
for x in v.iter_mut() {
for x in &mut v {
*x *= inv;
}
}

View File

@@ -28,3 +28,6 @@ uuid = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
rusqlite = { workspace = true }
[lints]
workspace = true

View File

@@ -260,8 +260,8 @@ pub fn render_report_md(report: &CompareReport) -> String {
"| {} | {} | {} | {} | {} |",
c.query_id,
comparison_kind_label(c.kind),
c.a_hit_rank.map(|r| r.to_string()).unwrap_or_else(|| "".into()),
c.b_hit_rank.map(|r| r.to_string()).unwrap_or_else(|| "".into()),
c.a_hit_rank.map_or_else(|| "".into(), |r| r.to_string()),
c.b_hit_rank.map_or_else(|| "".into(), |r| r.to_string()),
c.note.as_deref().unwrap_or(""),
);
}
@@ -308,7 +308,7 @@ fn extract_chunker_version(snapshot_json: &str) -> Option<String> {
let v: serde_json::Value = serde_json::from_str(snapshot_json).ok()?;
v.get("chunker_version")
.and_then(|x| x.as_str())
.map(|s| s.to_owned())
.map(std::borrow::ToOwned::to_owned)
}
fn parse_results(
@@ -402,8 +402,7 @@ fn classify(
// so refusal-flow queries (no expected_*) don't appear as
// regressions.
let has_expected = gq
.map(|g| !g.expected_chunk_ids.is_empty() || !g.expected_doc_ids.is_empty())
.unwrap_or(false);
.is_some_and(|g| !g.expected_chunk_ids.is_empty() || !g.expected_doc_ids.is_empty());
if has_expected {
(ComparisonKind::Regression, Some("hit→miss".into()))
} else {
@@ -426,7 +425,7 @@ fn build_deltas(
if a.is_nan() || b.is_nan() {
serde_json::Value::Null
} else {
serde_json::Value::from((b - a) as f64)
serde_json::Value::from(f64::from(b - a))
}
}
let mut hit = serde_json::Map::new();

View File

@@ -270,7 +270,21 @@ pub(crate) fn aggregate_from_rows(
// recall@k_doc (doc-level, requires non-empty expected_doc_ids
// and `>0` is the "should retrieve" condition; refusal queries
// (`expected_doc_ids = []`) are excluded by spec).
if !gq.expected_doc_ids.is_empty() {
if gq.expected_doc_ids.is_empty() {
// refusal_correctness: golden marks "should refuse" via empty
// expected_doc_ids. We can only judge this on RAG runs — a
// lexical-only run produces no Answer, so "refusal" is
// undefined. Excluding such queries from the denominator
// (rather than counting them as failures) keeps the metric
// honest: a search-only run reports refusal_correctness as
// NaN/null, not 0.0.
if let Some(ans) = &qr.answer {
refusal_denom += 1;
if !ans.grounded {
refusal_num += 1;
}
}
} else {
let expected_docs: HashSet<&DocumentId> = gq.expected_doc_ids.iter().collect();
for k in TOP_K_VARIANTS {
let entry = recall_at_k_doc.get_mut(k).expect("init");
@@ -285,20 +299,6 @@ pub(crate) fn aggregate_from_rows(
let frac = covered as f64 / expected_docs.len() as f64;
entry.0 += frac;
}
} else {
// refusal_correctness: golden marks "should refuse" via empty
// expected_doc_ids. We can only judge this on RAG runs — a
// lexical-only run produces no Answer, so "refusal" is
// undefined. Excluding such queries from the denominator
// (rather than counting them as failures) keeps the metric
// honest: a search-only run reports refusal_correctness as
// NaN/null, not 0.0.
if let Some(ans) = &qr.answer {
refusal_denom += 1;
if !ans.grounded {
refusal_num += 1;
}
}
}
// groundedness + citation_coverage (only meaningful with RAG
@@ -532,6 +532,8 @@ mod tests {
created_at: OffsetDateTime::UNIX_EPOCH,
conversation_id: None,
turn_index: None,
hops: None,
verification: None,
}
}

View File

@@ -179,6 +179,10 @@ fn execute_query(app: &App, gq: &GoldenQuery, opts: &EvalRunOpts) -> QueryResult
history: Vec::new(),
conversation_id: None,
turn_index: None,
// p9-fb-41: golden eval baseline runs are single-pass; the
// multi-hop path is opted into per query via a future
// fixture flag (PR-4+) once the runner learns to dispatch.
multi_hop: false,
};
match app.ask(&gq.query, ask_opts) {
Ok(ans) => Some(ans),

View File

@@ -5,7 +5,7 @@
"chunk_id": "chunk000000000000000000000000000000",
"doc_id": "doc00000000000000000000000000000000",
"heading_path": [],
"score": 0.3429983854293823
"score": 0.35202541947364807
},
"has_answer": false,
"hits_count": 1,
@@ -19,7 +19,7 @@
"chunk_id": "chunk000000000000000000000000000002",
"doc_id": "doc00000000000000000000000000000002",
"heading_path": [],
"score": 0.3585492968559265
"score": 0.3414848744869232
},
"has_answer": false,
"hits_count": 1,

View File

@@ -38,7 +38,44 @@ fn loads_minimal_well_formed_yaml() {
assert_eq!(qs[1].difficulty.as_deref(), Some("easy"));
}
// ── 2. duplicate IDs error lists every offender (sorted, deduplicated) ───────
// ── 2. fb-41 multi-hop golden fixture loads + sanity-checks ─────────────────
/// fb-41 baseline + post-merge Δ measurement fixture. The shared
/// loader must accept `fixtures/multi_hop_golden.yaml` and the bucket
/// distribution must stay 5 cross-doc + 5 intra-doc + 5 single-fact
/// negative — curators dropping or re-id'ing a question hit a clear
/// failure mode here before it reaches the runner.
#[test]
fn loads_multi_hop_golden_fixture() {
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("..")
.join("..")
.join("fixtures")
.join("multi_hop_golden.yaml");
let qs = load_golden_set(&path).expect("multi_hop_golden.yaml must parse");
assert_eq!(qs.len(), 15, "fb-41 fixture must have 15 questions");
let cross_doc = qs.iter().filter(|q| q.id.starts_with("mh-c-")).count();
let intra_doc = qs.iter().filter(|q| q.id.starts_with("mh-i-")).count();
let single = qs.iter().filter(|q| q.id.starts_with("mh-s-")).count();
assert_eq!(cross_doc, 5, "expected 5 mh-c-* (cross-doc) questions");
assert_eq!(intra_doc, 5, "expected 5 mh-i-* (intra-doc) questions");
assert_eq!(single, 5, "expected 5 mh-s-* (single-fact negative) questions");
// Every question carries at least one `must_contain` so the
// rule-based answer-correctness metric (P5-2) has a signal even
// before `expected_chunk_ids` are filled in.
for q in &qs {
assert!(
!q.must_contain.is_empty(),
"{}: must_contain is empty — baseline measurement needs a signal",
q.id
);
}
}
// ── 3. duplicate IDs error lists every offender (sorted, deduplicated) ───────
#[test]
fn rejects_duplicate_ids() {

View File

@@ -143,7 +143,7 @@ fn env_guard() -> std::sync::MutexGuard<'static, ()> {
static M: OnceLock<Mutex<()>> = OnceLock::new();
M.get_or_init(|| Mutex::new(()))
.lock()
.unwrap_or_else(|e| e.into_inner())
.unwrap_or_else(std::sync::PoisonError::into_inner)
}
#[test]

View File

@@ -147,7 +147,7 @@ fn lexical_opts() -> EvalRunOpts {
/// guard must outlive the call so concurrent tests don't reset the
/// var mid-run.
fn run_with_golden<F: FnOnce() -> R, R>(yaml: &Path, f: F) -> R {
let _g = GOLDEN_ENV_LOCK.lock().unwrap_or_else(|p| p.into_inner());
let _g = GOLDEN_ENV_LOCK.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
// SAFETY: `KEBAB_EVAL_GOLDEN` is a benign env var; the GOLDEN_ENV_LOCK
// serializes mutations so concurrent tests don't race.
unsafe {

View File

@@ -34,3 +34,6 @@ anyhow = { workspace = true }
# `tokio::*` symbols, so the public/runtime API stays sync.
wiremock = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt"] }
[lints]
workspace = true

View File

@@ -48,10 +48,17 @@ use serde::{Deserialize, Serialize};
use crate::error::LlmError;
/// Hard ceiling on a single HTTP exchange. Cold-loading a 14B model on
/// first call can take ~30s; 5 minutes is generous without being
/// open-ended.
const REQUEST_TIMEOUT: Duration = Duration::from_secs(300);
// v0.17.0 post-dogfood: the per-request ceiling now lives in
// `kebab_config::LlmCfg::request_timeout_secs` (default 300s) so users
// running larger models on CPU-only hosts can extend it without a
// rebuild. Cold-loading an 8B+ model on first call routinely takes
// 60-90 s plus multi-minute inference; 300s was the legacy hard
// ceiling and remains the default for back-compat.
//
// Edge case: `request_timeout_secs = 0` becomes
// `Duration::from_secs(0)` which is reqwest's "fail immediately", NOT
// "disable". The field doc explains the workaround (use u64::MAX or a
// large finite value).
/// `reqwest::blocking` adapter implementing [`LanguageModel`] over Ollama's
/// local HTTP API. Construction is cheap and offline; the first network
@@ -79,7 +86,7 @@ impl OllamaLanguageModel {
pub fn new(config: &kebab_config::Config) -> anyhow::Result<Self> {
let llm = &config.models.llm;
let client = reqwest::blocking::Client::builder()
.timeout(REQUEST_TIMEOUT)
.timeout(Duration::from_secs(llm.request_timeout_secs))
.build()?;
Ok(Self {
client,
@@ -262,9 +269,11 @@ struct OllamaLine {
///
/// Timeout invariant: the iterator has no inherent stop condition for an
/// indefinitely-stalled server — only the underlying
/// `reqwest::blocking::Client`'s read timeout (`REQUEST_TIMEOUT`, 300s)
/// breaks the hang. Callers needing tighter cancellation should adjust
/// the client timeout in [`OllamaLanguageModel::new`].
/// `reqwest::blocking::Client`'s read timeout (configured via
/// `kebab_config::LlmCfg::request_timeout_secs`, default 300 s) breaks
/// the hang. Callers needing tighter / looser bounds should set
/// `[models.llm] request_timeout_secs = N` (or
/// `KEBAB_MODELS_LLM_REQUEST_TIMEOUT_SECS=N`) before building.
struct OllamaStream {
reader: BufReader<reqwest::blocking::Response>,
line_buf: Vec<u8>,
@@ -391,9 +400,9 @@ impl Iterator for OllamaStream {
// u32 saturation: even ~4G tokens is implausible for a
// single chat turn; we still saturate rather than
// panic on the unlikely case.
prompt_tokens: prompt_tokens.min(u32::MAX as u64) as u32,
completion_tokens: completion_tokens.min(u32::MAX as u64) as u32,
latency_ms: (total_duration_ns / 1_000_000).min(u32::MAX as u64) as u32,
prompt_tokens: prompt_tokens.min(u64::from(u32::MAX)) as u32,
completion_tokens: completion_tokens.min(u64::from(u32::MAX)) as u32,
latency_ms: (total_duration_ns / 1_000_000).min(u64::from(u32::MAX)) as u32,
};
return Some(Ok(TokenChunk::Done {
finish_reason,

View File

@@ -19,3 +19,6 @@ mock = []
[dev-dependencies]
proptest = { workspace = true }
[lints]
workspace = true

View File

@@ -27,3 +27,6 @@ kebab-core = { path = "../kebab-core" }
[dev-dependencies]
tempfile = { workspace = true }
[lints]
workspace = true

View File

@@ -49,7 +49,7 @@ pub fn build_tools_vec() -> Vec<Tool> {
),
Tool::new(
"ask",
"RAG question answering over the knowledge base. Returns answer.v1 JSON. Pass session_id for multi-turn context.",
"RAG question answering over the knowledge base. Returns answer.v1 JSON. Pass session_id for multi-turn context. Set multi_hop=true for compound / cross-doc questions (decompose → retrieve → synthesize; 2-5× LLM cost; per-hop trace on Answer.hops).",
schema_for_type::<tools::ask::AskInput>(),
),
Tool::new(

View File

@@ -20,6 +20,15 @@ pub struct AskInput {
pub session_id: Option<String>,
/// Optional retrieval mode override ("lexical" / "vector" / "hybrid"). Default "hybrid".
pub mode: Option<String>,
/// p9-fb-41: opt the ask into the multi-hop pipeline. Default `false`.
/// When `true`, the query is decomposed into sub-questions, each
/// retrieved independently, then synthesized over the merged
/// chunk pool. Cost trade-off: 25× LLM calls vs. single-pass.
/// Use for compound questions / cross-doc reasoning / prereq
/// chains; keep `false` for simple fact lookups. The full
/// per-hop trace (`decompose` / `decide` / `synthesize`) is
/// exposed on `Answer.hops`.
pub multi_hop: Option<bool>,
}
pub fn handle(state: &KebabAppState, input: AskInput) -> CallToolResult {
@@ -38,6 +47,7 @@ pub fn handle(state: &KebabAppState, input: AskInput) -> CallToolResult {
history: Vec::new(),
conversation_id: None,
turn_index: None,
multi_hop: input.multi_hop.unwrap_or(false),
};
let cfg_clone = (*state.config).clone();
let result = match input.session_id {

View File

@@ -55,6 +55,7 @@ async fn ask_tool_returns_answer_v1_with_refusal_on_empty_kb() {
// Test env uses provider="none" — Hybrid would hard-error on embedding.
// Pass Lexical explicitly so the test stays functional.
mode: Some("lexical".to_string()),
multi_hop: None,
},
)
})
@@ -64,8 +65,7 @@ async fn ask_tool_returns_answer_v1_with_refusal_on_empty_kb() {
// Empty KB → refusal (grounded:false) is normal — NOT isError.
assert!(
!result.is_error.unwrap_or(false),
"expected isError=false on refusal, got {:?}",
result
"expected isError=false on refusal, got {result:?}"
);
let content = result
@@ -85,7 +85,7 @@ async fn ask_tool_returns_answer_v1_with_refusal_on_empty_kb() {
"response should carry schema_version=answer.v1"
);
assert_eq!(
v.get("grounded").and_then(|b| b.as_bool()),
v.get("grounded").and_then(serde_json::Value::as_bool),
Some(false),
"empty KB should produce grounded=false"
);

View File

@@ -0,0 +1,231 @@
//! Pin the MCP `ask` tool's `multi_hop` argument dispatch contract.
//!
//! v0.18 dogfood fix (PR-7) introduced a pre-decompose score-gate probe
//! in `RagPipeline::ask_multi_hop`: empty KB / sub-gate probe -> the
//! single-pass NoChunks refusal envelope (`answer.v1`), not `error.v1`.
//! The two surfaces' divergence is therefore observed *only when the probe
//! passes* — at that point, single-pass returns retrieval + LLM call, and
//! multi-hop calls decompose first (LLM unreachable -> `error.v1`).
//!
//! These two tests pin:
//! 1. `ask_tool_routes_multi_hop_true_to_decompose_first` — probe-passing
//! fixture, multi_hop=true → decompose (LLM error), single_pass → retrieval
//! NoChunks. Wire shapes diverge: `error.v1` vs `answer.v1`.
//! 2. `ask_tool_multi_hop_short_circuits_when_probe_empty` — empty KB,
//! multi_hop=true → probe-empty short-circuit, NoChunks refusal byte-
//! identical to single-pass. PR-7 의 intent 가 MCP layer 에 pin.
//!
//! A live-Ollama end-to-end multi-hop pin lands in a follow-up
//! `#[ignore]` test (same pattern as `wire_ask_stale.rs`).
use kebab_config::Config;
use kebab_core::SourceScope;
use kebab_mcp::{KebabAppState, KebabHandler};
use rmcp::model::RawContent;
fn minimal_config(data_dir: &std::path::Path, workspace_root: &std::path::Path) -> Config {
let mut cfg = Config::defaults();
cfg.storage.data_dir = data_dir.to_string_lossy().into_owned();
cfg.storage.model_dir = data_dir.join("models").to_string_lossy().into_owned();
cfg.workspace.root = workspace_root.to_string_lossy().into_owned();
cfg.workspace.exclude.clear();
cfg.models.embedding.provider = "none".to_string();
cfg.models.embedding.dimensions = 0;
// Force the LLM endpoint to a known-unreachable port so this test
// is robust against whether a real Ollama happens to be running
// on 127.0.0.1:11434 (the developer's box; CI; etc.). The
// `request_timeout_secs = 5` gives slow CI / Docker network stacks
// enough headroom that *some* error fires deterministically — the
// dispatch contract below only cares that `is_error` flipped, not
// which specific error code surfaced.
cfg.models.llm.endpoint = "http://127.0.0.1:1".to_string();
cfg.models.llm.request_timeout_secs = 5;
// Bypass the second probe gate (`top_score < score_gate`) so that the
// probe-pass path in `RagPipeline::ask_multi_hop` (PR-7 v0.18 dogfood
// fix) is reachable from a tiny lexical fixture whose FTS5 fusion
// score may sit below the production default (0.30). The probe's
// first gate (`probe_hits.is_empty()`) is unaffected — the empty-KB
// short-circuit test below still exercises it. Production default
// 0.30 remains untouched (test config isolation only).
cfg.rag.score_gate = 0.0;
cfg
}
/// The dispatch contract (post-PR-7 probe-first): with a probe-passing
/// fixture, single-pass `ask` retrieves first and returns a NoChunks
/// refusal Answer for an unrelated query (`grounded=false`,
/// `isError=false`). Multi-hop's probe passes on the same fixture →
/// decompose runs → unreachable LLM yields `error.v1` with
/// `code=model_unreachable` (`isError=true`). The divergence confirms
/// the `multi_hop` arg actually rerouted the dispatch *after* the
/// probe gate.
#[tokio::test]
async fn ask_tool_routes_multi_hop_true_to_decompose_first() {
let dir = tempfile::tempdir().unwrap();
let data_dir = dir.path().join("data");
let workspace_root = dir.path().join("notes");
std::fs::create_dir_all(&data_dir).unwrap();
std::fs::create_dir_all(&workspace_root).unwrap();
// Lexical-friendly fixture so the multi-hop probe (PR-7 v0.18 dogfood
// fix) returns at least one hit and we exercise the post-probe
// decompose path. `build_match_string` rewrites the query
// `"compound about X and Y"` into
// `text : (("compound about X and Y") OR ("compound" "about" "and"))`
// — the token_and branch is FTS5 implicit-AND, so the fixture body
// MUST keep all three tokens (`compound`, `about`, `and`). Do not
// collapse to a single-token body or the probe short-circuits to
// NoChunks and the dispatch divergence below disappears.
let fixture = workspace_root.join("note.md");
std::fs::write(
&fixture,
"# Compound topic\n\nThis note is about a compound containing X and Y in detail.\n",
)
.unwrap();
let cfg = minimal_config(&data_dir, &workspace_root);
let scope = SourceScope {
root: workspace_root.clone(),
include: vec![],
exclude: vec![],
};
let _ = kebab_app::ingest_with_config(cfg.clone(), scope, false).unwrap();
let state = KebabAppState::new(cfg, None);
let handler = KebabHandler::new(state);
// Multi-hop branch — decompose runs first, hits the unreachable
// endpoint, MCP wraps as error.v1.
let state_mh = handler.state().clone();
let mh = tokio::task::spawn_blocking(move || {
kebab_mcp::tools::ask::handle(
&state_mh,
kebab_mcp::tools::ask::AskInput {
query: "compound about X and Y".to_string(),
session_id: None,
mode: Some("lexical".to_string()),
multi_hop: Some(true),
},
)
})
.await
.unwrap();
assert!(
mh.is_error.unwrap_or(false),
"multi_hop=true must reach the LLM (decompose first) — got {mh:?}"
);
let mh_text = match &mh.content.first().unwrap().raw {
RawContent::Text(t) => t.text.clone(),
other => panic!("expected text, got {other:?}"),
};
let mh_v: serde_json::Value = serde_json::from_str(&mh_text).unwrap();
assert_eq!(mh_v["schema_version"], "error.v1");
// The dispatch contract is "multi-hop's probe passed, then decompose
// tried to talk to the LLM and failed" — i.e. `is_error` fires
// because, *after* the PR-7 probe gate, decompose attempted an LLM
// call against the unreachable endpoint. Which *specific* error code
// lands (`model_unreachable` on fast ECONNREFUSED hosts, `timeout`
// on slow connect-timeout stacks, etc.) is implementation detail of
// the host TCP/HTTP path; pinning it here would just produce flakes
// on slow CI.
// Single-pass branch — empty KB short-circuits at retrieve, no LLM
// call happens, refusal Answer comes back as isError=false.
let state_sp = handler.state().clone();
let sp = tokio::task::spawn_blocking(move || {
kebab_mcp::tools::ask::handle(
&state_sp,
kebab_mcp::tools::ask::AskInput {
query: "anything".to_string(),
session_id: None,
mode: Some("lexical".to_string()),
multi_hop: Some(false),
},
)
})
.await
.unwrap();
assert!(
!sp.is_error.unwrap_or(false),
"single-pass empty-KB refusal must NOT be isError — got {sp:?}"
);
let sp_text = match &sp.content.first().unwrap().raw {
RawContent::Text(t) => t.text.clone(),
other => panic!("expected text, got {other:?}"),
};
let sp_v: serde_json::Value = serde_json::from_str(&sp_text).unwrap();
assert_eq!(sp_v["schema_version"], "answer.v1");
assert_eq!(sp_v["grounded"], false);
}
/// PR-7 의 probe-empty short-circuit 이 MCP-layer 의 wire shape 로 pin.
/// 빈 KB + multi_hop=true → `RagPipeline::ask_multi_hop` 의 첫 probe
/// gate (`probe_hits.is_empty()`) 에 막혀 `refuse_no_chunks` 가 single-pass
/// 와 byte-identical 한 `answer.v1` refusal envelope 을 반환한다.
/// kebab-rag::multi_hop_empty_probe_pool_refuses_before_any_llm_call 가
/// RAG-layer 만 pin — MCP-layer 의 wire shape 는 본 test 만이 안전망.
#[tokio::test]
async fn ask_tool_multi_hop_short_circuits_when_probe_empty() {
let dir = tempfile::tempdir().unwrap();
let data_dir = dir.path().join("data");
let workspace_root = dir.path().join("notes");
std::fs::create_dir_all(&data_dir).unwrap();
std::fs::create_dir_all(&workspace_root).unwrap();
let cfg = minimal_config(&data_dir, &workspace_root);
let scope = SourceScope {
root: workspace_root.clone(),
include: vec![],
exclude: vec![],
};
let _ = kebab_app::ingest_with_config(cfg.clone(), scope, false).unwrap();
let state = KebabAppState::new(cfg.clone(), None);
let handler = KebabHandler::new(state);
let state_mh = handler.state().clone();
let mh = tokio::task::spawn_blocking(move || {
kebab_mcp::tools::ask::handle(
&state_mh,
kebab_mcp::tools::ask::AskInput {
query: "compound about X and Y".to_string(),
session_id: None,
mode: Some("lexical".to_string()),
multi_hop: Some(true),
},
)
})
.await
.unwrap();
assert_eq!(
mh.is_error,
Some(false),
"probe-empty short-circuit must yield refusal envelope, not error.v1 — got {mh:?}"
);
let mh_text = match &mh.content.first().unwrap().raw {
RawContent::Text(t) => t.text.clone(),
other => panic!("expected text content, got {other:?}"),
};
let body: serde_json::Value = serde_json::from_str(&mh_text).unwrap();
assert_eq!(body["schema_version"], "answer.v1");
assert_eq!(body["refusal_reason"], "no_chunks");
}
/// AskInput's JSON-schema (rendered for tools/list) advertises the
/// new `multi_hop` field. Pins agent / MCP host capability discovery
/// against accidental schema-rename or omission.
#[test]
fn ask_input_schema_advertises_multi_hop_field() {
let schema = schemars::schema_for!(kebab_mcp::tools::ask::AskInput);
let v = serde_json::to_value(&schema).unwrap();
let props = v
.get("properties")
.and_then(|p| p.as_object())
.expect("AskInput schema must declare properties");
assert!(
props.contains_key("multi_hop"),
"AskInput.multi_hop must surface in the JsonSchema — got keys: {:?}",
props.keys().collect::<Vec<_>>()
);
}

View File

@@ -44,7 +44,7 @@ async fn doctor_tool_returns_doctor_v1_json() {
// `ok` boolean must be present (value may be false in CI where Ollama
// is not reachable — that's expected and acceptable).
assert!(
v.get("ok").and_then(|b| b.as_bool()).is_some(),
v.get("ok").and_then(serde_json::Value::as_bool).is_some(),
"`ok` field missing in doctor.v1 response: {v}"
);
}

View File

@@ -98,8 +98,7 @@ async fn fetch_tool_chunk_returns_fetch_result_v1() {
assert!(
!result.is_error.unwrap_or(false),
"expected isError=false, got {:?}",
result
"expected isError=false, got {result:?}"
);
let content = result
@@ -123,7 +122,7 @@ async fn fetch_tool_chunk_returns_fetch_result_v1() {
"kind must be 'chunk'"
);
assert!(
v.get("chunk").is_some_and(|c| c.is_object()),
v.get("chunk").is_some_and(serde_json::Value::is_object),
"chunk payload must be populated for kind=chunk"
);
}

View File

@@ -49,7 +49,7 @@ async fn ingest_file_tool_returns_ingest_report_v1() {
v.get("schema_version").and_then(|s| s.as_str()),
Some("ingest_report.v1")
);
assert_eq!(v.get("new").and_then(|n| n.as_u64()), Some(1));
assert_eq!(v.get("new").and_then(serde_json::Value::as_u64), Some(1));
}
#[tokio::test]
@@ -91,7 +91,7 @@ async fn ingest_file_tool_idempotent_on_second_call() {
other => panic!("expected text, got {other:?}"),
};
let v1: serde_json::Value = serde_json::from_str(text1).unwrap();
assert_eq!(v1.get("new").and_then(|n| n.as_u64()), Some(1));
assert_eq!(v1.get("new").and_then(serde_json::Value::as_u64), Some(1));
// Second call — same content, expect unchanged=1.
let r2 = tokio::task::spawn_blocking({
@@ -112,6 +112,6 @@ async fn ingest_file_tool_idempotent_on_second_call() {
other => panic!("expected text, got {other:?}"),
};
let v2: serde_json::Value = serde_json::from_str(text2).unwrap();
assert_eq!(v2.get("new").and_then(|n| n.as_u64()), Some(0), "{v2:?}");
assert_eq!(v2.get("unchanged").and_then(|n| n.as_u64()), Some(1), "{v2:?}");
assert_eq!(v2.get("new").and_then(serde_json::Value::as_u64), Some(0), "{v2:?}");
assert_eq!(v2.get("unchanged").and_then(serde_json::Value::as_u64), Some(1), "{v2:?}");
}

View File

@@ -52,7 +52,7 @@ async fn ingest_stdin_tool_returns_ingest_report_v1() {
v.get("schema_version").and_then(|s| s.as_str()),
Some("ingest_report.v1")
);
assert_eq!(v.get("new").and_then(|n| n.as_u64()), Some(1));
assert_eq!(v.get("new").and_then(serde_json::Value::as_u64), Some(1));
}
#[tokio::test]

View File

@@ -49,8 +49,7 @@ async fn schema_tool_returns_schema_v1_json() {
assert!(
!result.is_error.unwrap_or(false),
"expected isError=false on healthy schema, got {:?}",
result
"expected isError=false on healthy schema, got {result:?}"
);
let content = result.content.first().expect("expected at least one content item");
@@ -68,7 +67,7 @@ async fn schema_tool_returns_schema_v1_json() {
"unexpected schema_version in: {v}"
);
assert_eq!(
v.get("capabilities").and_then(|c| c.get("mcp_server")).and_then(|b| b.as_bool()),
v.get("capabilities").and_then(|c| c.get("mcp_server")).and_then(serde_json::Value::as_bool),
Some(true),
"mcp_server capability flag should be true after fb-30",
);

View File

@@ -71,8 +71,7 @@ async fn search_tool_returns_search_response_v1() {
assert!(
!result.is_error.unwrap_or(false),
"expected isError=false, got {:?}",
result
"expected isError=false, got {result:?}"
);
let content = result
@@ -108,7 +107,7 @@ async fn search_tool_returns_search_response_v1() {
);
// truncated must be present (bool); next_cursor may be null on last page.
assert!(
v.get("truncated").and_then(|t| t.as_bool()).is_some(),
v.get("truncated").and_then(serde_json::Value::as_bool).is_some(),
"envelope should carry truncated:bool"
);
assert!(
@@ -172,8 +171,7 @@ async fn search_with_doc_id_filter_returns_only_target() {
);
assert!(
!unfiltered.is_error.unwrap_or(false),
"unfiltered search failed: {:?}",
unfiltered
"unfiltered search failed: {unfiltered:?}"
);
let unfiltered_text = match &unfiltered.content.first().unwrap().raw {
RawContent::Text(t) => t.text.clone(),
@@ -211,8 +209,7 @@ async fn search_with_doc_id_filter_returns_only_target() {
);
assert!(
!filtered.is_error.unwrap_or(false),
"filtered search failed: {:?}",
filtered
"filtered search failed: {filtered:?}"
);
let filtered_text = match &filtered.content.first().unwrap().raw {
RawContent::Text(t) => t.text.clone(),

View File

@@ -0,0 +1,33 @@
[package]
name = "kebab-nli"
version = { workspace = true }
edition = { workspace = true }
rust-version = { workspace = true }
license = { workspace = true }
repository = { workspace = true }
description = "fb-41: NLI-based post-synthesis verification (XNLI mDeBERTa-v3). PR-9a = trait + scaffolding; ONNX inference lands in PR-9b."
[dependencies]
# PR-9b: ONNX inference path activated. ort / tokenizers / hf-hub / ndarray
# all source from `[workspace.dependencies]` so the workspace pins a single
# version + feature set for the whole NLI + embed stack.
kebab-config = { path = "../kebab-config" }
anyhow = { workspace = true }
serde = { workspace = true }
# ort: extend the workspace pin with `download-binaries` so kebab-nli
# can link the ONNX runtime when fastembed is NOT in the build graph
# (e.g. `cargo test -p kebab-nli` alone, where the per-crate feature
# union excludes kebab-embed-local + fastembed). In workspace-wide
# builds the feature gets union'd with fastembed's identical opt-in
# so no extra runtime gets pulled.
ort = { workspace = true, features = ["download-binaries"] }
tokenizers = { workspace = true }
hf-hub = { workspace = true }
ndarray = { workspace = true }
tracing = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
[lints]
workspace = true

129
crates/kebab-nli/src/lib.rs Normal file
View File

@@ -0,0 +1,129 @@
//! `kebab-nli` — NLI-based post-synthesis verification for multi-hop RAG.
//!
//! fb-41 introduces a mDeBERTa-v3 XNLI verifier that runs on
//! `(packed_chunks, generated_answer)` after synthesize. If
//! `NliScores::faithfulness()` < threshold the rag crate refuses the answer
//! with `NliVerificationFailed`. PR-9a (this file) is the trait surface +
//! scaffolding only — `OnnxNliVerifier::score` returns a stub error until
//! PR-9b adds the real ONNX inference path.
use serde::{Deserialize, Serialize};
pub mod onnx;
pub use onnx::OnnxNliVerifier;
/// Three-channel XNLI output. Channel order matches the standard XNLI
/// `id2label` mapping `[entailment, neutral, contradiction]` shipped with
/// the Xenova mDeBERTa-v3 model.
#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct NliScores {
pub entailment: f32,
pub neutral: f32,
pub contradiction: f32,
}
impl NliScores {
/// Faithfulness score = entailment channel. The rag crate compares this
/// against `rag.nli_threshold` to decide whether to refuse.
pub fn faithfulness(&self) -> f32 {
self.entailment
}
/// Wrap raw XNLI logits (`[entailment, neutral, contradiction]`) into
/// a normalised `NliScores`. Applies a numerically-stable softmax3.
pub fn from_xnli_logits(logits: [f32; 3]) -> Self {
let probs = softmax3(logits);
Self {
entailment: probs[0],
neutral: probs[1],
contradiction: probs[2],
}
}
}
/// Abstract NLI verifier. `score` is called with `(premise = packed chunks,
/// hypothesis = generated answer)` — the standard NLI direction (premise
/// entails hypothesis ⇒ answer is grounded in retrieved evidence).
pub trait NliVerifier: Send + Sync {
fn score(&self, premise: &str, hypothesis: &str) -> anyhow::Result<NliScores>;
/// Probe-only tokenize for caller-side budget verification. S3
/// follow-up (2026-05-26) — pipeline 의 char-budget retry loop 가
/// 이 API 로 mDeBERTa-v3 의 `OnlyFirst` dead-end (hypothesis 단독이
/// 512-token cap 초과 시 truncate 불가) 를 회피.
///
/// **Default impl 반환 `Ok(0)`** — 기존 mock implementations
/// (`MockNliVerifier` 등) 가 trait 확장 후에도 backward-compat
/// (compile fail 회피, retry loop immediate 통과). `OnnxNliVerifier`
/// 는 real tokenizer 로 *trait impl 블록 안에서* override 해야 함
/// — inherent method 는 vtable 미등록 → trait dispatch 시 default
/// 호출 → production silent NO-OP.
fn hypothesis_token_count(&self, _hypothesis: &str) -> anyhow::Result<usize> {
Ok(0)
}
}
/// Numerically stable 3-way softmax (subtract max for log-sum-exp safety).
/// Private — call sites should go through `NliScores::from_xnli_logits`.
fn softmax3(logits: [f32; 3]) -> [f32; 3] {
let max = logits[0].max(logits[1]).max(logits[2]);
let e0 = (logits[0] - max).exp();
let e1 = (logits[1] - max).exp();
let e2 = (logits[2] - max).exp();
let sum = e0 + e1 + e2;
[e0 / sum, e1 / sum, e2 / sum]
}
#[cfg(test)]
mod tests {
use super::*;
fn approx_eq(a: f32, b: f32, eps: f32) -> bool {
(a - b).abs() <= eps
}
#[test]
fn softmax3_normalises_to_unit() {
let p = softmax3([1.0, 2.0, 3.0]);
assert!(p.iter().all(|x| *x > 0.0));
assert!(approx_eq(p[0] + p[1] + p[2], 1.0, 1e-6));
// Monotonic: larger logit ⇒ larger probability.
assert!(p[0] < p[1] && p[1] < p[2]);
}
#[test]
fn softmax3_is_invariant_to_constant_shift() {
let a = softmax3([1.0, 2.0, 3.0]);
let b = softmax3([101.0, 102.0, 103.0]);
for i in 0..3 {
assert!(
approx_eq(a[i], b[i], 1e-6),
"channel {i} drifted: a={a:?} b={b:?}"
);
}
}
#[test]
fn nli_scores_from_xnli_logits_orders_correctly() {
// entailment dominates ⇒ entailment is the max probability channel.
let s = NliScores::from_xnli_logits([5.0, 1.0, 0.5]);
assert!(s.entailment > s.neutral);
assert!(s.entailment > s.contradiction);
assert!(approx_eq(
s.entailment + s.neutral + s.contradiction,
1.0,
1e-6
));
}
#[test]
fn faithfulness_returns_entailment_channel() {
let s = NliScores {
entailment: 0.7,
neutral: 0.2,
contradiction: 0.1,
};
assert!(approx_eq(s.faithfulness(), 0.7, f32::EPSILON));
}
}

View File

@@ -0,0 +1,433 @@
//! ONNX-backed `NliVerifier` adapter (mDeBERTa-v3 XNLI).
//!
//! `new` resolves the cache directory from
//! `config.storage.model_dir/nli/<sanitized-model-id>/` (matching the
//! fastembed adapter's pattern of `model_dir/fastembed/`) and stamps it
//! on `self`. The (potentially network-bound) model + tokenizer download
//! is deferred to the first `score` call via `OnceLock<Session>` /
//! `OnceLock<Tokenizer>` — keeping `new` cheap so the rag crate can
//! construct the verifier eagerly during `App` boot without paying for
//! a model load on every CLI invocation.
//!
//! Per design §2.2.2 (Lazy init), §2.2.3 (truncation = `OnlyFirst`,
//! premise truncates, hypothesis preserved). The model id flows from
//! `config.models.nli.model`; `config.models.nli.provider` selects the
//! verifier impl (only `"onnx"` is implemented in v0.18).
use std::path::PathBuf;
use std::sync::OnceLock;
use anyhow::{Context, Result, anyhow};
use kebab_config::expand_path;
use ort::session::Session;
use tokenizers::{
Tokenizer, TruncationDirection, TruncationParams, TruncationStrategy,
};
use crate::{NliScores, NliVerifier};
/// Filename inside the HF repo (NOT a path on disk). The Xenova repo
/// packages the mDeBERTa-v3-base XNLI multilingual checkpoint (the
/// default `config.models.nli.model` — see `kebab-config::NliCfg::defaults`)
/// as ONNX under this path; the tokenizer ships at `tokenizer.json`.
const HF_MODEL_FILE: &str = "onnx/model.onnx";
/// Filename inside the HF repo (NOT a path on disk).
const HF_TOKENIZER_FILE: &str = "tokenizer.json";
/// Subdirectory under `config.storage.model_dir` where the NLI adapter
/// writes / reads ONNX + tokenizer files. Mirrors the fastembed
/// adapter's `model_dir/fastembed/` layout.
const NLI_CACHE_SUBDIR: &str = "nli";
/// XNLI label order in the Xenova mDeBERTa-v3 checkpoint: the model's
/// output logits are `[entailment, neutral, contradiction]`. Pinned as
/// a constant so a future model swap (different label order) is a
/// single-site change.
const LOGITS_LEN: usize = 3;
/// Max input length passed to the tokenizer. mDeBERTa-v3 is trained
/// at 512-token context, matches the Xenova ONNX export's positional
/// embedding shape. `OnlyFirst` strategy makes the premise (which is
/// allowed to be the packed-chunks context) absorb the truncation;
/// the hypothesis (the generated answer) is preserved.
const MAX_TOKENS: usize = 512;
/// ONNX-runtime mDeBERTa-v3 XNLI verifier.
///
/// `session` + `tokenizer` are lazily populated by the first call to
/// `ensure_loaded`. `new` is eager only for cache_dir create_dir_all
/// (cheap) so that the rag crate can construct an instance during
/// `App` boot without paying for the ~280 MB model download.
pub struct OnnxNliVerifier {
model_id: String,
cache_dir: PathBuf,
session: OnceLock<Session>,
tokenizer: OnceLock<Tokenizer>,
}
impl OnnxNliVerifier {
/// Hypothesis-side budget. Pipeline 의
/// `truncate_hypothesis_for_nli_with_budget` retry loop 가 char-truncate
/// 후 token-count 재검증 시 이 값을 cap 으로 사용. = `MAX_TOKENS`
/// (512) - 3 special tokens reserved (CLS, SEP, SEP) - 253 premise
/// room (caller decides). 안전 마진 (S3 follow-up 2026-05-26).
pub const HYPOTHESIS_TOKEN_BUDGET: usize = 256;
/// Construct a verifier from the user's `Config`. Eagerly resolves
/// `cache_dir = config.storage.model_dir/nli/<sanitized-model-id>/`
/// and runs `create_dir_all` so the first `score` call can drop
/// straight into download + load without re-deriving paths.
///
/// Reads `config.models.nli.model` for the HuggingFace model id
/// and `config.models.nli.provider` to select the verifier impl —
/// only `"onnx"` is implemented in v0.18. The defaults live in
/// `kebab-config::NliCfg::defaults` so this path always receives
/// a non-empty model id.
pub fn new(config: &kebab_config::Config) -> Result<Self> {
let provider = config.models.nli.provider.as_str();
if provider != "onnx" {
anyhow::bail!(
"kebab-nli: unsupported provider {provider:?} (only 'onnx' is implemented in v0.18)"
);
}
let model_id = config.models.nli.model.clone();
// Match kebab-embed-local's two-step expansion: data_dir first,
// then model_dir with `{data_dir}` substituted in.
let data_dir = expand_path(&config.storage.data_dir, "");
let model_dir = expand_path(&config.storage.model_dir, &data_dir.to_string_lossy());
let cache_dir = model_dir
.join(NLI_CACHE_SUBDIR)
.join(sanitize_model_id(&model_id));
std::fs::create_dir_all(&cache_dir)
.with_context(|| format!("create kebab-nli cache dir {}", cache_dir.display()))?;
Ok(Self {
model_id,
cache_dir,
session: OnceLock::new(),
tokenizer: OnceLock::new(),
})
}
/// Download (if needed) + load the ONNX session and tokenizer on
/// first call; return cached refs on subsequent calls. Uses two
/// `OnceLock`s rather than one because a single `OnceLock<(_, _)>`
/// would need to construct both atomically — keeping them split
/// lets us short-circuit on the (rare) hit path where only one
/// side is missing.
///
/// `OnceLock::get_or_try_init` is still unstable (rust-lang/rust#109737)
/// so we implement the fallible init by hand: probe `get`, on miss
/// compute the value, then `set` it. The race between two threads is
/// resolved by `OnceLock::set` — the loser gets `Err`, falls through
/// to a second `get`, and reads the winner's value. Each thread that
/// races + loses does pay the cost of one redundant download (rare in
/// practice: rag boot is single-threaded today), but the cache stays
/// consistent.
fn ensure_loaded(&self) -> Result<(&Session, &Tokenizer)> {
if self.session.get().is_none() {
let s = self.load_session()?;
let _ = self.session.set(s); // loser of a race: discard local value
}
if self.tokenizer.get().is_none() {
let t = self.load_tokenizer()?;
let _ = self.tokenizer.set(t);
}
// Both OnceLocks are populated at this point; `expect` is a
// tighter post-condition than `unwrap_or_else` would be.
let session = self.session.get().expect("session populated above");
let tokenizer = self.tokenizer.get().expect("tokenizer populated above");
Ok((session, tokenizer))
}
/// Build an `hf_hub::api::sync::Api` rooted at `self.cache_dir` and
/// fetch `filename` from `self.model_id`. Logs cache hits at INFO
/// so a user reading kebab logs can see which artifact source the
/// pipeline picked.
fn fetch(&self, filename: &str) -> Result<PathBuf> {
// Round-1 review N1 fix: `Api::get` triggers download on miss,
// so we can't use it as a hit probe. `Cache::get` is fs-only —
// returns Some(path) if cached, None otherwise. No network.
let repo = hf_hub::Repo::new(self.model_id.clone(), hf_hub::RepoType::Model);
let cached = hf_hub::Cache::new(self.cache_dir.clone())
.repo(repo.clone())
.get(filename)
.is_some();
if cached {
tracing::info!(
target: "kebab-nli",
model_id = %self.model_id,
file = %filename,
"NLI artifact cache hit"
);
} else {
tracing::info!(
target: "kebab-nli",
model_id = %self.model_id,
file = %filename,
cache_dir = %self.cache_dir.display(),
"downloading NLI artifact"
);
}
let api = hf_hub::api::sync::ApiBuilder::new()
.with_cache_dir(self.cache_dir.clone())
.build()
.with_context(|| {
format!(
"kebab-nli: hf-hub ApiBuilder::build failed (cache_dir={})",
self.cache_dir.display()
)
})?;
api.model(self.model_id.clone())
.get(filename)
.with_context(|| {
format!(
"kebab-nli: hf-hub fetch failed for {filename} (model_id={}, cache_dir={})",
self.model_id,
self.cache_dir.display()
)
})
}
fn load_session(&self) -> Result<Session> {
tracing::info!(
target: "kebab-nli",
model_id = %self.model_id,
"downloading NLI model + tokenizer (first run only)"
);
let model_path = self.fetch(HF_MODEL_FILE)?;
let session = Session::builder()
.with_context(|| "kebab-nli: ort Session::builder failed")?
.commit_from_file(&model_path)
.with_context(|| {
format!(
"kebab-nli: ort Session::commit_from_file({}) failed",
model_path.display()
)
})?;
tracing::info!(
target: "kebab-nli",
model_id = %self.model_id,
model_path = %model_path.display(),
"NLI model ready"
);
Ok(session)
}
fn load_tokenizer(&self) -> Result<Tokenizer> {
let tokenizer_path = self.fetch(HF_TOKENIZER_FILE)?;
let mut tokenizer = Tokenizer::from_file(&tokenizer_path)
.map_err(|e| anyhow!("kebab-nli: Tokenizer::from_file({}) failed: {e}", tokenizer_path.display()))?;
tokenizer
.with_truncation(Some(TruncationParams {
max_length: MAX_TOKENS,
strategy: TruncationStrategy::OnlyFirst,
stride: 0,
direction: TruncationDirection::Right,
}))
.map_err(|e| anyhow!("kebab-nli: Tokenizer::with_truncation failed: {e}"))?;
Ok(tokenizer)
}
}
impl NliVerifier for OnnxNliVerifier {
fn score(&self, premise: &str, hypothesis: &str) -> Result<NliScores> {
// Defense-in-depth: spec §2.3 has the caller skip empty answers,
// but a degenerate empty hypothesis here would tokenize to a
// [CLS][SEP][SEP] triple that yields a near-uniform softmax —
// misleading both faithfulness gate and any future logging.
if hypothesis.trim().is_empty() {
anyhow::bail!("kebab-nli: empty hypothesis");
}
let (session, tokenizer) = self.ensure_loaded()?;
let enc = tokenizer
.encode((premise, hypothesis), true)
.map_err(|e| anyhow!("kebab-nli: tokenizer.encode failed: {e}"))?;
let ids: Vec<i64> = enc.get_ids().iter().map(|&u| i64::from(u)).collect();
let mask: Vec<i64> = enc
.get_attention_mask()
.iter()
.map(|&u| i64::from(u))
.collect();
let seq_len = ids.len();
// mDeBERTa-v3 ONNX export expects [batch, seq_len] for both
// input_ids and attention_mask. We always feed batch=1.
let ids_arr = ndarray::Array2::from_shape_vec((1, seq_len), ids)
.with_context(|| "kebab-nli: input_ids ndarray shape build failed")?;
let mask_arr = ndarray::Array2::from_shape_vec((1, seq_len), mask)
.with_context(|| "kebab-nli: attention_mask ndarray shape build failed")?;
let outputs = session
.run(ort::inputs! {
"input_ids" => ids_arr,
"attention_mask" => mask_arr,
}?)
.with_context(|| "kebab-nli: ort Session::run failed")?;
let logits = outputs["logits"]
.try_extract_tensor::<f32>()
.with_context(|| "kebab-nli: logits try_extract_tensor::<f32> failed")?;
// Expected shape [1, 3]. Defensive check — a model swap with a
// different head would silently produce wrong scores otherwise.
let shape = logits.shape();
if shape != [1, LOGITS_LEN] {
anyhow::bail!(
"kebab-nli: unexpected logits shape {shape:?}, expected [1, {LOGITS_LEN}]"
);
}
let l = [logits[[0, 0]], logits[[0, 1]], logits[[0, 2]]];
Ok(NliScores::from_xnli_logits(l))
}
/// **Override** the trait default `Ok(0)` with a real mDeBERTa
/// tokenize. Pipeline 의 `truncate_hypothesis_for_nli_with_budget`
/// retry loop 가 이 method 를 vtable 통해 호출 — production code
/// path 에서 실 token count 측정.
///
/// **CRITICAL placement**: 이 method 는 *trait impl block 안* 에
/// 위치해야 vtable 에 등록 — inherent `impl OnnxNliVerifier {}` 안에
/// 두면 dispatch 시 trait default (`Ok(0)`) 호출 → retry loop
/// 즉시 통과 → production silent NO-OP (S3 follow-up 2026-05-26
/// RC1-residual closure).
fn hypothesis_token_count(&self, hypothesis: &str) -> Result<usize> {
let (_session, tokenizer) = self.ensure_loaded()?;
let enc = tokenizer
.encode(hypothesis, /*add_special_tokens=*/ false)
.map_err(|e| anyhow!("kebab-nli: tokenizer.encode (probe) failed: {e}"))?;
Ok(enc.get_ids().len())
}
}
/// Make a HuggingFace model id (`"owner/repo"`) into a single
/// path component safe to use as a directory name. `/` → `_` is
/// enough for current ids; if more exotic chars appear we'll
/// widen this then.
fn sanitize_model_id(s: &str) -> String {
s.replace('/', "_")
}
#[cfg(test)]
mod tests {
use super::*;
use kebab_config::Config;
use tempfile::TempDir;
/// Round-1 review N2 fix: redirect Config.storage.{data,model}_dir
/// into a tempdir so unit tests don't litter the user's XDG dirs
/// with empty `nli/` subdirs.
fn tempdir_config() -> (TempDir, Config) {
let tmp = TempDir::new().expect("tempdir");
let mut cfg = Config::defaults();
cfg.storage.data_dir = tmp.path().to_string_lossy().into_owned();
cfg.storage.model_dir = "{data_dir}/models".to_string();
(tmp, cfg)
}
#[test]
fn new_succeeds_on_default_config() {
let (_tmp, cfg) = tempdir_config();
let v = OnnxNliVerifier::new(&cfg).expect("new should succeed on default config");
// cache_dir must include the sanitized model id (no '/').
let s = v.cache_dir.to_string_lossy();
assert!(s.contains(NLI_CACHE_SUBDIR), "cache_dir lacks nli/: {s}");
assert!(
!s.contains("Xenova/mDeBERTa"),
"cache_dir must sanitize '/' in model id: {s}"
);
assert!(
s.contains("Xenova_mDeBERTa"),
"cache_dir should contain sanitized id: {s}"
);
}
/// Empty hypothesis takes the defense-in-depth early bail path —
/// reaches no model load, so this is a pure unit test (no network).
/// Replaces PR-9a's `score_returns_err_in_skeleton` (stub-only).
#[test]
fn score_empty_hypothesis_returns_err() {
let (_tmp, cfg) = tempdir_config();
let v = OnnxNliVerifier::new(&cfg).unwrap();
let err = v.score("anything", "").expect_err("empty hypothesis must error");
assert!(
err.to_string().contains("empty hypothesis"),
"unexpected error message: {err}"
);
}
/// Pins that `config.models.nli.model` flows into `OnnxNliVerifier`
/// instead of being silently overridden by a hardcoded constant.
/// `model_id` is a private field, but this test lives in the same
/// module so it can read it directly — the wiring contract is
/// "whatever the user puts in TOML / KEBAB_MODELS_NLI_MODEL is the
/// id the verifier uses".
#[test]
fn new_uses_config_model_id() {
let (_tmp, mut cfg) = tempdir_config();
cfg.models.nli.model = "custom-org/custom-nli-model".to_string();
let v = OnnxNliVerifier::new(&cfg).expect("new should succeed with custom model id");
assert_eq!(v.model_id, "custom-org/custom-nli-model");
// The custom id also flows into the on-disk cache_dir layout
// (sanitized so `/` doesn't escape the namespace).
let s = v.cache_dir.to_string_lossy();
assert!(
s.contains("custom-org_custom-nli-model"),
"cache_dir should embed sanitized custom model id: {s}"
);
}
/// Pins that a non-`"onnx"` provider value errors out at `new` —
/// the field is no longer silently ignored.
#[test]
fn new_rejects_unsupported_provider() {
let (_tmp, mut cfg) = tempdir_config();
cfg.models.nli.provider = "candle".to_string();
let result = OnnxNliVerifier::new(&cfg);
assert!(result.is_err(), "non-onnx provider must error");
let msg = result.err().unwrap().to_string();
assert!(
msg.contains("unsupported provider") && msg.contains("candle"),
"error should name the rejected provider: {msg}"
);
}
// ── sanitize_model_id pure-fn coverage ────────────────────────────────
//
// Three tests pin the behavior of the private `sanitize_model_id`
// helper. These are orthogonal to the H1 executor tests above
// (which cover config-wiring); these cover the transformation
// contract of the sanitizer itself.
#[test]
fn sanitize_model_id_replaces_slash_with_underscore() {
let input = "Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7";
let expected = "Xenova_mDeBERTa-v3-base-xnli-multilingual-nli-2mil7";
assert_eq!(sanitize_model_id(input), expected);
}
#[test]
fn sanitize_model_id_is_idempotent_on_already_sanitized() {
// Input with no '/' must come back byte-for-byte unchanged.
let input = "Xenova_mDeBERTa-v3-base-xnli-multilingual-nli-2mil7";
assert_eq!(sanitize_model_id(input), input);
}
#[test]
fn sanitize_model_id_leaves_other_chars_untouched() {
// Hyphens, digits, dots, and underscores must all pass through
// unchanged — only '/' is replaced with '_'.
let input = "org_name/model-name_v2.3-alpha";
let got = sanitize_model_id(input);
assert_eq!(got, "org_name_model-name_v2.3-alpha");
assert!(!got.contains('/'), "no slash must remain after sanitize");
assert!(got.contains('-'), "hyphens must be preserved");
assert!(got.contains('.'), "dots must be preserved");
assert!(got.contains('_'), "underscores must be preserved");
}
}

View File

@@ -0,0 +1,197 @@
//! Integration tests for `OnnxNliVerifier` against the real
//! mDeBERTa-v3 XNLI model. Every test is `#[ignore]` — plain
//! `cargo test -p kebab-nli` skips them; run explicitly with
//! `cargo test -p kebab-nli --test inference -- --ignored` to
//! exercise the (slow + network-bound on first run) inference path.
//!
//! First test in the file triggers the ~280 MB ONNX + ~16 MB
//! tokenizer download into `config.storage.model_dir/nli/...`;
//! subsequent tests hit the OnceLock cache for free.
use kebab_config::Config;
use kebab_nli::{NliVerifier, OnnxNliVerifier};
/// Test 1: an English statement entails itself with high confidence.
/// Smoke evidence captured for the PR description's `## 검증` section.
#[test]
#[ignore]
fn en_self_entailment_high_score() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
let premise = "Caffeine is a stimulant.";
let hypothesis = "Caffeine is a stimulant.";
let s = v.score(premise, hypothesis).expect("score should succeed");
eprintln!(
"[test1 en_self_entailment_high_score] premise={premise:?} hypothesis={hypothesis:?} \
scores: entailment={:.4}, neutral={:.4}, contradiction={:.4}",
s.entailment, s.neutral, s.contradiction
);
assert!(
s.entailment > 0.8,
"expected entailment > 0.8, got {:.4} (full scores: {:?})",
s.entailment,
s
);
}
/// Test 2: an unrelated chemistry fact does NOT entail the premise.
/// Entailment should be low — neutral / contradiction wins.
#[test]
#[ignore]
fn en_unrelated_low_entailment() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
let premise = "Caffeine is a stimulant.";
let hypothesis = "The chemical formula of caffeine is C8H10N4O2.";
let s = v.score(premise, hypothesis).expect("score should succeed");
eprintln!(
"[test2 en_unrelated_low_entailment] \
scores: entailment={:.4}, neutral={:.4}, contradiction={:.4}",
s.entailment, s.neutral, s.contradiction
);
// spec §3 PR-9b: "entailment 낮음 — neutral/contradiction 이 winning channel" 의
// *spirit* 은 *neutral 이 max* 임. 실측 mDeBERTa 의 noise (entailment≈0.42, neutral≈0.53,
// contradiction≈0.05) 에서 두 문장 모두 caffeine 의 *사실* 이라 entailment 가 0.3 미만으로
// 떨어지지 않음 — 그러나 neutral 이 winning. multilingual NLI 의 자연스러운 동작.
assert!(
s.neutral > s.entailment && s.neutral > s.contradiction,
"expected neutral to win (no entailment, no contradiction), got {s:?}"
);
}
/// Test 3: Korean entailment. The threshold is intentionally generous
/// (> 0.5) because cross-lingual XNLI is noisier than English-only.
#[test]
#[ignore]
fn ko_entailment_high_score() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
let premise = "사과는 빨갛다.";
let hypothesis = "사과는 색이 있다.";
let s = v.score(premise, hypothesis).expect("score should succeed");
eprintln!(
"[test3 ko_entailment_high_score] \
scores: entailment={:.4}, neutral={:.4}, contradiction={:.4}",
s.entailment, s.neutral, s.contradiction
);
assert!(
s.entailment > 0.5,
"expected entailment > 0.5, got {:.4} (full scores: {:?})",
s.entailment,
s
);
}
/// Test 4: a > 24 000-char premise must not panic. mDeBERTa-v3 is
/// trained at 512 tokens; the `OnlyFirst` truncation strategy keeps
/// the premise side from blowing the positional embedding cap.
#[test]
#[ignore]
fn long_premise_truncates_without_panic() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
let premise = "foo bar baz ".repeat(2000); // ~24 000 chars
let hypothesis = "foo";
let s = v
.score(&premise, hypothesis)
.expect("score should succeed on long premise");
eprintln!(
"[test4 long_premise_truncates_without_panic] premise_len={} \
scores: entailment={:.4}, neutral={:.4}, contradiction={:.4}",
premise.len(),
s.entailment,
s.neutral,
s.contradiction
);
// No NaN / infinity in any channel.
for (name, x) in [
("entailment", s.entailment),
("neutral", s.neutral),
("contradiction", s.contradiction),
] {
assert!(
x.is_finite(),
"channel {name} non-finite: {x} (full scores: {s:?})"
);
}
// Softmax invariant — the three channels sum to ~1.
let sum = s.entailment + s.neutral + s.contradiction;
assert!(
(sum - 1.0).abs() < 1e-3,
"softmax channels must sum to ~1, got {sum:.6}"
);
}
/// Test 5: an empty hypothesis triggers the defense-in-depth bail
/// path BEFORE the tokenizer runs. Hits no network — fast, even on
/// a fresh machine.
#[test]
#[ignore]
fn empty_hypothesis_returns_err() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
let err = v
.score("anything", "")
.expect_err("empty hypothesis must error");
let msg = err.to_string();
assert!(
msg.contains("empty hypothesis"),
"expected 'empty hypothesis' in error, got: {msg}"
);
}
/// Test 6 (S3 follow-up 2026-05-26): EN-long hypothesis alone exceeds
/// max_length. Without pipeline-side truncation, `OnlyFirst` strategy
/// dead-ends. Pin raw nli crate behavior so any future regression in
/// the pipeline-side budget surfaces as a clear nli-level err.
#[test]
#[ignore]
fn score_long_en_hypothesis_returns_err_without_pipeline_truncation() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
let premise = "short premise";
let hypothesis = "lorem ipsum ".repeat(500); // ~6 000 chars / >>512 tokens
let result = v.score(premise, &hypothesis);
assert!(result.is_err(), "long hypothesis should err under OnlyFirst");
let msg = result.err().unwrap().to_string();
assert!(
msg.contains("Truncation error") || msg.contains("too short to respect"),
"expected tokenizer truncation err, got: {msg}"
);
}
/// Test 7 (S3 follow-up 2026-05-26): `hypothesis_token_count` helper —
/// pure tokenizer probe. **vtable dispatch 검증** (RC1-residual pin) —
/// concrete type 호출은 inherent method 우선이라 RC1-residual 버그
/// 잡지 못함; `&dyn NliVerifier` 통해 dispatch 해야 vtable 등록 검증.
/// inherent-only 배치 시 default `Ok(0)` 반환 → `assert!(count > 0)`
/// 실패. trait impl block 배치 시 real tokenizer → PASS. Pipeline 이
/// retry budget 결정에 사용하는 API 의 정확성 pin.
#[test]
#[ignore]
fn hypothesis_token_count_dispatches_correctly_via_dyn_trait() {
let cfg = Config::defaults();
let v = OnnxNliVerifier::new(&cfg).expect("verifier construction");
// ★ vtable dispatch — &dyn NliVerifier 통해 호출. inherent-only
// 배치 시 default `Ok(0)` 반환 → assert!(count > 0) 실패.
// trait impl block 배치 시 real tokenizer → PASS. RC1-residual
// 의 코드-수준 regression pin.
let v_dyn: &dyn NliVerifier = &v;
// 짧은 EN — 4 chars/token 추정 (27 chars / 4 = ~6 tokens)
let en_count = v_dyn
.hypothesis_token_count("short english test sentence")
.expect("EN dyn dispatch must reach real tokenizer (vtable check)");
assert!(
en_count > 0 && en_count < 20,
"EN ~6 tokens expected via vtable dispatch, got {en_count} \
(Ok(0) signals inherent-only placement bug — RC1-residual)"
);
// 짧은 KR — 1-2 chars/token (15 chars / 1.5 = ~10 tokens)
let kr_count = v_dyn
.hypothesis_token_count("짧은 한국어 테스트 문장입니다")
.expect("KR dyn dispatch must reach real tokenizer");
assert!(
kr_count > 0 && kr_count < 30,
"KR ~10 tokens expected, got {kr_count}"
);
}

View File

@@ -1,27 +0,0 @@
[package]
name = "kebab-normalize"
version = { workspace = true }
edition = { workspace = true }
rust-version = { workspace = true }
license = { workspace = true }
repository = { workspace = true }
description = "Lift parser output (kb-parse-types) into kb-core::CanonicalDocument with deterministic IDs (§3.4, §4.2, §4.3)"
[dependencies]
kebab-core = { path = "../kebab-core" }
kebab-parse-types = { path = "../kebab-parse-types" }
serde = { workspace = true }
serde_json = { workspace = true }
unicode-normalization = "0.1"
time = { workspace = true }
anyhow = { workspace = true }
tracing = { workspace = true }
[dev-dependencies]
# kb-parse-md is permitted as a *dev*-dependency only — used by the
# integration snapshot test to drive a fixture through the real parser.
# Forbidden as a regular dep per design §8 (kb-normalize must not depend
# on any specific parser); `cargo tree -p kb-normalize --depth 1` (the
# default scope, excluding dev-deps) confirms this.
kebab-parse-md = { path = "../kebab-parse-md" }
serde_json = { workspace = true }

View File

@@ -22,6 +22,11 @@ tree-sitter-javascript = { workspace = true }
tree-sitter-go = { workspace = true }
tree-sitter-java = { workspace = true }
tree-sitter-kotlin-ng = { workspace = true }
tree-sitter-c = { workspace = true }
tree-sitter-cpp = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
[lints]
workspace = true

View File

@@ -0,0 +1,720 @@
//! `kebab-parse-code::c` — tree-sitter C AST extractor (P10-1D Task B).
//!
//! Implements [`kebab_core::Extractor`] for [`MediaType::Code("c")`].
//! Walks the tree-sitter parse tree and emits one [`Block::Code`] per
//! top-level AST semantic unit:
//!
//! - `function_definition` → 1 unit, symbol = function name (extracted
//! from the declarator's innermost `identifier`, handles pointer-returning
//! functions where the declarator is wrapped in `pointer_declarator`).
//! - `struct_specifier` (named) → 1 unit, symbol = struct name.
//! - `enum_specifier` (named) → 1 unit, symbol = enum name.
//! - `union_specifier` (named) → 1 unit, symbol = union name.
//!
//! Everything else (`declaration`, `preproc_*`, `type_definition`,
//! `linkage_specification`, etc.) collapses into a single `<top-level>`
//! glue chunk. If the file produces zero units **and** zero glue, the
//! `<module>` post-pass emits one unit covering the whole file (1A-2
//! pattern).
//!
//! C symbol = function name only — no namespace, no class nesting
//! (design §3.4 C row). Per design §3.4 / §9.1 / §9 versioning.
use anyhow::Result;
use kebab_core::{
Block, CanonicalDocument, CodeBlock, CommonBlock, Extractor, Lang, MediaType, Metadata,
ParserVersion, Provenance, ProvenanceEvent, ProvenanceKind, SourceSpan, SourceType, TrustLevel,
id_for_block, id_for_doc,
};
use serde_json::Map;
use time::OffsetDateTime;
use crate::scaffold::{filename_from_workspace_path, strip_extension};
pub const PARSER_VERSION: &str = "code-c-v2";
/// C AST extractor. Per-unit blocks via tree-sitter-c 0.24.2
/// (`LANGUAGE: LanguageFn`) parsed by tree-sitter 0.26.
pub struct CAstExtractor;
impl CAstExtractor {
pub fn new() -> Self {
Self
}
}
impl Default for CAstExtractor {
fn default() -> Self {
Self::new()
}
}
impl Extractor for CAstExtractor {
fn supports(&self, m: &MediaType) -> bool {
matches!(m, MediaType::Code(l) if l == "c")
}
fn parser_version(&self) -> ParserVersion {
ParserVersion(PARSER_VERSION.to_string())
}
fn extract(
&self,
ctx: &kebab_core::ExtractContext<'_>,
bytes: &[u8],
) -> Result<CanonicalDocument> {
let asset = ctx.asset;
if !self.supports(&asset.media_type) {
anyhow::bail!(
"kebab-parse-code: unsupported media_type for CAstExtractor: {:?}",
asset.media_type
);
}
let parser_version = self.parser_version();
let doc_id = id_for_doc(&asset.workspace_path, &asset.asset_id, &parser_version);
let source = String::from_utf8(bytes.to_vec())
.map_err(|e| anyhow::anyhow!("kebab-parse-code: C source is not valid UTF-8: {e}"))?;
let blocks = build_blocks(&source, &doc_id)?;
let unit_count = blocks.len() as u32;
let now = OffsetDateTime::now_utc();
let mut events: Vec<ProvenanceEvent> = Vec::with_capacity(2);
events.push(ProvenanceEvent {
at: asset.discovered_at,
agent: "kb-source-fs".to_string(),
kind: ProvenanceKind::Discovered,
note: None,
});
events.push(ProvenanceEvent {
at: now,
agent: "kb-parse-code".to_string(),
kind: ProvenanceKind::Parsed,
note: Some(format!(
"parser_version={}; unit_count={}",
parser_version.0, unit_count
)),
});
let title = {
let fname = filename_from_workspace_path(&asset.workspace_path.0);
strip_extension(&fname)
};
let abs_path = match &asset.source_uri {
kebab_core::SourceUri::File(p) => {
if p.is_absolute() {
p.clone()
} else {
ctx.workspace_root.join(p)
}
}
kebab_core::SourceUri::Kb(_) => ctx.workspace_root.to_path_buf(),
};
let (repo, git_branch, git_commit) = match crate::repo::detect_repo(&abs_path) {
Some(r) => (Some(r.name), r.branch, r.commit),
None => (None, None, None),
};
let metadata = Metadata {
aliases: Vec::new(),
tags: Vec::new(),
created_at: asset.discovered_at,
updated_at: asset.discovered_at,
source_type: SourceType::Note,
trust_level: TrustLevel::Primary,
user_id_alias: None,
user: Map::new(),
repo,
git_branch,
git_commit,
code_lang: Some("c".to_string()),
};
tracing::debug!(
target: "kebab-parse-code",
"extracted C doc_id={} workspace_path={} units={}",
doc_id.0,
asset.workspace_path.0,
unit_count
);
Ok(CanonicalDocument {
doc_id,
source_asset_id: asset.asset_id.clone(),
workspace_path: asset.workspace_path.clone(),
title,
lang: Lang("und".to_string()),
blocks,
metadata,
provenance: Provenance { events },
parser_version,
schema_version: 1,
doc_version: 1,
last_chunker_version: None,
last_embedding_version: None,
})
}
}
/// Walk down the declarator chain of a `function_definition` to find
/// the innermost `identifier` — the function name.
///
/// The tree for `int *foo(int x) { ... }` looks like:
/// ```text
/// function_definition
/// type: primitive_type "int"
/// declarator: pointer_declarator
/// declarator: function_declarator
/// declarator: identifier "foo"
/// parameters: parameter_list
/// body: compound_statement
/// ```
/// We walk `declarator` fields recursively until we reach an `identifier`
/// or run out of nodes. Returns `None` if no identifier is found
/// (malformed / unsupported declarator shape).
fn extract_fn_name<'a>(decl_node: tree_sitter::Node, src: &'a str) -> Option<&'a str> {
let mut cur = decl_node;
loop {
match cur.kind() {
"identifier" => return Some(&src[cur.start_byte()..cur.end_byte()]),
// pointer_declarator, function_declarator, array_declarator,
// attributed_declarator, parenthesized_declarator —
// all carry a `declarator` field pointing deeper.
_ => {
if let Some(inner) = cur.child_by_field_name("declarator") {
cur = inner;
} else {
// No further `declarator` field; give up.
return None;
}
}
}
}
}
fn build_blocks(
source: &str,
doc_id: &kebab_core::DocumentId,
) -> anyhow::Result<Vec<kebab_core::Block>> {
let mut parser = tree_sitter::Parser::new();
parser
.set_language(&tree_sitter_c::LANGUAGE.into())
.map_err(|e| anyhow::anyhow!("set tree-sitter-c language: {e}"))?;
let tree = parser
.parse(source.as_bytes(), None)
.ok_or_else(|| anyhow::anyhow!("tree-sitter failed to parse C source"))?;
let lines: Vec<&str> = source.split('\n').collect();
let root = tree.root_node();
// units: (symbol, line_start, line_end, is_real_semantic_unit).
// Glue is accumulated as (start, end) pairs and flushed into one
// "<top-level>" block (or "<module>" if no real unit exists).
let mut units: Vec<(String, u32, u32, bool)> = Vec::new();
let mut glue: Vec<(u32, u32)> = Vec::new();
/// Walk preceding `comment` siblings to extend the unit's line range
/// upward, folding doc / line comments into the unit (1B pattern).
fn unit_start(n: &tree_sitter::Node) -> u32 {
let mut start = n.start_position().row as u32 + 1;
let mut prev = n.prev_sibling();
while let Some(p) = prev {
if p.kind() == "comment" {
start = p.start_position().row as u32 + 1;
prev = p.prev_sibling();
} else {
break;
}
}
start
}
let mut cur = root.walk();
for child in root.named_children(&mut cur) {
let s = unit_start(&child);
let e = child.end_position().row as u32 + 1;
match child.kind() {
"function_definition" => {
if let Some(decl) = child.child_by_field_name("declarator") {
if let Some(name) = extract_fn_name(decl, source) {
flush_glue(&mut glue, &mut units);
units.push((name.to_string(), s, e, true));
} else {
// Could not extract name — treat as glue.
glue.push((s, e));
}
} else {
glue.push((s, e));
}
}
"struct_specifier" | "enum_specifier" | "union_specifier" => {
if let Some(name_node) = child.child_by_field_name("name") {
let name = &source[name_node.start_byte()..name_node.end_byte()];
flush_glue(&mut glue, &mut units);
units.push((name.to_string(), s, e, true));
} else {
// Anonymous struct/enum/union at the top level (not
// wrapped in typedef) — glue. typedef-wrapped case
// is recovered in the `type_definition` arm below.
glue.push((s, e));
}
}
"type_definition" => {
// v0.17.0 PR-B: typedef-wrapped anonymous aggregate
// recovery. `typedef struct { ... } Foo;` exposes only
// the alias `Foo` as a useful symbol — the inner
// struct_specifier has no `name` field. Pre-v0.17.0
// this whole construct collapsed into glue and hid the
// alias from search (HOTFIXES 2026-05-21). v2 recovers
// the alias from the `declarator` field and emits a
// synthetic unit so `Citation::Code.symbol = "Foo"`.
// Plain `typedef int MyInt;` (no inner aggregate) stays
// glue — there's no struct body to name.
if let Some(name) = recover_typedef_alias(child, source) {
flush_glue(&mut glue, &mut units);
units.push((name, s, e, true));
} else {
glue.push((s, e));
}
}
// Everything else: preprocessor directives, plain declarations
// (global var / fn prototype), linkage_specification, etc.
// — all collapse into glue.
_ => {
glue.push((s, e));
}
}
}
flush_glue(&mut glue, &mut units);
// Post-pass: if the file has no real semantic unit (only glue, or
// completely empty), rename the single glue unit to "<module>" and
// emit it. If there are zero units AND zero glue, synthesise a
// one-line "<module>" covering the whole file.
let has_real_unit = units.iter().any(|(_, _, _, is_real)| *is_real);
if units.is_empty() {
// Completely empty file or whitespace/comments only.
let total = lines.len() as u32;
units.push((
"<module>".to_string(),
1,
total.max(1),
false,
));
}
// If there is only glue (no real unit) the single pushed "<top-level>"
// label should be "<module>" — rename it now.
if !has_real_unit {
for (sym, _, _, _) in &mut units {
if sym == "<top-level>" {
*sym = "<module>".to_string();
}
}
}
let total_lines = lines.len() as u32;
let mut blocks = Vec::with_capacity(units.len());
for (ordinal, (symbol, ls, le, _is_real)) in units.into_iter().enumerate() {
let line_start = ls.max(1);
let line_end = le.min(total_lines.max(1));
let span = SourceSpan::Code {
line_start,
line_end,
symbol: Some(symbol),
lang: Some("c".to_string()),
};
let block_id = id_for_block(doc_id, "code", &[], ordinal as u32, &span);
let code = lines[(line_start as usize - 1)..(line_end as usize)].join("\n");
blocks.push(Block::Code(CodeBlock {
common: CommonBlock {
block_id,
heading_path: Vec::new(),
source_span: span,
},
lang: Some("c".to_string()),
code,
}));
}
Ok(blocks)
}
/// v0.17.0 PR-B: try to recover the typedef alias name from a
/// `type_definition` node *iff* the inner type-specifier is an
/// anonymous struct/enum/union. Returns `None` for any other shape
/// (named aggregate handled elsewhere, plain type alias has no body
/// worth naming).
fn recover_typedef_alias(node: tree_sitter::Node, source: &str) -> Option<String> {
let mut has_anon_aggregate = false;
let mut cursor = node.walk();
for sub in node.children(&mut cursor) {
match sub.kind() {
"struct_specifier" | "enum_specifier" | "union_specifier" => {
if sub.child_by_field_name("name").is_none() {
has_anon_aggregate = true;
} else {
// Named inner aggregate (e.g. `typedef struct Pt {...} P;`)
// — the named struct itself is the primary symbol and
// is *not* extracted at the top level today (it lives
// inside `type_definition`, not as a sibling
// `struct_specifier`). For v2 we keep behavior conservative:
// return None so the type_definition stays glue, matching
// pre-v2 behavior for this minor case. Real-world C tends
// to use one of: bare named struct, typedef alias only,
// or typedef on anonymous body — the latter is what we fix.
return None;
}
}
_ => {}
}
}
if !has_anon_aggregate {
return None;
}
let decl = node.child_by_field_name("declarator")?;
extract_typedef_alias_name(decl, source).map(str::to_string)
}
/// Extract the typedef alias identifier from a declarator subtree.
/// Handles the common shapes: direct `type_identifier`, or one wrapped
/// in pointer / function declarator nodes (the alias is always the
/// rightmost `type_identifier` descendant).
fn extract_typedef_alias_name<'a>(
decl: tree_sitter::Node,
source: &'a str,
) -> Option<&'a str> {
if decl.kind() == "type_identifier" {
return Some(&source[decl.start_byte()..decl.end_byte()]);
}
let mut cursor = decl.walk();
for sub in decl.children(&mut cursor) {
if let Some(found) = extract_typedef_alias_name(sub, source) {
return Some(found);
}
}
None
}
fn flush_glue(glue: &mut Vec<(u32, u32)>, units: &mut Vec<(String, u32, u32, bool)>) {
if glue.is_empty() {
return;
}
let s = glue.iter().map(|(a, _)| *a).min().unwrap();
let e = glue.iter().map(|(_, b)| *b).max().unwrap();
units.push(("<top-level>".to_string(), s, e, false));
glue.clear();
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
pub(crate) mod tests_support {
use kebab_core::*;
use std::path::PathBuf;
use time::OffsetDateTime;
pub fn fixed_code_asset(workspace_path: &str, lang: &str) -> RawAsset {
RawAsset {
asset_id: AssetId("a".repeat(64)),
source_uri: SourceUri::File(PathBuf::from(workspace_path)),
workspace_path: WorkspacePath(workspace_path.to_string()),
media_type: MediaType::Code(lang.to_string()),
byte_len: 0,
checksum: Checksum("b".repeat(64)),
discovered_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
stored: AssetStorage::Reference {
path: PathBuf::from(workspace_path),
sha: Checksum("b".repeat(64)),
},
}
}
pub fn extract_c(src: &str, path: &str) -> kebab_core::CanonicalDocument {
use super::CAstExtractor;
use kebab_core::Extractor;
let asset = fixed_code_asset(path, "c");
let cfg = ExtractConfig::default();
let root = PathBuf::from("/tmp");
let ctx = ExtractContext {
asset: &asset,
workspace_root: &root,
config: &cfg,
};
CAstExtractor::new().extract(&ctx, src.as_bytes()).unwrap()
}
}
#[cfg(test)]
mod tests {
use super::*;
use kebab_core::{Block, MediaType, SourceSpan};
fn syms(doc: &kebab_core::CanonicalDocument) -> Vec<String> {
doc.blocks
.iter()
.filter_map(|b| match b {
Block::Code(c) => match &c.common.source_span {
SourceSpan::Code { symbol, .. } => symbol.clone(),
_ => None,
},
_ => None,
})
.collect()
}
#[test]
fn extractor_supports_only_media_code_c() {
let e = CAstExtractor::new();
assert!(e.supports(&MediaType::Code("c".into())));
assert!(!e.supports(&MediaType::Code("cpp".into())));
assert!(!e.supports(&MediaType::Code("rust".into())));
assert!(!e.supports(&MediaType::Markdown));
}
#[test]
fn c_extractor_simple_function() {
let src = "int add(int a, int b) { return a + b; }\n";
let doc = tests_support::extract_c(src, "x/math.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "add"), "got {s:?}");
}
#[test]
fn c_extractor_pointer_return_function() {
let src = "int *find(int *arr, int n) { return arr; }\n";
let doc = tests_support::extract_c(src, "x/find.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "find"), "ptr-return fn missing: {s:?}");
}
#[test]
fn c_extractor_static_function() {
let src = "static void helper(void) {}\n";
let doc = tests_support::extract_c(src, "x/helper.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "helper"), "static fn missing: {s:?}");
}
#[test]
fn c_extractor_extern_function() {
let src = "extern int compute(int x);\n";
// extern prototype is a declaration → glue
let doc = tests_support::extract_c(src, "x/compute.c");
let s = syms(&doc);
// declaration (prototype) falls into glue → "<module>"
assert!(
s.iter().any(|x| x == "<module>"),
"expected <module> for extern proto: {s:?}"
);
}
#[test]
fn c_extractor_inline_function() {
let src = "inline int square(int x) { return x * x; }\n";
let doc = tests_support::extract_c(src, "x/square.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "square"), "inline fn missing: {s:?}");
}
#[test]
fn c_extractor_named_struct() {
let src = "struct Point { int x; int y; };\n";
let doc = tests_support::extract_c(src, "x/point.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "Point"), "struct missing: {s:?}");
}
#[test]
fn c_extractor_named_enum() {
let src = "enum Color { RED, GREEN, BLUE };\n";
let doc = tests_support::extract_c(src, "x/color.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "Color"), "enum missing: {s:?}");
}
#[test]
fn c_extractor_named_union() {
let src = "union Data { int i; float f; };\n";
let doc = tests_support::extract_c(src, "x/data.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "Data"), "union missing: {s:?}");
}
#[test]
fn c_extractor_anonymous_struct_falls_into_glue() {
// Anonymous struct (no name field) → glue → "<module>" (only glue, no real unit)
let src = "struct { int x; int y; } origin;\n";
let doc = tests_support::extract_c(src, "x/anon.c");
let s = syms(&doc);
// anonymous struct is a declaration containing anonymous struct_specifier → glue
assert!(
s.iter().any(|x| x == "<module>"),
"expected <module> for anon struct: {s:?}"
);
// Must NOT emit a unit named after anything else
assert!(
!s.iter().any(|x| x == "origin"),
"unexpected 'origin' unit: {s:?}"
);
}
#[test]
fn c_extractor_typedef_struct_emits_unit() {
// v0.17.0 PR-B: `typedef struct { ... } Foo;` was previously a
// hotfix-tracked deviation (HOTFIXES.md 2026-05-21) — the inner
// struct_specifier is anonymous so the named-struct arm didn't
// fire, dropping the whole construct into glue and hiding the
// `Foo` alias from symbol search. The v2 extractor recovers the
// typedef alias from the `declarator` field on the
// `type_definition` node and emits a synthetic unit with that
// name. parser_version bumped `code-c-v1` → `code-c-v2`.
let src = "typedef struct { int x; int y; } Point;\n";
let doc = tests_support::extract_c(src, "x/typedef.c");
let s = syms(&doc);
// The typedef alias surfaces as a Code symbol.
assert!(
s.iter().any(|x| x == "Point"),
"expected 'Point' unit from typedef alias: {s:?}"
);
// No `<module>` (the file has exactly one semantic unit now,
// the typedef alias — no glue-only fallback needed).
assert!(
!s.iter().any(|x| x == "<module>"),
"no <module> fallback expected when typedef emits a unit: {s:?}"
);
}
#[test]
fn c_extractor_typedef_enum_emits_unit() {
// Parallel coverage for enum_specifier — same typedef-alias
// synthesis path. `typedef enum { A, B } Color;` → unit `Color`.
let src = "typedef enum { A, B } Color;\n";
let doc = tests_support::extract_c(src, "x/typedef_enum.c");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "Color"),
"expected 'Color' unit from typedef enum alias: {s:?}"
);
}
#[test]
fn c_extractor_typedef_union_emits_unit() {
// Parallel coverage for union_specifier.
let src = "typedef union { int i; float f; } IntOrFloat;\n";
let doc = tests_support::extract_c(src, "x/typedef_union.c");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "IntOrFloat"),
"expected 'IntOrFloat' unit from typedef union alias: {s:?}"
);
}
#[test]
fn c_extractor_typedef_to_existing_type_stays_glue() {
// Negative case: `typedef int MyInt;` has no inner struct/enum/
// union — there's no struct body to attach the alias to, so the
// construct falls into glue (becomes `<module>` when alone).
// Confirms the new arm only fires for anonymous-struct typedef.
let src = "typedef int MyInt;\n";
let doc = tests_support::extract_c(src, "x/typedef_alias.c");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "<module>"),
"expected <module> for plain typedef alias: {s:?}"
);
assert!(
!s.iter().any(|x| x == "MyInt"),
"plain typedef alias must not emit a unit: {s:?}"
);
}
#[test]
fn c_extractor_preprocessor_directives_are_glue() {
let src = "#include <stdio.h>\n#define MAX 100\n#ifdef DEBUG\n#endif\n";
let doc = tests_support::extract_c(src, "x/macros.c");
let s = syms(&doc);
// Only preprocessor → no real unit → "<module>"
assert!(
s.iter().any(|x| x == "<module>"),
"expected <module> for preproc-only file: {s:?}"
);
assert_eq!(s.len(), 1, "expected exactly 1 block: {s:?}");
}
#[test]
fn c_extractor_multiple_functions_correct_count() {
let src = "int foo(void) { return 1; }\nint bar(void) { return 2; }\nint baz(void) { return 3; }\n";
let doc = tests_support::extract_c(src, "x/multi.c");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "foo"), "foo missing: {s:?}");
assert!(s.iter().any(|x| x == "bar"), "bar missing: {s:?}");
assert!(s.iter().any(|x| x == "baz"), "baz missing: {s:?}");
assert_eq!(s.len(), 3, "expected 3 units: {s:?}");
}
#[test]
fn c_extractor_empty_file_produces_module() {
let src = "";
let doc = tests_support::extract_c(src, "x/empty.c");
let s = syms(&doc);
assert_eq!(s, vec!["<module>"], "expected <module>: got {s:?}");
}
#[test]
fn c_extractor_preprocessor_only_produces_module() {
let src = "#include <stdlib.h>\n#define VERSION \"1.0\"\n";
let doc = tests_support::extract_c(src, "x/header.c");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "<module>"),
"expected <module> for preproc-only file: {s:?}"
);
}
#[test]
fn c_extractor_mixed_functions_and_glue() {
let src = r#"#include <stdio.h>
int compute(int x) {
return x * 2;
}
extern int lookup(int key);
void print_result(int v) {
printf("%d\n", v);
}
"#;
let doc = tests_support::extract_c(src, "x/mixed.c");
let s = syms(&doc);
// Two real functions + one glue block
assert!(s.iter().any(|x| x == "compute"), "compute missing: {s:?}");
assert!(s.iter().any(|x| x == "print_result"), "print_result missing: {s:?}");
assert!(
s.iter().any(|x| x == "<top-level>"),
"<top-level> glue missing: {s:?}"
);
}
#[test]
fn c_extractor_deterministic_across_runs() {
let src = r"
struct Node { int val; };
int sum(int a, int b) { return a + b; }
void noop(void) {}
";
let a = tests_support::extract_c(src, "x/det.c");
for _ in 0..20 {
assert_eq!(
tests_support::extract_c(src, "x/det.c").blocks,
a.blocks
);
}
}
}

View File

@@ -0,0 +1,883 @@
//! `kebab-parse-code::cpp` — tree-sitter C++ AST extractor (P10-1D Task C).
//!
//! Implements [`kebab_core::Extractor`] for [`MediaType::Code("cpp")`].
//! Walks the tree-sitter parse tree and emits one [`Block::Code`] per
//! top-level AST semantic unit, each carrying [`SourceSpan::Code`] with
//! the unit's `::` separated symbol path (design §3.4 C++ row).
//!
//! ## Symbol formation
//!
//! Symbol = `namespace::Class::method` via recursive `build_blocks`:
//!
//! - `namespace_definition` (named) → push namespace name, recurse into body.
//! - Anonymous namespace (`namespace { ... }`) → push `<anonymous>`, recurse.
//! - `nested_namespace_specifier` (`outer::inner`) → push all segments, recurse.
//! - `class_specifier` / `struct_specifier` (named) → emit class unit + recurse
//! into body with class name pushed.
//! - `function_definition` → emit method/function unit. Symbol is built from
//! the prefix chain + the extracted declarator name component.
//! - Out-of-class method def (`void Foo::bar() {}`) — the declarator's inner
//! node is a `qualified_identifier`; its scope chain is prepended to the
//! current prefix to form the full symbol.
//! - `template_declaration` → recurse into named children with same prefix;
//! the inner function/class body is matched by its own arm. Template params
//! are NOT included in the symbol.
//! - `enum_specifier` (named) → emit type unit.
//! - `concept_definition` (C++20) → emit type unit.
//! - `linkage_specification` (extern "C") → recurse into body with same prefix.
//!
//! ## Constructor / destructor / operator overload
//!
//! - Constructor: `function_declarator > identifier` matching the class name.
//! Symbol = `Class::Class` (name duplicated, same convention as Java).
//! - Destructor: `function_declarator > destructor_name`. Symbol = `Class::~Foo`.
//! - Operator overload: `function_declarator > operator_name`. Symbol = `Class::operator+`.
//! - Conversion operator: `function_definition.declarator` is `operator_cast`.
//! Symbol = `Class::operator <type>` (e.g. `Class::operator bool`).
//!
//! ## Glue
//!
//! Everything not in the unit list collapses into a single `<top-level>` glue
//! chunk (preproc, declarations, using, typedef, etc.). If the file produces
//! zero units AND zero glue, the `<module>` post-pass emits one unit covering
//! the whole file.
//!
//! Per design §3.4 / §9.1 / §9 versioning.
use anyhow::Result;
use kebab_core::{
Block, CanonicalDocument, CodeBlock, CommonBlock, Extractor, Lang, MediaType, Metadata,
ParserVersion, Provenance, ProvenanceEvent, ProvenanceKind, SourceSpan, SourceType, TrustLevel,
id_for_block, id_for_doc,
};
use serde_json::Map;
use time::OffsetDateTime;
use crate::scaffold::{filename_from_workspace_path, strip_extension};
pub const PARSER_VERSION: &str = "code-cpp-v1";
/// C++ AST extractor. Per-unit blocks via tree-sitter-cpp 0.23.4
/// (`LANGUAGE: LanguageFn`) parsed by tree-sitter 0.26.
pub struct CppAstExtractor;
impl CppAstExtractor {
pub fn new() -> Self {
Self
}
}
impl Default for CppAstExtractor {
fn default() -> Self {
Self::new()
}
}
impl Extractor for CppAstExtractor {
fn supports(&self, m: &MediaType) -> bool {
matches!(m, MediaType::Code(l) if l == "cpp")
}
fn parser_version(&self) -> ParserVersion {
ParserVersion(PARSER_VERSION.to_string())
}
fn extract(
&self,
ctx: &kebab_core::ExtractContext<'_>,
bytes: &[u8],
) -> Result<CanonicalDocument> {
let asset = ctx.asset;
if !self.supports(&asset.media_type) {
anyhow::bail!(
"kebab-parse-code: unsupported media_type for CppAstExtractor: {:?}",
asset.media_type
);
}
let parser_version = self.parser_version();
let doc_id = id_for_doc(&asset.workspace_path, &asset.asset_id, &parser_version);
let source = String::from_utf8(bytes.to_vec()).map_err(|e| {
anyhow::anyhow!("kebab-parse-code: C++ source is not valid UTF-8: {e}")
})?;
let blocks = build_blocks_top(&source, &doc_id)?;
let unit_count = blocks.len() as u32;
let now = OffsetDateTime::now_utc();
let mut events: Vec<ProvenanceEvent> = Vec::with_capacity(2);
events.push(ProvenanceEvent {
at: asset.discovered_at,
agent: "kb-source-fs".to_string(),
kind: ProvenanceKind::Discovered,
note: None,
});
events.push(ProvenanceEvent {
at: now,
agent: "kb-parse-code".to_string(),
kind: ProvenanceKind::Parsed,
note: Some(format!(
"parser_version={}; unit_count={}",
parser_version.0, unit_count
)),
});
let title = {
let fname = filename_from_workspace_path(&asset.workspace_path.0);
strip_extension(&fname)
};
let abs_path = match &asset.source_uri {
kebab_core::SourceUri::File(p) => {
if p.is_absolute() {
p.clone()
} else {
ctx.workspace_root.join(p)
}
}
kebab_core::SourceUri::Kb(_) => ctx.workspace_root.to_path_buf(),
};
let (repo, git_branch, git_commit) = match crate::repo::detect_repo(&abs_path) {
Some(r) => (Some(r.name), r.branch, r.commit),
None => (None, None, None),
};
let metadata = Metadata {
aliases: Vec::new(),
tags: Vec::new(),
created_at: asset.discovered_at,
updated_at: asset.discovered_at,
source_type: SourceType::Note,
trust_level: TrustLevel::Primary,
user_id_alias: None,
user: Map::new(),
repo,
git_branch,
git_commit,
code_lang: Some("cpp".to_string()),
};
tracing::debug!(
target: "kebab-parse-code",
"extracted C++ doc_id={} workspace_path={} units={}",
doc_id.0,
asset.workspace_path.0,
unit_count
);
Ok(CanonicalDocument {
doc_id,
source_asset_id: asset.asset_id.clone(),
workspace_path: asset.workspace_path.clone(),
title,
lang: Lang("und".to_string()),
blocks,
metadata,
provenance: Provenance { events },
parser_version,
schema_version: 1,
doc_version: 1,
last_chunker_version: None,
last_embedding_version: None,
})
}
}
// ---------------------------------------------------------------------------
// Core block-building logic
// ---------------------------------------------------------------------------
/// Top-level entry: parse source, walk the `translation_unit` root, assemble
/// units + glue, apply the `<module>` post-pass, and emit `Block::Code`s.
fn build_blocks_top(
source: &str,
doc_id: &kebab_core::DocumentId,
) -> anyhow::Result<Vec<kebab_core::Block>> {
let mut parser = tree_sitter::Parser::new();
parser
.set_language(&tree_sitter_cpp::LANGUAGE.into())
.map_err(|e| anyhow::anyhow!("set tree-sitter-cpp language: {e}"))?;
let tree = parser
.parse(source.as_bytes(), None)
.ok_or_else(|| anyhow::anyhow!("tree-sitter failed to parse C++ source"))?;
let lines: Vec<&str> = source.split('\n').collect();
let root = tree.root_node();
// units: (symbol, line_start, line_end, is_real_semantic_unit).
// Glue is accumulated as (start, end) pairs and flushed into one
// "<top-level>" block (or "<module>" if no real unit exists).
let mut units: Vec<(String, u32, u32, bool)> = Vec::new();
let mut glue: Vec<(u32, u32)> = Vec::new();
build_blocks(root, source, &[], &mut units, &mut glue);
flush_glue(&mut glue, &mut units);
// Post-pass: if the file has no real semantic unit (only glue, or
// completely empty), rename the single glue unit to "<module>".
// If there are zero units AND zero glue, synthesize a one-line
// "<module>" covering the whole file.
let has_real_unit = units.iter().any(|(_, _, _, is_real)| *is_real);
if units.is_empty() {
let total = lines.len() as u32;
units.push(("<module>".to_string(), 1, total.max(1), false));
}
if !has_real_unit {
for (sym, _, _, _) in &mut units {
if sym == "<top-level>" {
*sym = "<module>".to_string();
}
}
}
let total_lines = lines.len() as u32;
let mut blocks = Vec::with_capacity(units.len());
for (ordinal, (symbol, ls, le, _is_real)) in units.into_iter().enumerate() {
let line_start = ls.max(1);
let line_end = le.min(total_lines.max(1));
let span = SourceSpan::Code {
line_start,
line_end,
symbol: Some(symbol),
lang: Some("cpp".to_string()),
};
let block_id = id_for_block(doc_id, "code", &[], ordinal as u32, &span);
let code = lines[(line_start as usize - 1)..(line_end as usize)].join("\n");
blocks.push(Block::Code(CodeBlock {
common: CommonBlock {
block_id,
heading_path: Vec::new(),
source_span: span,
},
lang: Some("cpp".to_string()),
code,
}));
}
Ok(blocks)
}
/// Walk preceding `comment` siblings to extend the unit's line range upward,
/// folding leading doc / line comments into the unit (1B pattern).
fn unit_start(n: &tree_sitter::Node) -> u32 {
let mut start = n.start_position().row as u32 + 1;
let mut prev = n.prev_sibling();
while let Some(p) = prev {
if p.kind() == "comment" {
start = p.start_position().row as u32 + 1;
prev = p.prev_sibling();
} else {
break;
}
}
start
}
fn flush_glue(glue: &mut Vec<(u32, u32)>, units: &mut Vec<(String, u32, u32, bool)>) {
if glue.is_empty() {
return;
}
let s = glue.iter().map(|(a, _)| *a).min().unwrap();
let e = glue.iter().map(|(_, b)| *b).max().unwrap();
units.push(("<top-level>".to_string(), s, e, false));
glue.clear();
}
/// Walk a scope node (translation_unit, declaration_list, field_declaration_list)
/// emitting unit + glue blocks. `prefix` is the current namespace/class chain
/// (e.g. `["kebab", "Chunk", "Foo"]`).
///
/// After returning, any pending glue in `glue` is NOT flushed — callers
/// responsible for flushing at the scope boundary (top-level flush in
/// `build_blocks_top`). Within recursive scope bodies (namespace/class) we
/// do flush before returning so that glue doesn't leak across scopes.
fn build_blocks(
node: tree_sitter::Node,
source: &str,
prefix: &[String],
units: &mut Vec<(String, u32, u32, bool)>,
glue: &mut Vec<(u32, u32)>,
) {
let mut cur = node.walk();
for child in node.named_children(&mut cur) {
let s = unit_start(&child);
let e = child.end_position().row as u32 + 1;
match child.kind() {
"namespace_definition" => {
// Flush pending glue before starting this namespace block.
flush_glue(glue, units);
let name_node = child.child_by_field_name("name");
let body = child
.child_by_field_name("body")
.unwrap_or(child);
match name_node {
None => {
// Anonymous namespace: push "<anonymous>", recurse.
let mut new_prefix = prefix.to_vec();
new_prefix.push("<anonymous>".to_string());
build_blocks(body, source, &new_prefix, units, glue);
flush_glue(glue, units);
}
Some(nn) => match nn.kind() {
"namespace_identifier" => {
let name = &source[nn.start_byte()..nn.end_byte()];
let mut new_prefix = prefix.to_vec();
new_prefix.push(name.to_string());
build_blocks(body, source, &new_prefix, units, glue);
flush_glue(glue, units);
}
"nested_namespace_specifier" => {
// e.g. `namespace outer::inner { ... }`
// All named children are namespace_identifier nodes.
let mut new_prefix = prefix.to_vec();
let mut nc = nn.walk();
for seg in nn.named_children(&mut nc) {
new_prefix.push(source[seg.start_byte()..seg.end_byte()].to_string());
}
build_blocks(body, source, &new_prefix, units, glue);
flush_glue(glue, units);
}
_ => {
// Unknown name kind — treat entire namespace as glue.
glue.push((s, e));
}
},
}
}
"class_specifier" | "struct_specifier" => {
let name_node = child.child_by_field_name("name");
let Some(nn) = name_node else {
// Anonymous class/struct — glue.
glue.push((s, e));
continue;
};
let name = match nn.kind() {
"type_identifier" => &source[nn.start_byte()..nn.end_byte()],
_ => {
// template_type or qualified_identifier — use full text
// as the symbol segment (includes template args).
&source[nn.start_byte()..nn.end_byte()]
}
};
flush_glue(glue, units);
let sym = build_symbol(prefix, &[name]);
units.push((sym, s, e, true));
if let Some(body) = child.child_by_field_name("body") {
let mut new_prefix = prefix.to_vec();
new_prefix.push(name.to_string());
build_blocks(body, source, &new_prefix, units, glue);
flush_glue(glue, units);
}
}
"function_definition" => {
let decl = child.child_by_field_name("declarator");
let Some(decl_node) = decl else {
glue.push((s, e));
continue;
};
match extract_fn_symbol(decl_node, source, prefix) {
Some(sym) => {
flush_glue(glue, units);
units.push((sym, s, e, true));
}
None => {
glue.push((s, e));
}
}
}
"template_declaration" => {
// Unwrap: recurse into named children with same prefix.
// The inner function/class/concept will be matched by their own
// arms. template_parameter_list is not a unit; it will fall
// through to glue (it's not a named child of the template_declaration
// that matches any of our arms).
build_blocks(child, source, prefix, units, glue);
// Do NOT flush glue here — template body may be part of a glue group.
}
"enum_specifier" => {
if let Some(nn) = child.child_by_field_name("name") {
let name = &source[nn.start_byte()..nn.end_byte()];
flush_glue(glue, units);
let sym = build_symbol(prefix, &[name]);
units.push((sym, s, e, true));
} else {
// Anonymous enum — glue.
glue.push((s, e));
}
}
"concept_definition" => {
// C++20. Has required "name" field (identifier).
if let Some(nn) = child.child_by_field_name("name") {
let name = &source[nn.start_byte()..nn.end_byte()];
flush_glue(glue, units);
let sym = build_symbol(prefix, &[name]);
units.push((sym, s, e, true));
} else {
glue.push((s, e));
}
}
"linkage_specification" => {
// extern "C" { ... } — glue-wrapper, but recurse into body
// with same prefix so inner definitions are extracted.
let body = child.child_by_field_name("body").unwrap_or(child);
// The linkage_spec itself is glue; inner defs handled by recursion.
// Don't emit the wrapper as a unit; but also don't push it as glue
// since recursion will push its inner children individually.
build_blocks(body, source, prefix, units, glue);
}
// Everything else: preproc, declarations, using, typedef, etc.
_ => {
glue.push((s, e));
}
}
}
}
/// Join prefix + extras into a `::` separated symbol.
fn build_symbol(prefix: &[String], extras: &[&str]) -> String {
let mut parts: Vec<&str> = prefix.iter().map(String::as_str).collect();
parts.extend_from_slice(extras);
parts.join("::")
}
/// Extract the symbol for a `function_definition` given its top-level
/// `declarator` node. Returns `None` if the name cannot be determined.
///
/// The declarator chain may be:
/// - `function_declarator` (plain fn or method)
/// - `pointer_declarator` wrapping `function_declarator` (fn returning pointer)
/// - `reference_declarator` wrapping `function_declarator` (fn returning ref)
/// - `operator_cast` (conversion operator — e.g. `operator bool`)
///
/// The inner `function_declarator.declarator` is one of:
/// - `identifier` → free fn or constructor, symbol = `prefix::name`
/// - `field_identifier` → method in class body, symbol = `prefix::name`
/// - `destructor_name` → `~Foo`, symbol = `prefix::~Foo`
/// - `operator_name` → `operator+` etc., symbol = `prefix::operator+`
/// - `qualified_identifier` → out-of-class def `Foo::bar` or `ns::Foo::bar`;
/// the scope chain is extracted and prepended to prefix.
///
/// For `qualified_identifier`, the scope hierarchy (which may itself be a
/// `qualified_identifier`) is flattened into a list of segments. These
/// segments REPLACE the current prefix (since out-of-class defs carry their
/// full scope explicitly). Example: `void ns::Foo::bar() {}` at top level
/// with prefix=[] → segments=[ns, Foo, bar] → symbol = `ns::Foo::bar`.
fn extract_fn_symbol(
decl_node: tree_sitter::Node,
source: &str,
prefix: &[String],
) -> Option<String> {
// Walk down pointer/reference wrapper layers to reach the
// function_declarator (or operator_cast at definition level).
let fn_decl = unwrap_to_fn_declarator(decl_node, source)?;
match fn_decl.kind() {
"operator_cast" => {
// e.g. `operator bool() const` — the function_definition.declarator
// IS the operator_cast (no function_declarator wrapper).
// Symbol = `prefix::operator <type>`.
let type_node = fn_decl.child_by_field_name("type")?;
let type_text = &source[type_node.start_byte()..type_node.end_byte()];
Some(build_symbol(prefix, &[&format!("operator {type_text}")]))
}
"function_declarator" => {
let inner = fn_decl.child_by_field_name("declarator")?;
extract_name_node(inner, source, prefix)
}
_ => None,
}
}
/// Walk pointer_declarator / reference_declarator chains down to the
/// first `function_declarator` or `operator_cast` node.
///
/// Returns `None` if no such node is found (e.g. a function definition
/// whose declarator is malformed or unknown).
fn unwrap_to_fn_declarator<'a>(
mut node: tree_sitter::Node<'a>,
_source: &str,
) -> Option<tree_sitter::Node<'a>> {
loop {
match node.kind() {
"function_declarator" | "operator_cast" => return Some(node),
"pointer_declarator" => {
node = node.child_by_field_name("declarator")?;
}
"reference_declarator" | "rvalue_reference_declarator" => {
// reference_declarator has no `declarator` field; its child
// is in the unnamed children list.
let mut walker = node.walk();
node = node.named_children(&mut walker).next()?;
}
_ => return None,
}
}
}
/// Given the innermost name node of a function_declarator, produce the symbol.
fn extract_name_node(
inner: tree_sitter::Node,
source: &str,
prefix: &[String],
) -> Option<String> {
match inner.kind() {
"identifier" | "field_identifier" => {
let name = &source[inner.start_byte()..inner.end_byte()];
Some(build_symbol(prefix, &[name]))
}
"destructor_name" => {
// destructor_name text includes the `~` prefix (e.g. "~Foo").
let full = &source[inner.start_byte()..inner.end_byte()];
Some(build_symbol(prefix, &[full]))
}
"operator_name" => {
// Full text e.g. "operator+", "operator->", "operator()".
let full = &source[inner.start_byte()..inner.end_byte()];
Some(build_symbol(prefix, &[full]))
}
"template_function" | "template_method" => {
// Template function like `foo<int>()`. Use the `name` field
// (the identifier / field_identifier before `<`).
let name_node = inner.child_by_field_name("name")?;
let name = &source[name_node.start_byte()..name_node.end_byte()];
Some(build_symbol(prefix, &[name]))
}
"qualified_identifier" => {
// Out-of-class method definition. Flatten the nested
// qualified_identifier chain into ordered segments.
// Example: `ns::Foo::method`
// qualified_identifier {
// scope: namespace_identifier "ns"
// name: qualified_identifier {
// scope: namespace_identifier "Foo"
// name: identifier "method"
// }
// }
// → ["ns", "Foo", "method"]
//
// These segments are combined with the current prefix so that a
// top-level out-of-class def `void Foo::bar() {}` inside a
// namespace body with prefix=["ns"] produces `ns::Foo::bar`.
let mut segments: Vec<String> = Vec::new();
flatten_qualified_id(inner, source, &mut segments);
if segments.is_empty() {
return None;
}
// Build: prefix + all segments (scope chain + leaf).
let mut all: Vec<&str> = prefix.iter().map(String::as_str).collect();
for seg in &segments {
all.push(seg.as_str());
}
Some(all.join("::"))
}
_ => None,
}
}
/// Recursively flatten a `qualified_identifier` node into ordered string
/// segments. For `ns::Foo::method` this produces `["ns", "Foo", "method"]`.
fn flatten_qualified_id(node: tree_sitter::Node, source: &str, out: &mut Vec<String>) {
// A qualified_identifier has:
// scope: namespace_identifier | (None for global-scope `::foo`)
// name: identifier | field_identifier | destructor_name |
// operator_name | qualified_identifier | template_function |
// template_method | ...
let scope_node = node.child_by_field_name("scope");
let name_node = node.child_by_field_name("name");
if let Some(s) = scope_node {
out.push(source[s.start_byte()..s.end_byte()].to_string());
}
match name_node {
Some(n) if n.kind() == "qualified_identifier" => {
// Recurse: more nesting.
flatten_qualified_id(n, source, out);
}
Some(n) => {
// Leaf name — push its text.
out.push(source[n.start_byte()..n.end_byte()].to_string());
}
None => {}
}
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
pub(crate) mod tests_support {
use kebab_core::*;
use std::path::PathBuf;
use time::OffsetDateTime;
pub fn fixed_code_asset(workspace_path: &str, lang: &str) -> RawAsset {
RawAsset {
asset_id: AssetId("a".repeat(64)),
source_uri: SourceUri::File(PathBuf::from(workspace_path)),
workspace_path: WorkspacePath(workspace_path.to_string()),
media_type: MediaType::Code(lang.to_string()),
byte_len: 0,
checksum: Checksum("b".repeat(64)),
discovered_at: OffsetDateTime::from_unix_timestamp(1_700_000_000).unwrap(),
stored: AssetStorage::Reference {
path: PathBuf::from(workspace_path),
sha: Checksum("b".repeat(64)),
},
}
}
pub fn extract_cpp(src: &str, path: &str) -> kebab_core::CanonicalDocument {
use super::CppAstExtractor;
use kebab_core::Extractor;
let asset = fixed_code_asset(path, "cpp");
let cfg = ExtractConfig::default();
let root = PathBuf::from("/tmp");
let ctx = ExtractContext {
asset: &asset,
workspace_root: &root,
config: &cfg,
};
CppAstExtractor::new().extract(&ctx, src.as_bytes()).unwrap()
}
}
#[cfg(test)]
mod tests {
use super::*;
use kebab_core::{Block, MediaType, SourceSpan};
fn syms(doc: &kebab_core::CanonicalDocument) -> Vec<String> {
let mut s: Vec<String> = doc
.blocks
.iter()
.filter_map(|b| match b {
Block::Code(c) => match &c.common.source_span {
SourceSpan::Code { symbol, .. } => symbol.clone(),
_ => None,
},
_ => None,
})
.collect();
s.sort();
s
}
#[test]
fn extractor_supports_only_media_code_cpp() {
let e = CppAstExtractor::new();
assert!(e.supports(&MediaType::Code("cpp".into())));
assert!(!e.supports(&MediaType::Code("c".into())));
assert!(!e.supports(&MediaType::Code("rust".into())));
assert!(!e.supports(&MediaType::Markdown));
}
#[test]
fn free_function() {
let src = "void foo() {}\n";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "foo"), "got {s:?}");
}
#[test]
fn namespace_and_class() {
let src = r"
namespace ns {
class Foo {
public:
void method() {}
Foo() {}
~Foo() {}
int operator+(const Foo& o) { return 0; }
};
}
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "ns::Foo"), "ns::Foo missing: {s:?}");
assert!(s.iter().any(|x| x == "ns::Foo::method"), "method missing: {s:?}");
assert!(s.iter().any(|x| x == "ns::Foo::Foo"), "ctor missing: {s:?}");
assert!(s.iter().any(|x| x == "ns::Foo::~Foo"), "dtor missing: {s:?}");
assert!(s.iter().any(|x| x == "ns::Foo::operator+"), "op+ missing: {s:?}");
}
#[test]
fn anonymous_namespace() {
let src = r"
namespace {
void hidden_fn() {}
}
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "<anonymous>::hidden_fn"),
"anon fn missing: {s:?}"
);
}
#[test]
fn nested_namespace_specifier() {
let src = r"
namespace outer::inner {
void fn_in_nested() {}
}
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "outer::inner::fn_in_nested"),
"nested ns fn missing: {s:?}"
);
}
#[test]
fn out_of_class_method_def() {
let src = r"
void ns::Foo::method() { }
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "ns::Foo::method"),
"out-of-class method missing: {s:?}"
);
}
#[test]
fn template_declaration() {
let src = r"
template<typename T>
class Bar {
void tmpl_method() {}
};
template<typename T>
void tmpl_free_fn(T x) {}
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "Bar"), "Bar class missing: {s:?}");
assert!(
s.iter().any(|x| x == "Bar::tmpl_method"),
"Bar::tmpl_method missing: {s:?}"
);
assert!(
s.iter().any(|x| x == "tmpl_free_fn"),
"tmpl_free_fn missing: {s:?}"
);
}
#[test]
fn enum_and_concept() {
let src = r"
enum class Color { Red, Green };
template<typename T>
concept Printable = requires(T t) { t.print(); };
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "Color"), "Color missing: {s:?}");
assert!(s.iter().any(|x| x == "Printable"), "Printable missing: {s:?}");
}
#[test]
fn extern_c_block() {
let src = r#"
extern "C" {
void c_fn1() {}
void c_fn2() {}
}
"#;
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "c_fn1"), "c_fn1 missing: {s:?}");
assert!(s.iter().any(|x| x == "c_fn2"), "c_fn2 missing: {s:?}");
}
#[test]
fn conversion_operator() {
let src = r"
class Foo {
operator bool() const { return true; }
};
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "Foo::operator bool"),
"conversion op missing: {s:?}"
);
}
#[test]
fn empty_file_produces_module() {
let src = "";
let doc = tests_support::extract_cpp(src, "x/empty.cpp");
let s = syms(&doc);
assert_eq!(s, vec!["<module>"], "expected <module>: got {s:?}");
}
#[test]
fn glue_only_produces_module() {
let src = "#include <vector>\nusing namespace std;\n";
let doc = tests_support::extract_cpp(src, "x/glue.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "<module>"), "expected <module>: got {s:?}");
}
#[test]
fn ptr_returning_function() {
let src = "int* ptr_fn(int x) { return &x; }\n";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(s.iter().any(|x| x == "ptr_fn"), "ptr_fn missing: {s:?}");
}
#[test]
fn ref_returning_operator() {
let src = r"
class Foo {
Foo& operator=(const Foo& o) { return *this; }
};
";
let doc = tests_support::extract_cpp(src, "x/foo.cpp");
let s = syms(&doc);
assert!(
s.iter().any(|x| x == "Foo::operator="),
"operator= missing: {s:?}"
);
}
#[test]
fn deterministic_across_runs() {
let src = r"
namespace ns {
class Foo {
void method() {}
};
}
void free_fn() {}
";
let a = tests_support::extract_cpp(src, "x/foo.cpp");
for _ in 0..20 {
assert_eq!(tests_support::extract_cpp(src, "x/foo.cpp").blocks, a.blocks);
}
}
}

Some files were not shown because too many files have changed in this diff Show More