Files deleted from disk (rm a.md) were leaving stale documents + chunks +
embeddings in the store, surfacing as ghost citations in search/ask.
Existing purge_orphan_at_workspace_path only handled content-changed
stale (WHERE workspace_path=? AND asset_id != ?) — file deletion has no
new asset_id.
Fix: post-walker-scan sweep. Compute (stored_paths - scanned_paths),
for each candidate check filesystem existence — only purge when the
file is TRULY missing. Scope-narrowing case (file on disk but outside
include glob) is explicitly NOT purged to protect users from accidental
data loss via config edits.
Adds:
- DocumentStore::all_workspace_paths trait method + SqliteStore impl
- purge_deleted_workspace_path in store-sqlite (returns chunk_ids for
vector delete; deletes doc CASCADE + asset row + copied storage file)
- sweep_deleted_files in kebab-app::ingest path; called once per ingest
before the per-asset loop
- IngestReport.purged_deleted_files counter (additive, serde default)
- CLI ingest summary mentions purge count when > 0
- 2 integration tests: file_deletion_auto_purge + include_scope_narrowing_does_NOT_purge
dogfood discovery (PR #142 1B + multi-root: kebab-docs + httpx + zod
+ lodash). Per user decision: only filesystem deletion auto-purges;
scope narrowing requires explicit kebab reset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dogfood-discovered routing additions (PR #147) land:
- .mts / .cts → MediaType::Code(typescript)
- .mdx → MediaType::Markdown
minor bump 사유: 사용자 도그푸딩 surface 확장 — 이전에 skip 되던 28+ 파일이
이제 색인됨. design §10.4 dogfooding-ready surface 확장 = minor trigger.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dogfood-discovered fix (PR #146) lands: idempotent re-ingest now correctly
returns Unchanged for twin files (identical content at different paths)
via document-centric try_skip_unchanged lookup.
patch bump 사유: advertised idempotency 의 정상 동작 복원. 새 wire / config / surface 변경 없음.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round 1 review found the function-level doc-comment still described the
old asset-side algorithm (item 2 asset-row checksum, item 3 id_for_doc
miss). Updated to the document-centric flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Identical-content files at different workspace paths share one assets row
(assets.asset_id = blake3 content hash, PRIMARY KEY). The UPSERT
`ON CONFLICT(asset_id) DO UPDATE SET workspace_path = excluded` made
twin files overwrite each other's workspace_path on every ingest, so
`get_asset_by_workspace_path(path1)` returned the OTHER twin's row (or
None) — break idempotent unchanged-detection for both files.
Fix: switch try_skip_unchanged to document-centric lookup. `documents.
workspace_path` is already UNIQUE (V001) and `id_for_doc(path, ...)`
includes path, so each twin has its own stable document row. Compare
`doc.source_asset_id` with the new asset's checksum instead of going
through the assets table.
Dogfood (multi-root: kebab-docs + httpx + zod + lodash) showed 27 of
726 docs marked Updated on every idempotent re-ingest — all 27 are
twin-file victims (empty `__init__.py` ×3, AGENTS.md ↔ CLAUDE.md
same content, duplicate logo PDFs/JPGs).
After: re-ingest reports 0 new / 0 updated / 726 unchanged.
No schema migration needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dogfood-discovered fixes (PR #145) land in production:
- schema.v1.repo_breakdown 가 실제로 채워짐 (이전: 항상 빈 BTreeMap)
- workspace.include glob 가 walker 에서 enforce 됨 (이전: 완전 무시)
patch bump 사유: 둘 다 advertised surface 의 정상 동작 복원.
새 wire / config / surface 변경 없음.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
config.workspace.include was completely ignored by the walker — connector.rs
log_scope_include_warning literally said "handled by extractor router" but
no extractor router exists. Dogfooding (PR #142 1B + multi-root corpus
kebab-docs + httpx + zod + lodash) showed user-set include of code+md still
ingested 84 .png + 8 .pdf files.
Fix: walker treats scope.include as an allow-list — empty Vec preserves
backward-compat (all files pass), non-empty requires file path to match at
least one pattern (AND with the existing exclude rules). Removed the
misleading debug log.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dogfooding (PR #142 1B + multi-root corpus: kebab-docs + httpx + zod + lodash)
revealed schema.v1.repo_breakdown is always {} despite the 1A-2 Task 9
having added the code_lang_breakdown sibling. The schema.rs:171 placeholder
`BTreeMap::new()` was left in place. Mirror Task 9's code_lang_breakdown
query for the repo field — same metadata_json JSON-path pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dogfood-discovered code_lang/repo filter bug (PR #144) fix lands in
production. patch bump because:
- 1A-1 advertised CLI flags --code-lang / --repo were live but inert
(SearchFilters fields propagated but never applied to retriever SQL)
- fix restores intended behavior; no new wire surface
- user has dogfooded against httpx + zod + lodash and re-validating
needs the fixed binary
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
p10-1A-1 (PR #139) added SearchFilters.code_lang + .repo fields and the CLI
--code-lang / --repo flags propagate them correctly into SearchFilters, but
neither the lexical retriever's FTS SQL nor the shared filter_chunks helper
(used by the vector retriever) ever applied them — so a code-lang-filtered
search returned all-doc hits (markdown / pdf / code mixed).
Discovered while dogfooding p10-1B with httpx + zod + lodash clones:
`kebab search 'AsyncClient' --code-lang python --json` returned markdown
hits from httpx/docs/ first.
Fix: add IN-list filters on json_extract(d.metadata_json, '$.code_lang')
and '$.repo' to both lexical.rs and filters.rs, mirroring the existing
media filter pattern. Two regression tests added in each crate covering
the new filter behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #142 round-1 fix commit 4503b5b 보고에는 lang.rs 의 (a) module_path_for_python
doc comment 갱신 (tests/examples/benches 가 의도적으로 strip 안 됨 명시) 과
(b) tests/test_foo.py → tests.test_foo 단언 추가가 포함됐다고 적혔으나,
실제 commit 에는 lang.rs 변경이 staging 되지 않아 main 에 안 들어감 (review
loop round 2 이 working tree 상태만 신뢰하고 commit 검증을 안 함).
이번 PR 이 누락된 (5)+(6) 항목만 retro 적용. lang.rs +9 lines (test 1 +
doc 4 + 주석 2 + 빈줄 2). cargo test -p kebab-parse-code --lib → 20/20 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Round 1 push-back on TS/JS class-method decorator handling was based on
an inaccurate doc comment in typescript.rs that claimed decorators are
method_definition children; tree-sitter-typescript 0.23 actually places
them as class_body preceding siblings. Round 2 correctly identified the
cross-language inconsistency with Python's decorated_definition arm.
Fix: extend unit_start backward walk in typescript.rs to also accept
'decorator' siblings (three-line change + corrected doc comment).
javascript.rs is unaffected: tree-sitter-javascript stores the decorator
as a named child INSIDE method_definition, so method_definition.start_row
already covers the decorator line without any sibling walk.
Adds three regression tests:
- class_method_decorator_folded_into_method_unit (TS): asserts @Log() is
inside the emitted method unit code and line_start == 2.
- ts_class_decorator_folded_into_class_unit (TS): class-level @Injectable()
folded into the class unit, line_start == 1.
- js_class_method_decorator_already_folded_by_grammar (JS): documents
that JS already includes the decorator via grammar semantics.
verify: per-crate cargo test (20 passed) + clippy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
(1) tasks/HOTFIXES.md: add 2026-05-20 entry for path-sanitize gap in
module_path_for_python / _tsjs (promised in task spec line 55 but
not landed in round 0). Bidirectional cross-link added.
(2) crates/kebab-parse-code: dedup filename_from_workspace_path /
strip_extension / join_symbol via new pub(crate) module scaffold.rs.
Removed 9 byte-identical fn copies across rust/python/typescript/
javascript extractors. Pure refactor — no behavior change.
(3) crates/kebab-parse-code/tests/fixtures/sample.py: @staticmethod was
semantically inappropriate on a module-level fn (class-method
decorator). Changed to @no_type_check; test assertion updated.
(5)+(6) crates/kebab-parse-code/src/lang.rs: add tests/test_foo.py case
to module_path_for_python test + doc clarifying that tests/ /
examples/ / benches/ are intentionally not stripped.
(4) PUSH BACK — TS/JS class decorator handling is design intent of 1B
1차 (typescript.rs:242-244 + HOTFIXES entry 2 already in place).
No code change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors code_rust_ast_snapshot pattern. In-memory CanonicalDocument build so
no kebab-parse-code dep (boundary §6.3 respected).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single-grammar variant of typescript.rs — JS handles .jsx via the same
LanguageFn. No interface/type/enum arms; otherwise identical mapping +
workspace-path prefix via module_path_for_tsjs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task H implementer chose code-typescript-v1 but plan + design §3.3 use the
short form (chunker is code-ts-ast-v1 / code-js-ast-v1). Aligning parser
versions to match: rust=code-rust-v1 / python=code-python-v1 / ts=code-ts-v1
/ js=code-js-v1 (Task K). Fixes 2 sites: const PARSER_VERSION + integration
test assertion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Duplicate of code-rust-ast-v1 / code-python-ast-v1 with language-agnostic body unchanged.
Cross-chunker policy_hash identity asserted vs md-heading-v1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `kebab_parse_code::typescript::TypescriptAstExtractor` (PARSER_VERSION
`code-typescript-v1`), mirroring the Python extractor (P10-1B Task E) and
the Rust scaffold (P10-1A-2). One `Block::Code` per top-level AST semantic
unit (free fn / class / each method / interface / type alias / enum,
recursively per nested class), each carrying `SourceSpan::Code` with the
unit's dotted symbol path prefixed by `module_path_for_tsjs`.
Grammar selection per `tree-sitter-typescript` 0.23: the workspace path's
`.tsx` extension routes to `LANGUAGE_TSX`, everything else to
`LANGUAGE_TYPESCRIPT`. The `export_statement` arm unwraps a `declaration`
field (`function_declaration` / `class_declaration` / `interface_declaration`
/ `type_alias_declaration` / `enum_declaration`) using the OUTER statement's
line range so `export ` is folded in; for `export default function () {}`
and `export default class {}` (where the inner node sits under the `value`
field as `function_expression` / `class` with no `name`), the symbol leaf
is `default`. Bare value exports / re-exports fall into glue.
Glue grouping reuses the Python post-pass: `<module>` only when the entire
group is imports + bare re-exports; demoted to `<top-level>` if the file
produced any real unit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Duplicate of code-rust-ast-v1 with language-agnostic body unchanged. Cross-chunker
policy_hash identity asserted vs md-heading-v1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rust path observably unchanged (verified by existing code_ingest_smoke tests).
Python/TS/JS arms bail with TODO; per-lang extractor + chunker land in subsequent tasks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`runner_lexical_is_deterministic_per_query_payload` 가 full-suite 첫 실행에서
간헐적으로 `elapsed_ms: 0` vs `elapsed_ms: 1` 차이로 깨지는 timing flake 가
있었음 (PR #140 회차 0 의 full-suite 실행에서 관찰).
원인: per_query 전체 JSON 을 byte-identical 비교하는데 QueryResult.elapsed_ms
가 timing 기반이라 µs-scale wall-clock jitter 가 그대로 비교에 들어감. 의도는
"timing 외에 byte-identical" — 인접 snapshot test #7 은 projection 으로
timing 을 명시적으로 제외하지만 #6 은 누락.
Fix: 비교 직전 양쪽 run 의 elapsed_ms 를 0 으로 normalize. 의도 그대로
표현하고 다른 field 의 결정성 검증은 보존. 50회 반복 stress 통과 (이전:
간헐 실패).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #140 회차 2 actionable 4건:
- README.md: `citation.kind = "code"` 행에서 wire 필드 구조 정정 — citation 안에는 `lang`, SearchHit top-level 에는 `code_lang`/`repo` (round 1 SMOKE 정정과 동일 클래스)
- docs/SMOKE.md: 격리 config 블록에 `extra_skip_globs = []` 추가 (P10 섹션의 "위 격리 config 블록 참조" 와 정합)
- crates/kebab-parse-code/src/rust.rs: comment-only 파일 → 0 blocks 동작을 module doc 에 한 줄 명시 (pdf-page-v1 의 "empty page produces no chunks" 패턴과 동일)
- .gitignore: `/target/` 제거 — `/target` (no trailing slash) 이 디렉토리 + 파일 + 심링크 모두 매칭하므로 `/target/` (dir 전용) 는 redundant
verify: `cargo check -p kebab-parse-code` clean (주석/문서 외 영향 없음).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #140 회차 1 actionable 7건 반영:
- docs/SMOKE.md: parser_version "code-rust-ast-v1" → "code-rust-v1" (chunker_version 과 혼동); jq path .citation.code_lang → .citation.lang (wire 의 code_lang 은 SearchHit top-level)
- docs/ARCHITECTURE.md: Mermaid pcode→ptypes 잘못된 edge → pcode→core 로 정정 (kebab-parse-code Cargo.toml 실제 dep 와 일치); 디렉토리 트리에서 code-rust-ast-v1 chunker 표기 위치 kebab-parse-code → kebab-chunk 로 정정
- crates/kebab-app/src/app.rs: backfill_repo 의 .ok().flatten() 실패 silent swallow → tracing::warn 로 관측 가능, 비-abort 의도 보존
- crates/kebab-parse-code/src/rust.rs: impl_item arm 의 "function_item 만 unit 생성" 1A scope 한정 주석을 외부에서도 보이도록 arm 상단에 한 줄 추가 (내부 주석은 유지)
verify: kebab-parse-code 7/7 / kebab-app --lib 51/51 / code_ingest_smoke 3/3 green; touched-crate clippy clean (재부팅 전 검증).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `SqliteStore::code_lang_breakdown()` that queries
`json_extract(metadata_json, '$.code_lang')`, groups by it, and
skips NULL rows — returning `BTreeMap<String, u32>`.
Wire it into `collect_stats` in `kebab-app::schema`, replacing the
`BTreeMap::new()` placeholder inserted by 1A-1.
Test: `store::tests::code_lang_breakdown_counts_by_code_lang` asserts
rust=1 and that a null-code_lang doc does NOT appear in the map.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `MediaType::Code("rust")` dispatch arm in `ingest_one_asset`,
`ingest_one_code_asset` fn (faithful mirror of `ingest_one_pdf_asset`),
and `backfill_code_lang` post-processing in `App::search_uncached`.
Integration test `code_ingest_smoke.rs` verifies full pipeline:
ingest `.rs` → Citation::Code hit with lang/symbol/line_start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
