markdown ingest arm 이 그동안 유일하게 `App::extract_for` extractor
registry 를 우회하고 `kebab_parse_md::{parse_frontmatter, parse_blocks,
build_canonical_document}` free function 을 직접 호출했다 ("the single
biggest asymmetry"). 이를 image/pdf/code 와 동일하게 registry 경유로
통일.
- `kebab-parse-md` 에 `MarkdownExtractor` 신설 — 기존 free function 3종을
동일 순서·동일 인자로 감싸 `bytes → CanonicalDocument` 생산만 담당.
fm_span_end / count_lines_in / build_body_hints 헬퍼도 함께 이식.
- `App.extractors` registry 에 등록 (11 → 12 entry), markdown 이 `supports`
로 발견되도록 첫 entry 로 배치.
- `ExtractContext` 에 `source_id` / `source_trust` 필드 추가 — markdown
frontmatter 가 per-source trust 기본값을 override 하고 그 precedence 가
`parse_frontmatter` *내부*에서 결정되므로 ctx 가 carry 해야 함. 다른
extractor 는 None (post-extract 에서 source_id stamp 유지).
- 핸들러는 추출 stage 만 registry 로 이전 — version stamping / chunking /
embedding / store 는 그대로. IngestItem.warnings 는 pdf/code 처럼
`canonical.provenance` 의 Warning 이벤트에서 도출.
byte-identical 검증: parity gate-ingest (all-markdown 183 doc / 7676 chunk)
CHUNKS / SEARCH / ASK 모두 IDENTICAL. clippy 0, kebab-app + kebab-parse-md
test 전체 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Mc6W1fgsrbFKTsqA6P8La
kebab-config: EmbeddingModelCfg.endpoint: Option<String>(serde default, ollama용,
None→models.llm.endpoint 폴백) + provider 문서에 ollama + env
KEBAB_MODELS_EMBEDDING_ENDPOINT. kebab-app embedder(): provider match 에 ollama
분기(facade 경유). workspace member += kebab-embed-ollama, app dep 추가.
version 0.25.0 → 0.26.0(minor, +Cargo.lock) — 신규 임베딩 백엔드/모델은 CLAUDE.md
§Release 의 surface 변경 트리거.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
V009 의 FTS5 tokenizer 가 trigram → unicode61 + 한국어 형태소 분해
column 로 갱신됨. lexical_index_version 의 format 에
`fts5-v009-korean-morphological` suffix 추가하여 V007 baseline 과
구별. eval runner 의 config_snapshot 및 search cache 무효화에
자동 picks up.
기존 format: lex:{chunker_version}
신규 format: lex:{chunker_version}:fts5-v009-korean-morphological
Wire schema shape 변경 없음 (SearchHit.index_version 의 string
content 만 변화). lexical_index_version_is_returned_unchanged test
는 IndexVersion 의 임의 string 을 사용해 unchanged.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §11.1, §11.3
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S6)
V009 unicode61 + 형태소 tokenizer 환경에서 2-char 한국어 query 가
hit 가능해졌으므로 V007 시기의 "3자 이상 권장" hint 가 obsolete.
SearchResponse.hint field 는 wire schema 보존 위해 struct 에 유지 +
항상 None.
- kebab-app/src/app.rs: short_query_hint 함수 + doc-comment 삭제.
2 호출 site 가 hint = None 으로 정리.
- kebab-app/src/lib.rs: re-export 에서 short_query_hint 제거.
- kebab-tui/{app.rs,search.rs,run.rs}: short_query_hint field + 4
호출 cascade 제거.
- kebab-cli/tests/wire_search_response.rs:
search_plain_emits_short_query_hint_to_stderr test 삭제.
search_json_emits_hint_field_for_short_query →
search_json_hint_absent_for_short_query_v009 으로 교체
(hint 항상 None 검증).
- kebab-search/src/lexical.rs::build_match_string: V007 의 trigram
multi-token OR-combine 분기는 V009 환경에서 redundant 하나 보존
(future 확장성) — doc-comment 1 줄 추가.
Wire schema shape 변경 없음 (search_response.schema.json:33 의 hint
field 보존, struct 에 None 으로 항상 셋팅).
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §7.2, §7.3, §11.3
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S5)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V007 → V009 업그레이드 시 기존 chunks 의 tokenized_korean_text 가
NULL — 첫 App::open_with_config 호출 시 자동으로 lindera ko-dic
으로 분해 후 UPDATE. chunks_au trigger 가 chunks_fts 를 자동 재-index.
사용자 재-ingest 불필요.
- crates/kebab-store-sqlite/src/store.rs:
backfill_tokenized_korean_text(progress_cb, tokenize) API. 1000 row 마다
commit + progress 콜백. idempotent (IS NULL 필터로 partial
completion 재실행 안전). tokenizer 를 파라미터로 받아 §8 dep 경계 유지.
- crates/kebab-app/src/app.rs::open_with_config: run_migrations 직후
backfill 호출. 실패 시 warn log 만 (App open 은 성공 — vector/hybrid
mode 계속 가능). 500 row 마다 info log progress.
- crates/kebab-store-sqlite/tests/fts.rs:
backfill_tokenized_korean_text_populates_nullable_rows 단위 test
(idempotency 포함).
- clippy pre-existing 오류 수정 (redundant_closure, map_unwrap_or,
cast_lossless, uninlined_format_args — kebab-app/ingest_log.rs,
pdf_ocr_apply.rs, app.rs, tests/ocr_inspect_smoke.rs).
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §8.1, §8.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S4)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two new wire schemas land as additive minor: ocr_stats.v1 (corpus-wide
aggregate — total_events, success_rate, p50/p90/p99/max_ms, by_engine,
top-10 by_doc by failure count) and ocr_failures.v1 (per-doc or
corpus-wide recent failures, with --doc-id + --limit). Both ship via
new CLI subcommands `kebab inspect ocr-stats` / `inspect ocr-failures`.
App gains four facade methods: inspect_ocr_stats /
inspect_ocr_failures plus their *_with_config companions — required by
CLAUDE.md "the facade rule" so `--config <path>` is honored. The CLI
dispatch arms thread cfg explicitly into the _with_config form.
Runtime introspection emit (WIRE_SCHEMAS in schema.rs) gains two
entries; the meta JSON Schema (schema.schema.json) is untouched
because its wire.schemas is pattern-based, not enum-based.
ingest_log::percentiles extended to (p50, p90, p99, max). p99 surfaces
only via inspect ocr-stats; IngestSummary (round 1) stays 3-percentile.
SKILL.md synced with the two new schemas (AC-13).
Closure r2 G2 (facade *_with_config pair) + G3 (runtime emit, not
meta schema file) + closure r1 F4 (p99) resolved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
PR #140 회차 1 actionable 7건 반영:
- docs/SMOKE.md: parser_version "code-rust-ast-v1" → "code-rust-v1" (chunker_version 과 혼동); jq path .citation.code_lang → .citation.lang (wire 의 code_lang 은 SearchHit top-level)
- docs/ARCHITECTURE.md: Mermaid pcode→ptypes 잘못된 edge → pcode→core 로 정정 (kebab-parse-code Cargo.toml 실제 dep 와 일치); 디렉토리 트리에서 code-rust-ast-v1 chunker 표기 위치 kebab-parse-code → kebab-chunk 로 정정
- crates/kebab-app/src/app.rs: backfill_repo 의 .ok().flatten() 실패 silent swallow → tracing::warn 로 관측 가능, 비-abort 의도 보존
- crates/kebab-parse-code/src/rust.rs: impl_item arm 의 "function_item 만 unit 생성" 1A scope 한정 주석을 외부에서도 보이도록 arm 상단에 한 줄 추가 (내부 주석은 유지)
verify: kebab-parse-code 7/7 / kebab-app --lib 51/51 / code_ingest_smoke 3/3 green; touched-crate clippy clean (재부팅 전 검증).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `MediaType::Code("rust")` dispatch arm in `ingest_one_asset`,
`ingest_one_code_asset` fn (faithful mirror of `ingest_one_pdf_asset`),
and `backfill_code_lang` post-processing in `App::search_uncached`.
Integration test `code_ingest_smoke.rs` verifies full pipeline:
ingest `.rs` → Citation::Code hit with lang/symbol/line_start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Also fixes App::search_with_opts trace branch to use NoopRetriever
for SearchMode::Lexical, removing the embeddings requirement when
the user only wants lexical-mode trace.
Adds the `trace: Option<SearchTrace>` field to `SearchResponse` and
threads `SearchOpts.trace` through `App::search_with_opts`. When the
caller sets `opts.trace = true` the path bypasses the LRU search cache
and runs through `HybridRetriever::search_with_trace`, which dispatches
all 3 SearchModes internally; this means `--trace` requires embeddings
(same constraint as `--mode hybrid`). The non-trace path keeps its
exact prior behavior with `trace: None` stamped on the response.
Picked up Task 1 / Task 3 follow-ups in the same commit so the
workspace compiles: SearchOpts struct-literals in kebab-cli/main.rs +
kebab-mcp/tools/search.rs default the new `trace` field to false, and
the schema-wrapper test in kebab-cli/wire.rs fills the new
media_breakdown / lang_breakdown / index_bytes / stale_doc_count fields
on Stats with `Default::default()`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous round-1 fix dropped the speculative cursor branch on
the truncated path, leaving a contradiction with the docs:
- snippet-only shrunk → cursor emitted (returned == k_effective)
- k-popped → cursor null (returned < k_effective)
But docs promised the opposite.
R2 resolution: emit cursor whenever more hits may be reachable
(either retriever filled the page OR budget popped hits — the
popped ones remain fetchable from offset+returned). Drop the
artificial "widen vs paginate" copy; truncated and next_cursor
are now independent signals — caller may do either or both.
Updates: app.rs::search_with_opts logic + SearchResponse doc +
schema description + SKILL.md two bullets + max_tokens=0 test
asserts cursor IS emitted on k-pop case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- error_wire: StructuredError wrapper preserves ErrorV1 through
anyhow → classify pipeline. Adds downcast short-circuit so
cursor::decode's typed code = "stale_cursor" reaches the wire
instead of being string-formatted to code = "generic".
- app: search_with_opts now wraps cursor::decode error in
StructuredError instead of anyhow! string format.
- test: error_wire pins both negative (bare anyhow → not
stale_cursor) AND positive (StructuredError → stale_cursor)
invariants. CLI integration test runs end-to-end and asserts
error.v1.code on stderr.
- app: next_cursor only emitted on full-page (k-pop) path; drop
speculative emit on snippet-only truncation that would point at
a different page than the agent expected.
- cursor: differentiate malformed-base64 / malformed-payload /
revision-mismatch error messages; all keep code = stale_cursor.
- test: cursor_rejected fixture uses .expect() to fail loud on
cursor non-emission instead of silent skip.
- test: max_tokens=0 → 1-hit floor + truncated=true.
- docs: SKILL.md + schema description distinguish snippet-shrink
(widen) vs k-pop (paginate) truncated cases. HOTFIXES notes
--no-cache semantic shift (cached path + clear vs uncached path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Budget loop: snippet shorten → k pop → ≥1 hit floor. Cursor
encode/decode threads corpus_revision; mismatch surfaces as
stale_cursor anyhow error. App::search retained as thin wrapper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compute_stale: strict > boundary, threshold=0 disables, future
timestamps treated as fresh (clock skew safety). App::search
re-stamps on cache hit so config threshold changes take effect
without flushing the cache.
Also unblocks the workspace build by plugging placeholder
indexed_at/stale into the two AnswerCitation construction
sites in kebab-rag/pipeline.rs (the score-gate refusal path
forwards from SearchHit; the LLM-citation path uses
UNIX_EPOCH/false until Task 7 wires the real values through
pack_context).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `App::build_retriever(mode) -> Result<Arc<dyn Retriever>>` 추출.
`ask` 와 `ask_with_session` 모두 사용. 35+ 줄 retriever stack 중복
제거 — 미래 retriever 변경이 한 곳만.
- V005 migration `chat_sessions.sql` 의 `citations_json` doc 수정:
`Vec<Citation>` → `Vec<AnswerCitation>` (실제 stored type 과 일치).
AnswerCitation 가 marker + Citation 등 포함하므로 deserialize 시
type mismatch 회피.
15 app lib + 9 store chat_sessions + clippy 통과.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
도그푸딩 item 14 — CLI 에서도 multi-turn 가능하도록 `kebab ask
--session <id>` 추가. p9-fb-17 의 ChatSessionRepo 위에 build, 첫 호출
세션 자동 생성, 이후 호출이 prior turns 를 history 로 받아 follow-up.
external AI integration (Claude Code skill / MCP) 도 같은 facade 로
stateful 대화 가능.
## 핵심 변경
- **`App::ask_with_session(session_id, query, opts) -> Answer`** —
load session header → list_turns 로 prior history → 빌드 retriever
stack (lexical / vector / hybrid 같은 분기) → `RagPipeline::ask_
with_history` 호출 → 첫 호출이면 `chat_sessions` row 자동 생성
(title = first_question_title) → `chat_turns` 새 row append.
- **`App::first_question_title(question)`** helper — `trim() + nfc()
+ 40 chars cap`, fallback `"untitled"`. unicode-normalization
workspace dep 재사용.
- **`App::blake3_truncate(input)`** helper — `blake3(session_id ||
":" || turn_index)` 의 첫 16 byte 를 u128 으로, format!{:032x} 로
32-hex `turn_id`.
- **`ask_with_session_with_config`** facade — CLI 진입점.
- **CLI `--session <id>` flag** — `Cmd::Ask` 의 `session: Option<
String>` field, handler 가 None 이면 `ask_with_config` (기존
단발), Some(id) 면 `ask_with_session_with_config` 호출.
- **에러 정책**: session create / turn append 실패 시 warn 로그
남기고 answer 는 그대로 반환 — 사용자가 답변 받은 컴퓨트를 잃지
않음. 영속성 실패가 답변 응답을 가로막지 않는 conservative shape.
## 테스트
- `App::first_question_title` 3 unit (trim + cap, empty → untitled,
korean NFD → NFC)
- `App::blake3_truncate` 1 unit (deterministic + distinct across
varying session/index)
- 워크스페이스 전체 `cargo test --workspace --no-fail-fast -j 1` exit 0
- `cargo clippy --workspace --all-targets -- -D warnings` clean
## 문서
- README `kebab ask` 행: `--session` 안내 + chat_sessions 자동 생성
+ `kebab reset --data-only` wipe 안내
- README **외부 AI 통합** 절: Claude Code skill 이 `--session` 으로
multi-turn 가능하다는 한 문장 추가
- HANDOFF entry
- spec status planned → in_progress
## Out of scope (spec deviation)
- `--repl` (stdin loop) — spec 명시되어 있으나 stdin fixture 부담
으로 deferral. 별도 후속 task 또는 `--session` 사용자 경험 회신
후 결정.
- session list / show / delete 관리 명령 (spec 의 Out of scope).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `App::search` 의 두 cache.lock() 호출이 mutex poison 시 silently
bypass 하던 것을 `unwrap_or_else(|e| { warn!; e.into_inner() })`
recovery 로 교체. cache 가 poison 됐어도 다음 호출은 정상이고
한 번은 warn 로그가 남아 panic 흔적 추적 가능. lookup 후 lock
drop → retriever 호출 → 재 lock 으로 lock granularity 도 짧게.
- `clear_search_cache` 도 같은 recovery 패턴.
- `SearchCacheKey` doc 에 spec 와 impl 의 naming 차이 (index_version
vs corpus_revision) 명시 + HOTFIXES entry 추가. spec 의 index_
version 명칭이 design §9 의 기존 `IndexVersion` newtype (embedding
-index identity 라벨) 과 충돌해서 corpus_revision 으로 rename.
7 tests/search_lexical 통과. clippy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
도그푸딩 item 15 — TUI / 같은 process 안에서 동일 query 반복 시 SQLite
FTS + Lance + RRF 재계산이 매번 발생하던 비용 해소. in-process LRU
캐시 + 모노토닉 corpus_revision 카운터로 ingest commit 발생 시 모든
entry 자동 stale.
## 핵심 변경
- **SQLite V004 migration**: `kv (key TEXT PRIMARY KEY, value TEXT)
STRICT` + `corpus_revision = '0'` seed. 미래의 다른 scalar 도 같은
테이블에 들어갈 수 있는 generic shape.
- **`SqliteStore::corpus_revision()` / `bump_corpus_revision()`** —
`UPDATE ... CAST AS INTEGER + 1` atomic. INSERT-OR-IGNORE 도 함께
실행 (V004 seed 가 무슨 이유로 누락된 케이스 paranoid).
- **`kebab-app::ingest_with_config_cancellable`** — `new + updated > 0`
시 bump, no-op (skipped-only) reingest 는 cache 보존.
- **`App.search_cache: Option<Mutex<LruCache<SearchCacheKey, Vec<
SearchHit>>>>`** — `config.search.cache_capacity` (default 256, 0
비활성). `lru = "0.12"` workspace dep 추가.
- **`SearchCacheKey`** = `query_norm` (NFKC + trim + lowercase) +
`mode` + `k` + `snippet_chars` + `embedding_version` (vector/hybrid
만, lexical 은 빈 문자열) + `chunker_version` + `corpus_revision`
snapshot.
- **`App::search`** rewrite — cache 활성 시 lookup → miss 면 기존
`search_uncached` 호출 후 put. cache 비활성이거나 lock 실패면
straight-line.
- **`App::search_uncached`** (rename of pre-fb-19 `search` body) +
`search_uncached_with_config` facade — CLI `kebab search --no-cache`
로 진입.
- **`Config.search.cache_capacity: usize`** field, `#[serde(default)]`
로 기존 config 호환.
- **CLI `--no-cache`** flag — 디버깅용 (CLI 는 매 호출이 새 process
라 사실상 no-op 이지만 spec 명시 + 향후 long-lived process 호환).
- **frozen design §9 versioning** 표에 `corpus_revision` row 추가
(기존 `index_version` 라벨과 다른 차원: 라벨은 retrieval 형상,
corpus_revision 은 ingest commit ack).
## 테스트
- `kebab-store-sqlite` 신규 3 unit (fresh=0, monotonic bump, persist
across reopen)
- `kebab-app` 신규 4 integration (cached repeat 같은 hits, NFKC 정규화
로 case/whitespace collapse, --no-cache parity, first ingest bumps
corpus_revision)
- 워크스페이스 전체 `cargo test --workspace --no-fail-fast -j 1` exit 0
- `cargo clippy --workspace --all-targets -- -D warnings` clean
## 문서
- README `kebab search` 행: 캐시 동작 + `--no-cache` 안내 + corpus_
revision 무효화 메커니즘
- docs/SMOKE.md `[search]` 절에 `cache_capacity` 라인 추가
- HANDOFF: 2026-05-03 entry
- spec status planned → in_progress
## Out of scope
- patch-and-merge incremental (RRF 정규화 전체 hit set 기준이라 어려움)
- SQLite 영속 cache (P+)
- 다른 process 간 cache 공유 (in-process 만 — corpus_revision 이
cross-process 무효화는 O(1))
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
