별칭 dense 벡터({orig}#alias) hit 을 원본 chunk_id 로 strip 해 hydrate,
body+alias 중복은 첫(높은 score) 하나만 유지. overfetch 2→3 (dedup 후 k
확보). wire/RetrievalDetail 무변경. vector/hybrid 회귀 0, clippy green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- M1: chunk_aliases trigger 가드에 AND aliases <> '' (빈 문자열 미색인)
- M2: 재색인 멱등 테스트 (재-put 후 별칭 행 1개)
- N1: 본문 격리 음성 단언 (별칭 term 이 chunks_fts 로 누출 안 됨)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cmd::Eval now loads Config via cli.config (same pattern as all other
subcommands) before dispatching to the inner match. Each arm now calls
the *_with_config variant:
run_eval(&opts) → run_eval_with_config(&cfg, &opts)
compute_aggregate(run_id) → compute_aggregate_with_config(&cfg, run_id)
store_aggregate(run_id, ..) → store_aggregate_with_config(&cfg, run_id, ..)
Compare already called compare_runs_with_config but sourced cfg from
Config::load(None) — that redundant load is removed; cfg comes from
the shared binding above.
Fixes the same facade-rule regression pattern as P3-5 / P4-3: previously
`kebab --config /build/dogfood/config.toml eval run` silently evaluated
the XDG-default (empty) KB instead of the dogfood KB.
Also fixes runner.rs test that hardcoded rag-v2 after commit 5719969
bumped the default prompt_template_version to rag-v3.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SYSTEM_PROMPT_RAG_V3: 한국어 리터럴 refusal/hedge 문구를 언어 중립으로 교체.
- 근거가 부족하면 "근거가 부족하다"고 답한다. → 답변 언어로 근거가 부족함을 밝히고 [#번호] 인용 없이 답한다.
- 근거가 모호하면 "확실하지 않다" 라고 명시한다. → 근거가 모호하면 답변 언어로 불확실함을 명시한다.
MULTI_HOP_SYNTHESIZE_SYSTEM_PROMPT: 동일 패턴 두 곳 교체.
- 근거가 부족하면 "근거가 부족하다"고 답한다. → 답변 언어로 근거가 부족함을 밝히고 [#번호] 인용 없이 답한다.
- self-check 의 즉시 "근거가 부족하다" 라고만 답한다. → 즉시 답변 언어로 근거가 부족하다고만 답한다.
refusal 판정 로직(citation marker 기반)은 무변경 — 문구만 언어 중립화.
test rag_v3_contains_v2_rules_plus_language_rule: "확실하지 않다" assert → "불확실함" assert 로 갱신.
task spec grep: tasks/p9/p9-fb-15 의 rag-v2 언급은 도입 시점 historical 기술 → frozen 유지.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
#4 (사용자 요청): spec §6.2 의 Option β (sub-token 추가 emit) 를
v0.21.x P9 follow-up 에서 v0.20.1 implementation 으로 promote.
dogfood 의 ko-dic compound noun limitation (`대한민국`, `한국정부`,
`주민등록번호` 등 단일 token 정책) 해소.
Implementation (`crates/kebab-chunk/src/lib.rs::tokenize_korean_morphological`):
- 신규 helper `is_hangul()` — 한글 음절 (U+AC00..D7A3) + 자모
(U+1100..11FF, U+3130..318F) 판정.
- lindera output 의 각 morpheme 에 대해, 한글만 + 길이 ≥ 3 인 경우
sliding window 2-gram 추가 emit. `[한국정부, 한국, 국정, 정부]`
형태로 token list expand.
- 영어 / 숫자 / 혼합 token 은 supplement X (false positive 회피).
Tests (`crates/kebab-chunk/tests/tokenize_korean.rs`):
- `tokenize_korean_morphological_emits_2gram_for_long_morpheme`: 5 probe
fixture 중 supplement 발화 case 확인 (실측 `서울특별시` →
`[서울, 특별시, 특별, 별시]`, `대한민국` → `[대한민국, 대한,
한민, 민국]`).
- `tokenize_korean_morphological_no_2gram_for_english`: Rust optimization
fixture 에서 영어 substring (`Rus`, `ust`, `imi`) emit 없음 보장.
Dogfood evidence (`tasks/HOTFIXES.md` 2026-05-28 entry 보강):
- '대한', '한민', '민국' query 모두 hit (대한민국 의 sliding window).
- '특별', '주민', '등록' 같은 sub-token query hit.
- 영어 'tokenizer' query 는 corpus 부재로 0 hit (supplement X).
- Trade-off: DB size +20-30% (Korean-heavy), false positive 작은 risk.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §6.2 (Option β promote)
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (post-implementation enhancement)
S3 의 Chunk struct 갱신 (kebab-core 의 tokenized_korean_text:
Option<String> field 추가) 가 모든 chunk snapshot JSON 의 serde
serialize 결과를 변경시킴. 10 snapshot fixture (9 AST chunker +
markdown long-section) 의 baseline 을 V009 형태로 regenerate.
각 snapshot 의 변경 = chunk JSON 마다 `"tokenized_korean_text":
null` field 추가 (대부분의 fixture 가 영어 코드라 lindera 의 None
fallback). 동작 변경 없음 — serde representation 의 cascade만.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §6.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S3 follow-up via S11 sanity)
V009 morphological tokenizer 작업 (S3 chunk + S4 backfill + S5
short_query_hint 제거 + S7 신규 tests) 의 형식 정리. 동작 변경 없음.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S11)
신규 4 test 추가:
- crates/kebab-store-sqlite/tests/fts.rs:
- fts_v009_korean_morphological_2char_query_hits: tokenized_korean_text
column 이 채워진 chunk 의 '한국' 2-char query hit.
- fts_v009_english_whole_token_only: V007 trigram substring 매칭
회귀 (Path A) — 'token' query 가 'tokenizer' chunk 에서 0-hit.
- crates/kebab-app/tests/search_korean.rs:
- korean_morphological_2char_query_lexical_mode: end-to-end
한국어 wiki fixture ingest → '한국' / '서울' query hit.
- korean_morphological_mixed_english_korean_query: 'Rust' English
whole-token + '최적화' Korean morpheme hit.
crates/kebab-search/src/lexical.rs:
- build_match_string() 의 MIN_TRIGRAM_CHARS(3) → MIN_QUERY_CHARS(2).
V009 unicode61 은 최소 token 길이 제한 없어 2자 한국어 morpheme
query 가 통과되어야 함. 1자 단독은 여전히 필터.
- 관련 unit test 2개 V009 동작으로 갱신.
fixture text 는 lindera ko-dic 의 실제 segmentation 동작에 의존
(spec Appendix B prior-knowledge 예측). 실측 시 fixture 조정 가능.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §9.1, §9.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S7)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V009 의 FTS5 tokenizer 가 trigram → unicode61 + 한국어 형태소 분해
column 로 갱신됨. lexical_index_version 의 format 에
`fts5-v009-korean-morphological` suffix 추가하여 V007 baseline 과
구별. eval runner 의 config_snapshot 및 search cache 무효화에
자동 picks up.
기존 format: lex:{chunker_version}
신규 format: lex:{chunker_version}:fts5-v009-korean-morphological
Wire schema shape 변경 없음 (SearchHit.index_version 의 string
content 만 변화). lexical_index_version_is_returned_unchanged test
는 IndexVersion 의 임의 string 을 사용해 unchanged.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §11.1, §11.3
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S6)
V009 unicode61 + 형태소 tokenizer 환경에서 2-char 한국어 query 가
hit 가능해졌으므로 V007 시기의 "3자 이상 권장" hint 가 obsolete.
SearchResponse.hint field 는 wire schema 보존 위해 struct 에 유지 +
항상 None.
- kebab-app/src/app.rs: short_query_hint 함수 + doc-comment 삭제.
2 호출 site 가 hint = None 으로 정리.
- kebab-app/src/lib.rs: re-export 에서 short_query_hint 제거.
- kebab-tui/{app.rs,search.rs,run.rs}: short_query_hint field + 4
호출 cascade 제거.
- kebab-cli/tests/wire_search_response.rs:
search_plain_emits_short_query_hint_to_stderr test 삭제.
search_json_emits_hint_field_for_short_query →
search_json_hint_absent_for_short_query_v009 으로 교체
(hint 항상 None 검증).
- kebab-search/src/lexical.rs::build_match_string: V007 의 trigram
multi-token OR-combine 분기는 V009 환경에서 redundant 하나 보존
(future 확장성) — doc-comment 1 줄 추가.
Wire schema shape 변경 없음 (search_response.schema.json:33 의 hint
field 보존, struct 에 None 으로 항상 셋팅).
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §7.2, §7.3, §11.3
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S5)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V007 → V009 업그레이드 시 기존 chunks 의 tokenized_korean_text 가
NULL — 첫 App::open_with_config 호출 시 자동으로 lindera ko-dic
으로 분해 후 UPDATE. chunks_au trigger 가 chunks_fts 를 자동 재-index.
사용자 재-ingest 불필요.
- crates/kebab-store-sqlite/src/store.rs:
backfill_tokenized_korean_text(progress_cb, tokenize) API. 1000 row 마다
commit + progress 콜백. idempotent (IS NULL 필터로 partial
completion 재실행 안전). tokenizer 를 파라미터로 받아 §8 dep 경계 유지.
- crates/kebab-app/src/app.rs::open_with_config: run_migrations 직후
backfill 호출. 실패 시 warn log 만 (App open 은 성공 — vector/hybrid
mode 계속 가능). 500 row 마다 info log progress.
- crates/kebab-store-sqlite/tests/fts.rs:
backfill_tokenized_korean_text_populates_nullable_rows 단위 test
(idempotency 포함).
- clippy pre-existing 오류 수정 (redundant_closure, map_unwrap_or,
cast_lossless, uninlined_format_args — kebab-app/ingest_log.rs,
pdf_ocr_apply.rs, app.rs, tests/ocr_inspect_smoke.rs).
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §8.1, §8.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S4)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V004 seeds corpus_revision=0, V009 migration bumps to 1 (spec §5.2 —
LRU cache invalidation). Test previously asserted fresh store = 0;
now reads post-migration baseline dynamically and verifies that the
ingest commit increments past it.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §5.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S3 follow-up)
V007 trigram tokenizer 의 substring 매칭을 검증하던 3 test 는 V009
unicode61 으로 의도된 회귀 (spec §3 Non-Goals Path A) 가 발생하므로
obsolete:
- fts_trigram_korean_3char_substring_hits: '발생한' → '발생한다' hit
은 trigram 의 substring 매칭이라 V009 의 whole-token 매칭에서 fail.
- fts_trigram_korean_short_query_zero_hit_pinned: 2-char Korean
query 의 0-hit 동작은 V009 의 형태소 column 으로 해소되므로 이 핀
자체가 obsolete (S7 이 신규 2-char hit test 로 대체).
- fts_trigram_english_substring_hits: 'token' → 'tokenizer' hit 은
V009 unicode61 의 whole-token only 에서 fail.
신규 추가:
- fts_v009_unicode61_space_separated_korean_token_hits: V009 unicode61
의 whole-token 매칭 sanity (token '충돌은' hit, substring '발생한'
0-hit). S7 이 추가할 morphological 검증 test 와 별개의 baseline.
S7 (plan §2 Step 7) 가 v009_korean_morphological_2char_query_hits +
v009_english_whole_token_only 를 추가하여 회귀 + 신규 동작 모두 핀할
예정.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §3, §9.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S3 follow-up)
S3 spec compliance reviewer (sonnet) 가 2 blocker 발견:
1. crates/kebab-store-sqlite/src/documents.rs: get_chunk SELECT 가
tokenized_korean_text column 을 미조회 → DB 의 값이 read 시 유실.
SELECT column list + row → Chunk 변환 시 row.get 인덱스 추가.
ChunkRow struct + chunk_row_from_sql + get_chunk Chunk 생성 cascade.
2. crates/kebab-chunk/src/code_*_ast_v1.rs (9 file): make_chunk 가
tokenized_korean_text: None 하드코딩 → 한국어 주석을 가진 코드
파일이 FTS hit 안 됨. tier2_shared 와 동일 패턴으로
tokenize_korean_morphological(text) 호출 cascade.
이 commit 은 S3 의 rework — amend 아닌 별 commit (S3 boundary
유지). spec §6.2 invariant ("모든 chunker 가 chunk emit 직전에
tokenize 호출") 충족.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §6.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S3 rework)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V009 의 tokenized_korean_text column 에 들어갈 morpheme sequence
를 lindera ko-dic 으로 분해. chunk builder pipeline 의 chunk 생성
직후 시점에서 호출 → chunk struct 의 field 에 pre-fill → store
의 put_chunks 가 단일 transaction 안에서 INSERT.
- crates/kebab-core/src/chunk.rs: Chunk struct 에
tokenized_korean_text: Option<String> field 추가 (#[serde(default)]).
- crates/kebab-chunk/src/lib.rs: tokenize_korean_morphological()
helper + OnceLock 캐싱 + fallback (None) 정책.
- crates/kebab-chunk/Cargo.toml: lindera features = ["embed-ko-dic"]
추가 (DictionaryKind::KoDic 활성화에 필요).
- 모든 chunker (tier2_shared, md_heading_v1, pdf_page_v1, 9개
code AST v1): Chunk 리터럴에 tokenized_korean_text pre-fill.
- crates/kebab-store-sqlite/src/documents.rs::put_chunks: INSERT
SQL column list + placeholder + binding 갱신 (12번째 column).
- crates/kebab-chunk/tests/tokenize_korean.rs: 단위 테스트 2개.
lindera 3.0.7 API 정정: load_dictionary_from_kind →
load_embedded_dictionary, Token.text → Token.surface.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §6.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S3)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V007 trigram tokenizer 의 한국어 2자 query 0-hit 한계 (Bug #8) 해소를
위한 V009 migration 추가. unicode61 tokenizer 로 환원 + 한국어 형태소
분해 결과를 별 column `tokenized_korean_text` 에 pre-fill 하는 방식.
- migrations/V009__fts_korean_morphological.sql 신규: column ADD,
chunks_fts DROP+재정의, 3 trigger CASE expression, backfill INSERT,
corpus_revision bump.
- design §5.5 갱신: trigram → unicode61 + 형태소 column. CASE
expression trigger 본문.
- crates/kebab-store-sqlite/tests/fts.rs: V007 verbatim test 를
V009 source-of-truth 로 rename. v009_bumps_corpus_revision unit
test 추가.
- store.rs: clippy bool_to_int_with_if + cast_lossless 기존 경고 수정
(pdf_ocr_events 관련 코드, S1 작업 중 발견).
영어 substring 매칭은 V002 (whole-token only) 로 회귀 — spec §3
Non-Goals + 후속 release notes (v0.20.1) 에서 정직히 기술.
Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S1)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LoggingCfg gains two fields with serde defaults: keep_recent_runs
(default 100, top-N file retention) and retention_days (default 30,
time-based retention for both ndjson files and the SQLite mirror).
IngestLogWriter::open now runs cleanup_old_logs before creating a new
ingest-*.ndjson — delete iff (idx >= keep_recent) OR (modified <=
cutoff). ingest_with_config_opts also calls
SqliteStore::prune_pdf_ocr_events(retention_days) at ingest start so
the SQLite mirror tracks the same retention window.
Backward compat (AC-9): both new fields use #[serde(default = ...)],
so a pre-v0.20.x config with only [logging] ingest_log_enabled +
ingest_log_dir parses unchanged. kebab init writes the new defaults
automatically via Config::default() -> toml::to_string_pretty (AC-12).
docs/SMOKE.md config example synced.
Closure r1 F5: explicit OR-on-stale comment inside cleanup_old_logs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two new wire schemas land as additive minor: ocr_stats.v1 (corpus-wide
aggregate — total_events, success_rate, p50/p90/p99/max_ms, by_engine,
top-10 by_doc by failure count) and ocr_failures.v1 (per-doc or
corpus-wide recent failures, with --doc-id + --limit). Both ship via
new CLI subcommands `kebab inspect ocr-stats` / `inspect ocr-failures`.
App gains four facade methods: inspect_ocr_stats /
inspect_ocr_failures plus their *_with_config companions — required by
CLAUDE.md "the facade rule" so `--config <path>` is honored. The CLI
dispatch arms thread cfg explicitly into the _with_config form.
Runtime introspection emit (WIRE_SCHEMAS in schema.rs) gains two
entries; the meta JSON Schema (schema.schema.json) is untouched
because its wire.schemas is pattern-based, not enum-based.
ingest_log::percentiles extended to (p50, p90, p99, max). p99 surfaces
only via inspect ocr-stats; IngestSummary (round 1) stays 3-percentile.
SKILL.md synced with the two new schemas (AC-13).
Closure r2 G2 (facade *_with_config pair) + G3 (runtime emit, not
meta schema file) + closure r1 F4 (p99) resolved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pre-capture canonical.doc_id and Arc<SqliteStore> before the OCR
emit_progress closure so both the ndjson file and the SQLite mirror
carry the same doc_id for every event. File write is durable
(errors propagate); SQLite insert is non-critical (tracing::warn on
failure, ingest does not abort) per spec R-1.
LogEvent::Ocr gains a doc_id: Option<&str> field as an additive
Serde change — round 1 ndjson logs deserialize with doc_id=None.
Closure r1 F1: doc_id NULL in dual-write resolved via
let doc_id_for_log = canonical.doc_id.0.clone() pre-capture.
Closure r2 G1: Arc::clone(&app.sqlite) reused instead of opening a
second SqliteStore — eliminates double-open lock contention and
duplicate migration runs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add migrations/V008__pdf_ocr_events.sql with the events table + 3
indices (doc_id, run_id, ts). SqliteStore gains two pub fn:
record_pdf_ocr_event (insert one OCR sample) and prune_pdf_ocr_events
(delete rows older than retention_days; returns the affected row
count). Both follow the existing Mutex<Connection> lock pattern.
Wiring into ingest path lands in the next commit.
Closure r1 F2: explicit lock acquisition in both methods.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add extract_image_dimensions(bytes) helper using image::ImageReader
and fill the 2 PdfOcrProgress::Finished emit points in pdf_ocr_apply.rs
where page_image_bytes is in scope (OCR error path + success path).
The no-DCTDecode skip path leaves None as page_image_bytes is absent.
Result: LogEvent::Ocr carries non-null image_width/image_height on
successful raster decode, enabling future size-conditioned timeout tuning.
Closure r1 F3: kebab-app/Cargo.toml image features += "jpeg" added as
direct [dependencies] entry (not relying on feature unification via
kebab-parse-image).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
