feat(app): first-boot eager backfill for tokenized_korean_text

V007 → V009 업그레이드 시 기존 chunks 의 tokenized_korean_text 가
NULL — 첫 App::open_with_config 호출 시 자동으로 lindera ko-dic
으로 분해 후 UPDATE. chunks_au trigger 가 chunks_fts 를 자동 재-index.
사용자 재-ingest 불필요.

- crates/kebab-store-sqlite/src/store.rs:
  backfill_tokenized_korean_text(progress_cb, tokenize) API. 1000 row 마다
  commit + progress 콜백. idempotent (IS NULL 필터로 partial
  completion 재실행 안전). tokenizer 를 파라미터로 받아 §8 dep 경계 유지.
- crates/kebab-app/src/app.rs::open_with_config: run_migrations 직후
  backfill 호출. 실패 시 warn log 만 (App open 은 성공 — vector/hybrid
  mode 계속 가능). 500 row 마다 info log progress.
- crates/kebab-store-sqlite/tests/fts.rs:
  backfill_tokenized_korean_text_populates_nullable_rows 단위 test
  (idempotency 포함).
- clippy pre-existing 오류 수정 (redundant_closure, map_unwrap_or,
  cast_lossless, uninlined_format_args — kebab-app/ingest_log.rs,
  pdf_ocr_apply.rs, app.rs, tests/ocr_inspect_smoke.rs).

Spec: docs/superpowers/specs/2026-05-28-v0.20.x-korean-morphological-tokenizer-spec.md §8.1, §8.2
Plan: docs/superpowers/plans/2026-05-28-v0.20.x-korean-morphological-tokenizer-plan.md (S4)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-28 11:01:00 +00:00
parent e8f44a57e3
commit b63af20b72
6 changed files with 148 additions and 17 deletions

View File

@@ -13,6 +13,7 @@
//! that bypasses the `SqliteStore` mutex; that's fine because each test
//! gets its own tempdir and no concurrent mutator is in flight.
use kebab_chunk::tokenize_korean_morphological;
use kebab_store_sqlite::{SqliteStore, rebuild_chunks_fts};
use rusqlite::Connection;
@@ -453,6 +454,53 @@ fn v009_bumps_corpus_revision() {
);
}
// ── 5c. backfill_tokenized_korean_text ───────────────────────────────
#[test]
fn backfill_tokenized_korean_text_populates_nullable_rows() {
let env = common::TestEnv::new();
let store = SqliteStore::open(&env.config()).unwrap();
store.run_migrations().unwrap();
// chunks 에 한국어 row 두 개 INSERT (tokenized_korean_text 는 chunks_ai trigger
// 가 채우지만, 여기서는 raw_conn_no_fk 로 직접 INSERT 하므로 NULL 로 남음).
let conn = raw_conn_no_fk(&env);
insert_chunk(&conn, &"a".repeat(32), &"d".repeat(32), "[]", "한국 문화는 오래되었다");
insert_chunk(&conn, &"b".repeat(32), &"d".repeat(32), "[]", "서울특별시는 한국의 수도");
let null_count_before: i64 = conn
.query_row(
"SELECT COUNT(*) FROM chunks WHERE tokenized_korean_text IS NULL",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(null_count_before, 2);
drop(conn);
// backfill 호출 → lindera 가 두 row 모두 분해 성공 → 2 반환.
let processed = store
.backfill_tokenized_korean_text(|_, _| {}, tokenize_korean_morphological)
.unwrap();
assert_eq!(processed, 2, "both rows should be populated by lindera");
let conn = raw_conn_no_fk(&env);
let null_count_after: i64 = conn
.query_row(
"SELECT COUNT(*) FROM chunks WHERE tokenized_korean_text IS NULL",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(null_count_after, 0);
// idempotency: 두 번째 호출 → 0 (모든 row 가 이미 채워져 있음).
drop(conn);
let processed_again = store
.backfill_tokenized_korean_text(|_, _| {}, tokenize_korean_morphological)
.unwrap();
assert_eq!(processed_again, 0);
}
// ── 6. WAL cleanup: drop store before tempdir reaps WAL/SHM ──────────
/// Mirror the P1-6 pattern: opening + migrating + dropping the store