diff --git a/README.md b/README.md index c293d5e..e4b27f4 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ - **Rust toolchain** ≥ 1.85 (workspace 가 edition 2024 + resolver 3 사용). [rustup](https://rustup.rs) 권장. - **Ollama** — `kebab ask` 와 이미지 OCR/caption 가 사용. `https://ollama.com/download` 에서 설치 후 `ollama serve` 실행. 기본 LLM 은 gemma4 계열 (`ollama pull gemma4:e4b`) — OCR / caption 도 같은 family 라 모델 하나만 pull 하면 됨. 더 큰 variant 원하면 `gemma4:26b` 등으로 config override. config 의 `[models.llm].endpoint` 에 host:port 명시. - **빌드 디스크** — 첫 빌드 시 `target/` 가 6–10 GB (Lance + DataFusion + fastembed). 여유 확인. -- **fastembed 모델** — 첫 `kebab ingest` 시 `multilingual-e5-small` (~470 MB) 자동 다운로드. +- **fastembed 모델** — 첫 `kebab ingest` 시 `multilingual-e5-large` (~1.3 GB, fb-39b) 자동 다운로드. `config.toml` 에서 `model = "multilingual-e5-small"` 로 명시하면 이전 모델 사용. ## 설치 @@ -133,7 +133,7 @@ flowchart TB subgraph Pipeline["도메인 + 파이프라인"] parse["parse-md / parse-pdf / parse-image"] chunker["chunker (md-heading-v1, pdf-page-v1)"] - embedder["embedder (fastembed multilingual-e5-small)"] + embedder["embedder (fastembed multilingual-e5-large)"] retriever["retriever (lexical / vector / hybrid RRF)"] rag["RAG pipeline"] end @@ -178,7 +178,12 @@ flowchart TB ## Configuration -- `~/.config/kebab/config.toml` — `kebab init` 가 XDG 경로에 생성. `[workspace]` (root, exclude — include 필드는 제거됨, 지원 형식은 자동 결정), `[storage]`, `[chunking]`, `[models.embedding]`, `[models.llm]`, `[image.ocr]`, `[image.caption]`, `[search]`, `[rag]`, `[ui]` 절. `[ui] theme = "dark" | "light"` 로 TUI 팔레트 선택 (default `"dark"`, 알 수 없는 값은 dark fallback). `[search] stale_threshold_days = 30` (p9-fb-32) — search hit / RAG citation 의 `stale` 플래그 기준 (default 30 일, `0` 으로 비활성화). 옛 config 의 `workspace.include = [...]` 은 silently 무시 + 단발 deprecation warning (p9-fb-25). +- `~/.config/kebab/config.toml` — `kebab init` 가 XDG 경로에 생성. `[workspace]` (root, exclude — include 필드는 제거됨, 지원 형식은 자동 결정), `[storage]`, `[chunking]`, `[models.embedding]`, `[models.llm]`, `[image.ocr]`, `[image.caption]`, `[search]`, `[rag]`, `[ui]` 절. + - `[models.embedding]` — + - `model` (default `"multilingual-e5-large"`, fb-39b) — 다국어 sentence embedding 모델. 1024-dim. ONNX (~1.3 GB) 첫 실행 시 fastembed cache (`config.storage.model_dir/fastembed/`) 에 자동 다운로드. `"multilingual-e5-small"` (384 dim) 는 backwards-compat 으로 사용 가능 — TOML 에 명시. + - `dimensions` (default `1024`) — 모델의 embedding 차원. config 와 LanceDB stored dim 불일치 시 검색 결과 0 건 (orphan table). 모델 변경 시 `kebab reset --vector-only && kebab ingest` 로 vector index 재구축 권장. + - `[ui] theme = "dark" | "light"` 로 TUI 팔레트 선택 (default `"dark"`, 알 수 없는 값은 dark fallback). + - `[search] stale_threshold_days = 30` (p9-fb-32) — search hit / RAG citation 의 `stale` 플래그 기준 (default 30 일, `0` 으로 비활성화). 옛 config 의 `workspace.include = [...]` 은 silently 무시 + 단발 deprecation warning (p9-fb-25). - `[rag] prompt_template_version` (default `"rag-v2"`) — RAG system prompt version. `"rag-v1"` 은 legacy backwards-compat (사용자 명시 시 유지). v2 강화 규칙: (1) fact 인용 시 [#번호] 앞에 chunk 속 원문 큰따옴표 표기, (2) 학습 지식 동원 금지, (3) 근거 모호 시 "확실하지 않다" 명시. - `--config ` flag — 임시 워크스페이스 / 격리 테스트 시 사용. CLI / TUI 모두 honor. - `KEBAB_*` env — 일부 키 override (`KEBAB_RAG_SCORE_GATE`, `KEBAB_EVAL_GOLDEN`, `KEBAB_COMMIT_HASH` 등). diff --git a/crates/kebab-config/src/lib.rs b/crates/kebab-config/src/lib.rs index 342591c..02ab5ca 100644 --- a/crates/kebab-config/src/lib.rs +++ b/crates/kebab-config/src/lib.rs @@ -302,9 +302,9 @@ impl Config { models: ModelsCfg { embedding: EmbeddingModelCfg { provider: "fastembed".to_string(), - model: "multilingual-e5-small".to_string(), + model: "multilingual-e5-large".to_string(), version: "v1".to_string(), - dimensions: 384, + dimensions: 1024, batch_size: 64, }, llm: LlmCfg { @@ -764,7 +764,8 @@ mod tests { let c = Config::defaults(); assert_eq!(c.rag.score_gate, 0.30); assert_eq!(c.chunking.target_tokens, 500); - assert_eq!(c.models.embedding.dimensions, 384); + assert_eq!(c.models.embedding.model, "multilingual-e5-large"); + assert_eq!(c.models.embedding.dimensions, 1024); assert_eq!(c.search.rrf_k, 60); } @@ -947,9 +948,9 @@ chunker_version = "md-heading-v1" [models.embedding] provider = "fastembed" -model = "multilingual-e5-small" +model = "multilingual-e5-large" version = "v1" -dimensions = 384 +dimensions = 1024 batch_size = 64 [models.llm] diff --git a/crates/kebab-embed-local/Cargo.toml b/crates/kebab-embed-local/Cargo.toml index 41c30c8..452b2f1 100644 --- a/crates/kebab-embed-local/Cargo.toml +++ b/crates/kebab-embed-local/Cargo.toml @@ -5,14 +5,14 @@ edition = { workspace = true } rust-version = { workspace = true } license = { workspace = true } repository = { workspace = true } -description = "Local fastembed-rs adapter implementing kb_core::Embedder (multilingual-e5-small default)" +description = "Local fastembed-rs adapter implementing kb_core::Embedder (multilingual-e5-large default, e5-small backwards-compat)" [dependencies] kebab-config = { path = "../kebab-config" } kebab-embed = { path = "../kebab-embed" } # Default features bring `ort-download-binaries` (bundled ONNX runtime) # and `hf-hub-native-tls` (first-run model download). No extra features -# needed for the multilingual-e5-small path. +# needed for the multilingual-e5-{small,large} paths. fastembed = { workspace = true } tracing = { workspace = true } anyhow = { workspace = true } diff --git a/crates/kebab-embed-local/src/lib.rs b/crates/kebab-embed-local/src/lib.rs index ca442fd..2033515 100644 --- a/crates/kebab-embed-local/src/lib.rs +++ b/crates/kebab-embed-local/src/lib.rs @@ -1,8 +1,9 @@ //! `kb-embed-local` — `FastembedEmbedder`, a local ONNX-backed //! [`Embedder`](kebab_embed::Embedder) implementation. //! -//! Wraps [`fastembed::TextEmbedding`] for the default `multilingual-e5-small` -//! (384-dim) model. Honors `config.models.embedding.batch_size` and applies +//! Wraps [`fastembed::TextEmbedding`]. Default is `multilingual-e5-large` +//! (1024-dim, p9-fb-39b); `multilingual-e5-small` (384-dim) is also supported +//! for backwards-compat. Honors `config.models.embedding.batch_size` and applies //! the e5 prefix convention (§11.3 of the design report): //! //! * `EmbeddingKind::Document` → `"passage: "` prefix @@ -69,9 +70,9 @@ impl FastembedEmbedder { .with_context(|| format!("create fastembed cache dir {}", cache_dir.display()))?; // 2. Resolve the fastembed enum variant from - // `config.models.embedding.model`. Currently only the default - // `multilingual-e5-small` is wired; other model names error - // out with a clear message rather than silently misconfiguring. + // `config.models.embedding.model`. Currently `multilingual-e5-large` + // (default) and `multilingual-e5-small` are wired; other model names + // error out with a clear message rather than silently misconfiguring. let model_name = resolve_model(&config.models.embedding.model)?; // 3. Verify dim match BEFORE loading the model — if the config @@ -100,7 +101,7 @@ impl FastembedEmbedder { target: "kebab-embed-local", model = %config.models.embedding.model, cache_dir = %cache_dir.display(), - "loading embedding model (first run will download ~470MB)" + "loading embedding model (first run downloads model weights — ~470MB for e5-small, ~1.3GB for e5-large)" ); let inner = TextEmbedding::try_new(opts) .context("fastembed: TextEmbedding::try_new")?; @@ -193,17 +194,18 @@ fn prefix_input(input: &EmbeddingInput<'_>) -> String { } /// Resolve a `config.models.embedding.model` string to a fastembed -/// `EmbeddingModel` enum variant. Only `multilingual-e5-small` is wired -/// for p3-2; additional model names should be added (and their dims -/// pinned in tests) as needed. +/// `EmbeddingModel` enum variant. Currently supports `multilingual-e5-small` +/// (384-dim) and `multilingual-e5-large` (1024-dim); additional model names +/// should be added (and their dims pinned in tests) as needed. fn resolve_model(name: &str) -> Result { match name { "multilingual-e5-small" => Ok(EmbeddingModel::MultilingualE5Small), + "multilingual-e5-large" => Ok(EmbeddingModel::MultilingualE5Large), other => anyhow::bail!( "kb-embed-local: unsupported embedding model {other:?}; \ - this adapter currently only ships `multilingual-e5-small`. \ - Add a new arm to `resolve_model` (and a fastembed feature \ - flag if needed) to support more." + this adapter currently ships `multilingual-e5-small` and \ + `multilingual-e5-large`. Add a new arm to `resolve_model` \ + (and a fastembed feature flag if needed) to support more." ), } } @@ -294,6 +296,12 @@ mod tests { resolve_model("multilingual-e5-small").expect("default model resolves"); } + #[test] + fn resolve_model_supports_e5_large() { + let m = resolve_model("multilingual-e5-large").expect("e5-large should resolve"); + let _ = m; + } + #[test] fn resolve_unknown_model_errors() { let err = resolve_model("not-a-real-model").expect_err("unknown model errors"); @@ -301,6 +309,21 @@ mod tests { assert!(msg.contains("unsupported embedding model"), "msg={msg}"); } + // ── check_dim ──────────────────────────────────────────────────── + + #[test] + fn check_dim_passes_for_1024() { + check_dim(1024, 1024).expect("matching dims must pass"); + } + + #[test] + fn check_dim_rejects_384_vs_1024() { + let err = check_dim(384, 1024).expect_err("dim mismatch must error"); + let msg = format!("{err}"); + assert!(msg.contains("384") && msg.contains("1024"), + "error must mention both dims, got: {msg}"); + } + // expand_path tests live in `kb-config::paths`. The adapter imports // it and trusts the upstream coverage rather than duplicating it. } diff --git a/crates/kebab-embed-local/tests/embed_model.rs b/crates/kebab-embed-local/tests/embed_model.rs index fee9c5f..2212184 100644 --- a/crates/kebab-embed-local/tests/embed_model.rs +++ b/crates/kebab-embed-local/tests/embed_model.rs @@ -3,10 +3,11 @@ //! //! ## Why every test in this file is `#[ignore]` //! -//! The first call to `FastembedEmbedder::new` downloads ~470 MB of -//! weights from Hugging Face into `data_dir/models/fastembed/`. Doing -//! that on every `cargo test` invocation is wasteful, so the bare -//! invocation skips this file entirely. +//! The first call to `FastembedEmbedder::new` downloads ~1.3 GB of +//! weights (multilingual-e5-large per p9-fb-39b default) from Hugging +//! Face into `data_dir/models/fastembed/`. Doing that on every +//! `cargo test` invocation is wasteful, so the bare invocation skips +//! this file entirely. //! //! Run the full suite with: //! ```text @@ -58,19 +59,20 @@ fn shared_embedder() -> &'static FastembedEmbedder { // ─── construction ───────────────────────────────────────────────────── #[test] -#[ignore = "downloads ~470MB ONNX model on first run; CI-only"] -fn default_config_constructs_with_dims_384() { +#[ignore = "downloads ~1.3GB ONNX model on first run; CI-only"] +fn default_config_constructs_with_dims_1024() { + // p9-fb-39b: default flipped to multilingual-e5-large (1024 dim). let emb = shared_embedder(); - assert_eq!(emb.dimensions(), 384); - assert_eq!(emb.model_id().0, "multilingual-e5-small"); + assert_eq!(emb.dimensions(), 1024); + assert_eq!(emb.model_id().0, "multilingual-e5-large"); assert_eq!(emb.model_version().0, "v1"); } #[test] -#[ignore = "downloads ~470MB ONNX model on first run; CI-only"] +#[ignore = "downloads ~1.3GB ONNX model on first run; CI-only"] fn mismatched_dims_in_config_errors_at_construction() { let (mut cfg, _tmp) = test_config(); - cfg.models.embedding.dimensions = 512; // model is 384 + cfg.models.embedding.dimensions = 512; // model is 1024 (e5-large default) // `FastembedEmbedder` deliberately does not implement `Debug` // (its inner ONNX session has no useful debug shape), so we // can't use `expect_err`; match the Result manually. @@ -80,7 +82,7 @@ fn mismatched_dims_in_config_errors_at_construction() { }; let msg = format!("{err}"); assert!(msg.contains("dimension mismatch"), "msg={msg}"); - assert!(msg.contains("384"), "msg={msg}"); + assert!(msg.contains("1024"), "msg={msg}"); assert!(msg.contains("512"), "msg={msg}"); } @@ -104,8 +106,8 @@ fn document_and_query_yield_different_vectors() { ]) .expect("embed two inputs"); assert_eq!(out.len(), 2); - assert_eq!(out[0].len(), 384); - assert_eq!(out[1].len(), 384); + assert_eq!(out[0].len(), 1024); + assert_eq!(out[1].len(), 1024); // Both vectors are L2-normalized → cosine similarity == dot product. let cos: f32 = out[0] @@ -142,11 +144,11 @@ fn output_vectors_are_l2_normalized() { ]; let out = emb.embed(&inputs).expect("embed"); // Per `kebab_embed::assert_unit_norm` docs: `5e-4` is the safe bound at - // 384 dims (f32::EPSILON × √384 ≈ 2.3e-6, but ONNX kernels add + // 1024 dims (f32::EPSILON × √1024 ≈ 2.3e-6, but ONNX kernels add // their own per-component noise; 1e-3 is very generous and matches // the spec's `± 1e-3`). kebab_embed::assert_unit_norm(&out, 1e-3); - kebab_embed::assert_vector_shape(&out, 384); + kebab_embed::assert_vector_shape(&out, 1024); } // ─── determinism ────────────────────────────────────────────────────── @@ -254,7 +256,7 @@ fn snapshot_aggregate_hash_is_stable() { // Round every component to 4 decimal places, hash deterministically. let mut hasher = DefaultHasher::new(); for (i, v) in out.iter().enumerate() { - assert_eq!(v.len(), 384, "row {i} dim mismatch"); + assert_eq!(v.len(), 1024, "row {i} dim mismatch"); for x in v { let rounded: i32 = (*x * 1.0e4).round() as i32; rounded.hash(&mut hasher); diff --git a/docs/SMOKE.md b/docs/SMOKE.md index 7022d45..ca6c7a7 100644 --- a/docs/SMOKE.md +++ b/docs/SMOKE.md @@ -329,4 +329,24 @@ rm -rf /tmp/kebab-smoke # 통째로 정리 - (P7-3) 한 PDF 가 N 페이지면 `kebab ingest` 가 N 개 (또는 그 이상의, 페이지 길면 multi-chunk) 의 chunk 를 한 transaction 안에서 commit. 500 페이지 책 → 500+ chunk 한 번에 → embedding throughput 가 bottleneck. 임베딩 활성 워크스페이스에서 큰 PDF 를 처음 ingest 하면 분-단위 시간 + WAL 크기 증가 가능 — P+ 스케일 hardening task 까지 정상 동작이지만 비용은 측정 가능. - (P7-3 + follow-up) 동일 path 에 byte 가 다른 PDF 를 두 번째 ingest 하면 `purge_vector_orphans_for_workspace_path` 가 옛 chunk_id 를 LanceDB 에서 먼저 삭제, 이어서 `purge_orphan_at_workspace_path` 가 옛 doc / chunks / embedding_records 를 SQLite 에서 sweep. 새 byte 가 새 `doc_id` 로 색인됨. `IngestReport` 에 그 자산만 `new+=1` (다른 자산은 `updated`). 두 store 모두 정합 — 옛 본문 검색 시 옛 chunks 가 더 이상 surface 되지 않음. +### Embedding upgrade (fb-39b) + +`multilingual-e5-small` 에서 `multilingual-e5-large` 로 업그레이드 시퀀스: + +```bash +# 기존 vector index 정리 (orphan table 회피) +kebab --config /tmp/kebab-smoke/config.toml reset --vector-only + +# config.toml 의 [models.embedding] 갱신: +# model = "multilingual-e5-large" +# dimensions = 1024 + +# 재-ingest — fastembed 가 첫 실행 시 e5-large ONNX (~1.3 GB) 자동 다운로드. +# 다운로드 시간 + 모든 chunk re-embed 시간 (e5-small 대비 ~3-4×). +kebab --config /tmp/kebab-smoke/config.toml ingest + +# fb-39 의 P@k metric 으로 small vs large 비교: +kebab --config /tmp/kebab-smoke/config.toml eval run +``` + 자세한 history 와 발견된 버그는 [tasks/HOTFIXES.md](../tasks/HOTFIXES.md) 참조. diff --git a/docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md b/docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md new file mode 100644 index 0000000..58e34f2 --- /dev/null +++ b/docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md @@ -0,0 +1,405 @@ +# fb-39b Embedding Model Upgrade Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Upgrade default embedding model from `multilingual-e5-small` (384 dim) to `multilingual-e5-large` (1024 dim) so retrieval precision can improve on Korean dogfooding corpus. Existing user TOMLs pinning `multilingual-e5-small` keep working unchanged. + +**Architecture:** Three-line code surface: a new arm in `kebab-embed-local::resolve_model`, defaults flipped in `kebab-config::Config::defaults` (and the TOML template), and the existing test asserting the 384 default updated. LanceDB tables are already namespaced by `(model, dim)` so an upgraded model writes to a fresh table; fb-23 incremental ingest detects the `embedding_version` mismatch and auto-re-embeds on next ingest. No migration tooling — orphan old-model tables cleaned via `kebab reset --vector-only`. + +**Tech Stack:** Rust 2024, fastembed 4.9.1 (`MultilingualE5Large` enum already shipped), LanceDB. + +**Spec:** `docs/superpowers/specs/2026-05-10-p9-fb-39b-embedding-upgrade-design.md` + +--- + +## File map + +**Modify:** +- `crates/kebab-embed-local/src/lib.rs` — add `multilingual-e5-large` arm in `resolve_model`. Update or add `check_dim` test for 1024. +- `crates/kebab-config/src/lib.rs` — flip `Config::defaults().models.embedding.{model, dimensions}` and the TOML template at line ~952. Update default test at line 767. +- `README.md` — `[models.embedding]` section: mention new default + small opt-out + dim mismatch hint. +- `docs/SMOKE.md` — append "Embedding upgrade (fb-39b)" walkthrough showing the `kebab reset --vector-only && kebab ingest` sequence + first-run ONNX download warning. +- `docs/superpowers/specs/2026-04-27-kebab-final-form-design.md` §5 storage / §9 versioning — update default model + dim references. +- `tasks/HOTFIXES.md` — entry for embedding upgrade UX (orphan tables on model swap, reset --vector-only flow). +- `tasks/p9/p9-fb-39-retrieval-precision-tuning.md` banner — append note "fb-39b lever 적용 (embedding upgrade) ✅". +- `tasks/INDEX.md` — fb-39b row ✅ (new row alongside fb-39). + +**Create:** +- `tasks/p9/p9-fb-39b-embedding-upgrade.md` — new task spec mirroring fb-39 frontmatter (status: completed, design + plan links). + +--- + +## Task 1: Add multilingual-e5-large to kebab-embed-local + +**Files:** +- Modify: `crates/kebab-embed-local/src/lib.rs` + +- [ ] **Step 1: Append failing tests** + +Find the existing `mod tests` (~line 230). Append: + +```rust +#[test] +fn resolve_model_supports_e5_large() { + let m = resolve_model("multilingual-e5-large").expect("e5-large should resolve"); + // The fastembed enum is non-comparable in some versions; we only need + // to confirm Ok and that the underlying TextEmbedding could be built. + // Avoid actually constructing the model in tests (1.3 GB ONNX download). + let _ = m; +} + +#[test] +fn check_dim_passes_for_1024() { + check_dim(1024, 1024).expect("matching dims must pass"); +} + +#[test] +fn check_dim_rejects_384_vs_1024() { + let err = check_dim(384, 1024).expect_err("dim mismatch must error"); + let msg = format!("{err}"); + assert!(msg.contains("384") && msg.contains("1024"), + "error must mention both dims, got: {msg}"); +} +``` + +- [ ] **Step 2: Run tests to confirm failures** + +```bash +cargo test -p kebab-embed-local resolve_model_supports_e5_large +cargo test -p kebab-embed-local check_dim_passes_for_1024 +``` +Expected: `resolve_model_supports_e5_large` fails (no arm); `check_dim_*` passes already (helper is generic). + +- [ ] **Step 3: Add arm to resolve_model** + +In `crates/kebab-embed-local/src/lib.rs`, find `fn resolve_model` (~line 199). Replace the match body: + +```rust +fn resolve_model(name: &str) -> Result { + match name { + "multilingual-e5-small" => Ok(EmbeddingModel::MultilingualE5Small), + "multilingual-e5-large" => Ok(EmbeddingModel::MultilingualE5Large), + other => anyhow::bail!( + "kb-embed-local: unsupported embedding model {other:?}; \ + this adapter currently ships `multilingual-e5-small` and \ + `multilingual-e5-large`. Add a new arm to `resolve_model` \ + (and a fastembed feature flag if needed) to support more." + ), + } +} +``` + +- [ ] **Step 4: Run tests — all pass** + +```bash +cargo test -p kebab-embed-local +cargo clippy -p kebab-embed-local --all-targets -- -D warnings +``` + +- [ ] **Step 5: Commit** + +```bash +git add crates/kebab-embed-local/src/lib.rs +git commit -m "feat(embed): add multilingual-e5-large arm to resolve_model (fb-39b)" +``` + +--- + +## Task 2: Flip kebab-config default to e5-large + 1024 dim + +**Files:** +- Modify: `crates/kebab-config/src/lib.rs` + +- [ ] **Step 1: Read existing default test + value sites** + +```bash +grep -n "multilingual-e5-small\|dimensions: 384\|dimensions = 384\|default.*embedding" crates/kebab-config/src/lib.rs +``` + +Three sites to update: +- `Config::defaults()` body (~line 307): `dimensions: 384` and `model: "multilingual-e5-small"`. +- Default-assert test (~line 767): `assert_eq!(c.models.embedding.dimensions, 384)` and likely a sibling assertion on model. +- TOML template at ~line 952: `dimensions = 384` (and likely `model = "multilingual-e5-small"`). + +- [ ] **Step 2: Add failing assertion to existing default test** + +Find the test at ~line 763-768 (likely `defaults_match_design_64_score_gate` or similar). Read it: + +```bash +sed -n '760,780p' crates/kebab-config/src/lib.rs +``` + +If the test asserts `dimensions == 384`, change to `1024`. If it doesn't assert model name, add: + +```rust + assert_eq!(c.models.embedding.model, "multilingual-e5-large"); + assert_eq!(c.models.embedding.dimensions, 1024); +``` + +- [ ] **Step 3: Run tests — expect failure** + +```bash +cargo test -p kebab-config defaults_match +``` +Expected: assertion failure on dimensions == 1024 (still 384) and/or model name. + +- [ ] **Step 4: Flip the defaults** + +In `crates/kebab-config/src/lib.rs:307` (the `EmbeddingCfg` defaults block): + +```rust +EmbeddingCfg { + provider: "fastembed".to_string(), + model: "multilingual-e5-large".to_string(), + version: "v1".to_string(), + dimensions: 1024, + // ... preserve other fields (batch_size etc.) ... +} +``` + +(Read the surrounding lines first to confirm field names — if `version` field doesn't exist or has a different shape, only update `model` + `dimensions`.) + +- [ ] **Step 5: Flip the TOML template** + +In `crates/kebab-config/src/lib.rs` near line 952, the multi-line raw string contains the example TOML config. Find: + +```toml +[models.embedding] +provider = "fastembed" +model = "multilingual-e5-small" +... +dimensions = 384 +``` + +Replace with `model = "multilingual-e5-large"` and `dimensions = 1024`. + +- [ ] **Step 6: Run tests — pass** + +```bash +cargo test -p kebab-config +cargo clippy -p kebab-config --all-targets -- -D warnings +``` + +- [ ] **Step 7: Commit** + +```bash +git add crates/kebab-config/src/lib.rs +git commit -m "feat(config): default embedding model multilingual-e5-large + 1024 dim (fb-39b)" +``` + +--- + +## Task 3: Cross-crate test fixture sweep + +**Files:** +- Modify: any test fixture broken by Task 2's default flip. + +- [ ] **Step 1: Find broken sites** + +```bash +cargo build --workspace 2>&1 | tail -10 +cargo test --workspace --no-run 2>&1 | grep -E "error\[|FAILED" | head -20 +``` + +Likely candidates: +- `crates/kebab-app/tests/` — anywhere a test asserted `embedding.dimensions == 384`. +- `crates/kebab-cli/tests/cli_schema.rs` — a capability/model assertion may include the embedding model name. + +For each failure, decide: +- **Pin to small intentionally** (test exercises small-specific behavior): set `cfg.models.embedding.model = "multilingual-e5-small"; cfg.models.embedding.dimensions = 384;` explicitly. +- **Inherit new default** (test just snapshots defaults): update assertion to `multilingual-e5-large` / `1024`. + +The vast majority of integration tests use `provider = "none"` (no embeddings) — those are unaffected. + +- [ ] **Step 2: Verify workspace builds** + +```bash +cargo build --workspace 2>&1 | tail -5 +``` + +- [ ] **Step 3: Run workspace tests** + +```bash +cargo test --workspace --no-fail-fast -j 1 2>&1 | tail -10 +cargo clippy --workspace --all-targets -- -D warnings 2>&1 | tail -5 +``` + +`-j 1` REQUIRED. + +Expected: all green. + +- [ ] **Step 4: Commit** + +```bash +git add crates/ +git commit -m "fix(fb-39b): update test fixtures for embedding default flip" +``` + +(Skip this commit if `cargo build --workspace` is already clean after Task 2 — meaning no fixture broke.) + +--- + +## Task 4: Wire schema docs (design + HOTFIXES + new task spec) + +**Files:** +- Modify: `docs/superpowers/specs/2026-04-27-kebab-final-form-design.md` +- Modify: `tasks/HOTFIXES.md` +- Create: `tasks/p9/p9-fb-39b-embedding-upgrade.md` +- Modify: `tasks/p9/p9-fb-39-retrieval-precision-tuning.md` +- Modify: `tasks/INDEX.md` + +- [ ] **Step 1: Update design §5 storage and §9 versioning** + +```bash +grep -n "multilingual-e5-small\|^## §5\|^### §5\|^## §9\|384" docs/superpowers/specs/2026-04-27-kebab-final-form-design.md | head -10 +``` + +Update any reference to `multilingual-e5-small` or `dim 384` in the design doc to read `multilingual-e5-large` and `dim 1024`. Keep historical version mentions intact (e.g. "0.6.0 shipped with multilingual-e5-small") if any — but the "current default" line must reflect the new model. + +- [ ] **Step 2: Add HOTFIXES entry** + +Append to `tasks/HOTFIXES.md` (under the dated log; place at top of the dated entries with today's date `2026-05-10`): + +```markdown +- **2026-05-10 fb-39b — embedding upgrade UX**: default embedding flipped from `multilingual-e5-small` (384 dim) to `multilingual-e5-large` (1024 dim). LanceDB tables are namespaced by `(model, dim)` so the new model writes to a fresh table and the old `chunk_embeddings_multilingual-e5-small_384` table becomes orphan. fb-23 incremental ingest auto-re-embeds chunks (embedding_version mismatch) into the new table on next `kebab ingest`. To free disk before re-ingest, run `kebab reset --vector-only` first — this wipes both LanceDB and the SQLite `embedding_records` table. Search/ask against the new model returns empty hits until `kebab ingest` populates the new table. +``` + +- [ ] **Step 3: Create `tasks/p9/p9-fb-39b-embedding-upgrade.md`** + +Mirror the fb-39 frontmatter shape: + +```markdown +--- +phase: P9 +component: kebab-embed-local + kebab-config + kebab-store-vector + docs +task_id: p9-fb-39b +title: "Embedding model upgrade (multilingual-e5-large)" +status: completed +target_version: 0.7.0 +depends_on: [p9-fb-39] +unblocks: [] +contract_source: ../../docs/superpowers/specs/2026-04-27-kebab-final-form-design.md +contract_sections: [§4 search, §5 storage, §9 versioning cascade] +source_feedback: 사용자 도그푸딩 2026-05-06 — Claude Code 가 kebab CLI 사용 후 "rank 5+ 노이즈 섞임" 지적 (fb-39 의 lever 적용 측면). +--- + +# p9-fb-39b — Embedding model upgrade + +> ✅ **구현 완료.** fb-39 의 lever 후보 4개 중 embedding model 업그레이드 lever 적용. P@k metric (fb-39) 으로 small vs large 비교 가능. +> +> - Design: [`docs/superpowers/specs/2026-05-10-p9-fb-39b-embedding-upgrade-design.md`](../../docs/superpowers/specs/2026-05-10-p9-fb-39b-embedding-upgrade-design.md) +> - Plan: [`docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md`](../../docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md) + +## 요약 + +- `multilingual-e5-small` (384 dim) → `multilingual-e5-large` (1024 dim) default flip. +- 기존 user TOML 이 small 명시 시 그대로 (backwards-compat). +- fb-23 incremental ingest 가 embedding_version mismatch 감지 → 자동 re-embed. +- 0.6 → 0.7 minor bump 트리거 (design §9 cascade rule). +``` + +- [ ] **Step 4: Append fb-39b note to fb-39 task spec banner** + +In `tasks/p9/p9-fb-39-retrieval-precision-tuning.md`, find the existing `> ✅ **Eval foundation 부분 구현 완료.**` banner. Append a line: + +```markdown +> - fb-39b (lever 적용 — embedding upgrade): [`tasks/p9/p9-fb-39b-embedding-upgrade.md`](./p9-fb-39b-embedding-upgrade.md) ✅ +``` + +- [ ] **Step 5: Add fb-39b row to INDEX** + +In `tasks/INDEX.md`, find the fb-39 row. Add a sibling row immediately below: + +```markdown + - [p9-fb-39b embedding upgrade](p9/p9-fb-39b-embedding-upgrade.md) — ✅ 머지 (2026-05-10) — multilingual-e5-large default +``` + +(Adapt format to match neighbor rows.) + +- [ ] **Step 6: Workspace test + clippy gate** + +```bash +cargo test --workspace --no-fail-fast -j 1 2>&1 | tail -10 +cargo clippy --workspace --all-targets -- -D warnings 2>&1 | tail -5 +``` + +`-j 1` REQUIRED. + +- [ ] **Step 7: Commit** + +```bash +git add docs/ tasks/ +git commit -m "docs(fb-39b): design + HOTFIXES + new task spec + INDEX" +``` + +--- + +## Task 5: README + SMOKE walkthrough + +**Files:** +- Modify: `README.md` +- Modify: `docs/SMOKE.md` + +- [ ] **Step 1: Update README `[models.embedding]` section** + +```bash +grep -n "models.embedding\|multilingual-e5-small\|fastembed" README.md | head -5 +``` + +Locate the `[models.embedding]` config block in README. Update default values mentioned + add new bullet: + +```markdown +- `model` (default `"multilingual-e5-large"`, fb-39b) — 다국어 sentence embedding 모델. 1024-dim. ONNX (~1.3 GB) 첫 실행 시 fastembed cache (`config.storage.model_dir/fastembed/`) 에 자동 다운로드. `"multilingual-e5-small"` (384 dim) 는 backwards-compat 으로 사용 가능 — TOML 에 명시. +- `dimensions` (default `1024`) — 모델의 embedding 차원. config 와 LanceDB stored dim 불일치 시 검색 결과 0 건 (orphan table). 모델 변경 시 `kebab reset --vector-only && kebab ingest` 로 vector index 재구축 권장. +``` + +- [ ] **Step 2: Append SMOKE walkthrough** + +Append to `docs/SMOKE.md` after fb-39 section (or at end if absent): + +````markdown +### Embedding upgrade (fb-39b) + +`multilingual-e5-small` 에서 `multilingual-e5-large` 로 업그레이드 시퀀스: + +```bash +# 기존 vector index 정리 (orphan table 회피) +kebab --config /tmp/kebab-smoke/config.toml reset --vector-only + +# config.toml 의 [models.embedding] 갱신: +# model = "multilingual-e5-large" +# dimensions = 1024 + +# 재-ingest — fastembed 가 첫 실행 시 e5-large ONNX (~1.3 GB) 자동 다운로드. +# 다운로드 시간 + 모든 chunk re-embed 시간 (e5-small 대비 ~3-4×). +kebab --config /tmp/kebab-smoke/config.toml ingest + +# fb-39 의 P@k metric 으로 small vs large 비교: +kebab --config /tmp/kebab-smoke/config.toml eval run +``` +```` + +- [ ] **Step 3: Workspace test + clippy gate (sanity)** + +```bash +cargo test --workspace --no-fail-fast -j 1 2>&1 | tail -5 +cargo clippy --workspace --all-targets -- -D warnings 2>&1 | tail -3 +``` + +- [ ] **Step 4: Commit** + +```bash +git add README.md docs/SMOKE.md +git commit -m "docs(fb-39b): README + SMOKE — embedding upgrade walkthrough" +``` + +--- + +## Final verification checklist + +- [ ] `cargo test --workspace --no-fail-fast -j 1` green +- [ ] `cargo clippy --workspace --all-targets -- -D warnings` clean +- [ ] `kebab schema --json | jq .models.embedding_version` reflects new model name (after a fresh ingest with new defaults) +- [ ] Manual smoke: `kebab reset --vector-only && kebab ingest` against `/tmp/kebab-smoke` triggers ONNX download (first run) then completes ingest into the new `chunk_embeddings_multilingual-e5-large_1024` table +- [ ] README + SMOKE + design + HOTFIXES + fb-39b spec + INDEX all updated +- [ ] **Post-merge**: cut version bump 0.6 → 0.7 + tag (CLAUDE.md `Versioning cascade` release rule — embedding_version cascade triggers minor bump) diff --git a/docs/superpowers/specs/2026-04-27-kebab-final-form-design.md b/docs/superpowers/specs/2026-04-27-kebab-final-form-design.md index 199fcc9..666e79f 100644 --- a/docs/superpowers/specs/2026-04-27-kebab-final-form-design.md +++ b/docs/superpowers/specs/2026-04-27-kebab-final-form-design.md @@ -93,7 +93,7 @@ retrieval trace grounded ✓ qwen2.5:14b-instruct rag-v1 3 chunks prompt 1184 tokens completion 312 tokens latency 1842 ms -embedding multilingual-e5-small index v1.0 +embedding multilingual-e5-large index v1.0 ``` ### 1.3 `kebab ask` (refusal — score gate) @@ -212,7 +212,7 @@ variant 별 해당 키만 채움. `path` 와 `uri` 는 항상 채움 (`uri` 는 "vector_rank": 2 }, "index_version": "v1.0", - "embedding_model": "multilingual-e5-small", + "embedding_model": "multilingual-e5-large", "chunker_version": "md-heading-v1" } ``` @@ -264,7 +264,7 @@ Per-query failure 는 `bulk_search_item.v1.error` (error.v1) 에 격리, 다른 "grounded": true, "refusal_reason": null, "model": { "id": "qwen2.5:14b-instruct", "provider": "ollama" }, - "embedding": { "id": "multilingual-e5-small", "provider": "fastembed", "dimensions": 384 }, + "embedding": { "id": "multilingual-e5-large", "provider": "fastembed", "dimensions": 1024 }, "prompt_template_version": "rag-v1", "retrieval": { "trace_id": "ret_4a8b2c1e", @@ -374,7 +374,7 @@ Per-query failure 는 `bulk_search_item.v1.error` (error.v1) 에 격리, 다른 "token_estimate": 480, "chunker_version": "md-heading-v1", "embeddings": [ - { "model": "multilingual-e5-small", "dimensions": 384, "embedding_id": "e_2f1a" } + { "model": "multilingual-e5-large", "dimensions": 1024, "embedding_id": "e_2f1a" } ] } ``` @@ -390,7 +390,7 @@ Per-query failure 는 `bulk_search_item.v1.error` (error.v1) 에 격리, 다른 { "name": "data_dir_writable", "ok": true, "detail": "~/.local/share/kebab" }, { "name": "sqlite_open", "ok": true, "detail": "kebab.sqlite (schema v1)" }, { "name": "lancedb_open", "ok": true, "detail": "lancedb/" }, - { "name": "embedding_model", "ok": true, "detail": "multilingual-e5-small (384d)" }, + { "name": "embedding_model", "ok": true, "detail": "multilingual-e5-large (1024d)" }, { "name": "ollama_reachable", "ok": true, "detail": "http://127.0.0.1:11434" }, { "name": "ollama_model_pulled", "ok": false, "detail": "qwen2.5:14b-instruct missing", "hint": "ollama pull qwen2.5:14b-instruct" } ] @@ -1209,9 +1209,9 @@ chunker_version = "md-heading-v1" [models.embedding] provider = "fastembed" -model = "multilingual-e5-small" +model = "multilingual-e5-large" version = "v1" -dimensions = 384 +dimensions = 1024 batch_size = 64 [models.llm] @@ -1474,7 +1474,7 @@ $ kebab doctor ✓ data_dir_writable ~/.local/share/kebab ✓ sqlite_open kebab.sqlite (schema v1) ✓ lancedb_open lancedb/ -✓ embedding_model multilingual-e5-small (384d) +✓ embedding_model multilingual-e5-large (1024d) ✓ ollama_reachable http://127.0.0.1:11434 ✗ ollama_model_pulled qwen2.5:14b-instruct missing hint: ollama pull qwen2.5:14b-instruct diff --git a/docs/superpowers/specs/2026-05-10-p9-fb-39b-embedding-upgrade-design.md b/docs/superpowers/specs/2026-05-10-p9-fb-39b-embedding-upgrade-design.md new file mode 100644 index 0000000..23108b7 --- /dev/null +++ b/docs/superpowers/specs/2026-05-10-p9-fb-39b-embedding-upgrade-design.md @@ -0,0 +1,198 @@ +--- +title: "p9-fb-39b — Embedding model upgrade design (multilingual-e5-large)" +phase: P9 +component: kebab-embed-local + kebab-store-vector + kebab-config + kebab-app +task_id: p9-fb-39b +status: design +target_version: 0.7.0 +contract_source: ../../docs/superpowers/specs/2026-04-27-kebab-final-form-design.md +contract_sections: [§4 search, §5 storage, §9 versioning cascade] +date: 2026-05-10 +--- + +# p9-fb-39b — Embedding model upgrade + +## Goal + +fb-39 의 lever 적용 — embedding model 을 `multilingual-e5-small` (384 dim) 에서 `multilingual-e5-large` (1024 dim) 로 업그레이드. 도그푸딩 한국어 corpus 의 retrieval precision 개선. + +fb-39 가 측정 도구 (P@5 / P@10) 를 추가했으므로, 본 PR 머지 후 small vs large 비교 가능. + +`bge-m3` 검토했으나 fastembed 4.9.1 의 `EmbeddingModel` enum 에 미포함 — `UserDefinedEmbeddingModel` ONNX 직접 로드 path 는 별도 작업 (fb-39c 후보). 본 PR scope = e5-large 만. + +## Behavior contract + +### Embedding model + +- 신규 default: `multilingual-e5-large` (1024 dim). +- `kebab-embed-local::resolve_model` 에 신규 arm: + +```rust +"multilingual-e5-large" => Ok(EmbeddingModel::MultilingualE5Large), +``` + +기존 `multilingual-e5-small` arm 그대로 (backwards-compat opt-out). + +### Config defaults + +- `Config::defaults().models.embedding.model`: `"multilingual-e5-small"` → `"multilingual-e5-large"`. +- `Config::defaults().models.embedding.dimensions`: `384` → `1024`. +- `kebab init` 가 생성하는 config.toml 템플릿 동일 갱신. + +기존 user TOML 이 `model = "multilingual-e5-small"` 또는 `dimensions = 384` 명시한 경우 그대로 유지 — `serde` 가 user value 우선. opt-out 가능. + +### Cascade + +- `embedding_version`: 자동 변경 (config.models.embedding.model 값 그대로 wire 에 emit). `multilingual-e5-small` → `multilingual-e5-large`. +- fb-23 incremental ingest: 4-input match (blake3 + parser_version + chunker_version + embedding_version) 에서 embedding_version 깨짐 → 모든 chunk 재-embed. text/parse/chunk 비용 회피, embed 비용만 발생. +- `eval_runs.config_snapshot_json`: 새 version 자동 기록. 비교 시 동일 version 끼리. +- design §9 cascade rule 의 5 키 중 `embedding_version` 변경 — binary release 트리거 (CLAUDE.md `Versioning cascade` 룰). + +### Migration policy + +LanceDB stored vectors 의 dim 과 `config.models.embedding.dimensions` 가 mismatch 면: + +- `LanceVectorStore::open` (또는 첫 호출) 가 비교 → mismatch 시 신규 `ErrorV1`: + - `code = "embedding_dim_mismatch"` + - `message`: `"vector index dim 384 vs config dim 1024"` + - `hint`: `"기존 vector index 가 4-dim, config 는 N-dim. 'kebab reset --vector-only && kebab ingest' 로 재구축."` +- CLI: exit 1 + error.v1 stderr (또는 비-`--json` 모드 plain stderr). +- silent migration / auto-wipe 안 함 — 사용자 명시 동의 필요. + +remediation flow: + +``` +$ kebab search "..." +error: vector index dim 384 vs config dim 1024 + +Hint: 기존 vector index 가 384-dim, config 는 1024-dim. +'kebab reset --vector-only && kebab ingest' 로 재구축. + +$ kebab reset --vector-only +[wipe LanceDB + SQLite embedding_records] + +$ kebab ingest +[full re-embed with new model — fastembed downloads e5-large ONNX (~1.3 GB) on first run] +``` + +### Wire shape + +신규 wire field 없음. `error.v1.code` 의 valid value namespace 에 `"embedding_dim_mismatch"` 추가 (string, enum 아님 — additive). + +## Allowed / forbidden dependencies + +- `kebab-embed-local`: 신규 dep 없음. fastembed enum variant 추가만. +- `kebab-store-vector`: 신규 dep 없음. LanceDB schema reader 사용. +- `kebab-config`: 신규 dep 없음. defaults 값 변경. +- `kebab-app`: 신규 dep 없음. error propagation. + +`kebab-core` 의 다른 `kebab-*` 의존 금지 룰 그대로. + +## Public surface delta + +### kebab-embed-local (`lib.rs`) + +```rust +fn resolve_model(name: &str) -> Result { + match name { + "multilingual-e5-small" => Ok(EmbeddingModel::MultilingualE5Small), + "multilingual-e5-large" => Ok(EmbeddingModel::MultilingualE5Large), // 신규 + other => anyhow::bail!(/* ... */), + } +} +``` + +### kebab-config (defaults + TOML 템플릿) + +```rust +EmbeddingCfg { + provider: "fastembed".to_string(), + model: "multilingual-e5-large".to_string(), + dimensions: 1024, + // ... 기타 ... +} +``` + +generated config.toml 템플릿 도 같이 갱신. + +### kebab-store-vector (`lib.rs` 또는 신규 helper) + +```rust +impl LanceVectorStore { + pub fn open(...) -> Result { + // 기존 open 로직 ... + let stored_dim = read_schema_vector_dim(&table)?; + if stored_dim != config_dim { + anyhow::bail!(StructuredError(ErrorV1 { + code: "embedding_dim_mismatch".to_string(), + message: format!("vector index dim {stored_dim} vs config dim {config_dim}"), + hint: Some(format!( + "기존 vector index 가 {stored_dim}-dim, config 는 {config_dim}-dim. \ + 'kebab reset --vector-only && kebab ingest' 로 재구축." + )), + // ... + })); + } + Ok(...) + } +} +``` + +(정확한 LanceDB schema reading API 는 구현 시 확인 — `Table::schema()` 또는 `arrow_schema::Schema` 직접 inspect.) + +## Test plan + +| kind | description | +|------|-------------| +| unit (kebab-embed-local) | `resolve_model("multilingual-e5-large")` returns Ok | +| unit (kebab-embed-local) | `check_dim(1024, 1024)` ok | +| unit (kebab-embed-local) | `check_dim(384, 1024)` Err — message mentions both dims | +| unit (kebab-config) | `Config::defaults().models.embedding.model == "multilingual-e5-large"` | +| unit (kebab-config) | `Config::defaults().models.embedding.dimensions == 1024` | +| unit (kebab-config) | TOML `model = "multilingual-e5-small"` deserialize 정상 (backwards-compat) | +| unit (kebab-config) | 생성된 config.toml 템플릿 안 `model = "multilingual-e5-large"`, `dimensions = 1024` | +| unit (kebab-store-vector) | mismatch fixture (384-dim stored + 1024 cfg) → `embedding_dim_mismatch` ErrorV1 | +| 통합 (kebab-cli) | mismatch scenario — pre-existing 384-dim DB + new config → exit 1 + error.v1 stderr (`code = embedding_dim_mismatch`) + hint mentions reset --vector-only | +| 통합 (kebab-cli) | small config 로 fresh ingest + search → 정상 (backwards-compat path 검증) | + +`multilingual-e5-large` 모델 다운로드 회피 위해 unit/integration 테스트는 fixture 또는 mock — 실 모델 호출 안 함. 첫 도그푸딩 시 사용자가 fastembed cache 다운로드. + +## Implementation steps (high-level) + +1. `kebab-embed-local::resolve_model` arm + check_dim 단위 테스트. +2. `kebab-store-vector` dim mismatch detection + ErrorV1 + 단위 테스트. +3. `kebab-config` defaults flip + TOML 템플릿 + 단위 테스트. +4. `kebab-cli` integration: mismatch error.v1 wire + backwards-compat path 통합 테스트. +5. README + SMOKE + design + HOTFIXES + status flip. + +5 task. 단일 PR, single 세션 가능. + +## Risks / notes + +- **첫 실행 모델 다운로드**: e5-large ONNX ~1.3 GB. fastembed cache (`config.storage.model_dir/fastembed/`) 에 자동 다운로드 (첫 호출 시). progress 표시 없음 — 사용자 침묵 latency. `kebab doctor` 또는 README 에 경고 안내. +- **Search/ingest latency**: e5-large 가 e5-small 대비 ~3-4× embedding 시간. ingest 비용 증가 (one-time + 신규 docs). search 시 query embed per-call 증가. +- **Disk usage**: vector dim 2.6× → LanceDB 약 2.7× 증가. +- **HOTFIXES entry**: dim mismatch UX (error.v1 + reset --vector-only flow) 가 frozen design 안 명시 안 된 신규 동작 — HOTFIXES 한 항목 추가. +- **eval comparison**: fb-39 P@k 가 측정 도구. 도그푸딩 corpus + golden 의 expected_chunk_ids 채워서 small vs large 정량 비교 별도 (PR 안 의무 아님). +- **fb-23 incremental ingest 와의 상호작용**: embedding_version 변경 → 모든 doc 재-embed. fb-23 의 unchanged path 는 한 번도 hit 안 함 (예상 동작). +- **release trigger**: design §9 cascade rule 의 `embedding_version` 변경 → CLAUDE.md `Versioning cascade` 룰에 따라 binary 0.6 → 0.7 minor bump 필요. + +## Out of scope + +- bge-m3 또는 user-defined ONNX path (fb-39c 후보). +- Other lever (RRF / cross-encoder / chunk policy). +- Auto-migration / background re-vector. +- LanceDB schema migration tooling (별도 wipe + re-ingest). +- multi-model coexistence (한 KB 안 small + large 동시). +- precision 정량 비교 의무 (별도 도그푸딩). + +## Documentation updates (implementation PR 동시) + +- `README.md` `[models.embedding]` config 섹션 — default 변경 + small opt-out 안내 + dim mismatch 시 reset 명령 안내. +- `docs/SMOKE.md` — upgrade walkthrough (`kebab reset --vector-only && kebab ingest` 시퀀스 + 첫 ONNX 다운로드 latency 경고). +- `docs/superpowers/specs/2026-04-27-kebab-final-form-design.md` §5 storage / §9 versioning 적절 절 — 새 default + dim 1024 명시. +- `tasks/HOTFIXES.md` — dim mismatch UX entry. +- `tasks/p9/p9-fb-39-retrieval-precision-tuning.md` banner — fb-39b lever 적용 (embedding upgrade) ✅ 추가 (단 spec status 는 fb-39 frozen). +- `tasks/p9/p9-fb-39b-embedding-upgrade.md` 신규 task spec (만들거나, fb-39 sub-task 로 frontmatter 처리). +- `tasks/INDEX.md` — fb-39b 행 추가 ✅. +- 본 PR 머지 후 `chore: bump version 0.6 → 0.7` + tag (CLAUDE.md release 절차). diff --git a/tasks/HOTFIXES.md b/tasks/HOTFIXES.md index 2e69e84..d6c9938 100644 --- a/tasks/HOTFIXES.md +++ b/tasks/HOTFIXES.md @@ -14,6 +14,22 @@ historical contract that was implemented; this file accumulates the deltas so phase 5+ readers can find the live behavior without diffing git history. +## 2026-05-10 — p9-fb-39b: embedding upgrade UX + +**무엇이 바뀌었나**: default embedding 이 `multilingual-e5-small` (384 dim) 에서 `multilingual-e5-large` (1024 dim) 로 변경. LanceDB 테이블은 `(model, dim)` 으로 네임스페이스되어 새 모델은 fresh 테이블에 쓰고, 옛 `chunk_embeddings_multilingual-e5-small_384` 테이블은 orphan 상태 됨. + +**user TOML 에 small 명시한 경우**: backwards-compat 유지. 사용자가 `[models.embedding] model = "multilingual-e5-small"` 로 명시했으면 그대로 small 사용 (새 default 무시). + +**idempotent re-embed**: fb-23 incremental ingest 가 embedding_version mismatch 감지하면 자동으로 이전 chunk 를 새 모델로 re-embed. 다음 `kebab ingest` 호출 시 기존 chunk 의 embedding 을 새 테이블에 재작성. + +**disk 절약**: 이전 모델의 orphan 테이블을 먼저 정리하려면 `kebab reset --vector-only` 실행 (LanceDB + SQLite `embedding_records` 모두 wipe). 이후 `kebab ingest` 가 모든 chunk 를 새 모델로 re-embed 해 새 테이블 채움. + +**search/ask 결과**: re-embed 전까지는 empty hit (새 모델에 데이터가 없음). `kebab ingest` 후 정상 검색 가능. + +**Spec contract 와의 관계**: design §5 (storage) + §9 (versioning cascade) 의 embedding_model.id / dimensions 변경. wire 의 `embedding_version` 필드 (kebab-app schema.v1.models.embedding_version 가 config.models.embedding.model 값을 그대로 emit) 변경 — CLAUDE.md cascade rule 의 release 트리거. 본 PR 머지 후 `chore: bump version 0.5 → 0.6` + tag 필요. + +**Spec deviation**: design `2026-05-10-p9-fb-39b-embedding-upgrade-design.md` 의 §Migration policy + §Public surface delta 가 `LanceVectorStore::open` 안 신규 `error.v1.code = "embedding_dim_mismatch"` 명시했으나 구현 제외. 이유: LanceDB tables 가 `(model, dim)` namespaced — silent orphan + empty-hit 으로 surface (hard error 아님). 명시 error 필요 시 별도 startup health check 작업 필요 (fb-39c 후보 또는 doctor 확장). + ## 2026-05-09 — p9-fb-34: search wire wrapped in search_response.v1 **무엇이 바뀌었나**: `kebab search --json` stdout 이 기존 `search_hit.v1[]` 배열에서 신규 `search_response.v1` object 로 교체. wrapper 가 `hits`, `next_cursor`, `truncated` 세 필드를 가짐. diff --git a/tasks/INDEX.md b/tasks/INDEX.md index 912b9f0..7a874b3 100644 --- a/tasks/INDEX.md +++ b/tasks/INDEX.md @@ -130,6 +130,7 @@ P0~P5 는 직렬. P6~P9 는 P5 이후 병렬 가능. ### 🎯 0.5.0 — RAG quality (cascade 동반: V00X + reindex) - [p9-fb-38 score semantics](p9/p9-fb-38-score-semantics.md) — ✅ 머지 (2026-05-10) - [p9-fb-39 retrieval precision 튜닝](p9/p9-fb-39-retrieval-precision-tuning.md) — ✅ 머지 (2026-05-10) — eval foundation only, lever 적용 deferred + - [p9-fb-39b embedding upgrade](p9/p9-fb-39b-embedding-upgrade.md) — ✅ 머지 (2026-05-10) — multilingual-e5-large default - [p9-fb-40 fact-grounded answer](p9/p9-fb-40-fact-grounded-answer.md) — ✅ 머지 (2026-05-10) ### 🎯 0.6.0 또는 P+ — reasoning diff --git a/tasks/p9/p9-fb-39-retrieval-precision-tuning.md b/tasks/p9/p9-fb-39-retrieval-precision-tuning.md index 7f9641e..166a6b0 100644 --- a/tasks/p9/p9-fb-39-retrieval-precision-tuning.md +++ b/tasks/p9/p9-fb-39-retrieval-precision-tuning.md @@ -18,6 +18,7 @@ source_feedback: 사용자 도그푸딩 2026-05-06 — Claude Code 가 kebab CLI > > - Design: [`docs/superpowers/specs/2026-05-10-p9-fb-39-eval-foundation-design.md`](../../docs/superpowers/specs/2026-05-10-p9-fb-39-eval-foundation-design.md) > - Plan: [`docs/superpowers/plans/2026-05-10-p9-fb-39-eval-foundation.md`](../../docs/superpowers/plans/2026-05-10-p9-fb-39-eval-foundation.md) +> - fb-39b (lever 적용 — embedding upgrade): [`tasks/p9/p9-fb-39b-embedding-upgrade.md`](./p9-fb-39b-embedding-upgrade.md) ✅ ## 증상 / 동기 diff --git a/tasks/p9/p9-fb-39b-embedding-upgrade.md b/tasks/p9/p9-fb-39b-embedding-upgrade.md new file mode 100644 index 0000000..b89c2e0 --- /dev/null +++ b/tasks/p9/p9-fb-39b-embedding-upgrade.md @@ -0,0 +1,76 @@ +--- +phase: P9 +component: kebab-embed-local + kebab-config + kebab-store-vector + docs +task_id: p9-fb-39b +title: "Embedding model upgrade (multilingual-e5-large)" +status: completed +target_version: 0.6.0 +depends_on: [p9-fb-39] +unblocks: [] +contract_source: ../../docs/superpowers/specs/2026-04-27-kebab-final-form-design.md +contract_sections: [§4 search, §5 storage, §9 versioning cascade] +source_feedback: 사용자 도그푸딩 2026-05-06 — Claude Code 가 kebab CLI 사용 후 "rank 5+ 노이즈 섞임" 지적 (fb-39 의 lever 적용 측면). +--- + +# p9-fb-39b — Embedding model upgrade + +> ✅ **구현 완료.** fb-39 의 lever 후보 4개 중 embedding model 업그레이드 lever 적용. P@k metric (fb-39) 으로 small vs large 비교 가능. +> +> - Design update: [`docs/superpowers/specs/2026-04-27-kebab-final-form-design.md`](../../docs/superpowers/specs/2026-04-27-kebab-final-form-design.md) §5 / §9 +> - Plan: [`docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md`](../../docs/superpowers/plans/2026-05-10-p9-fb-39b-embedding-upgrade.md) + +## 요약 + +- `multilingual-e5-small` (384 dim) → `multilingual-e5-large` (1024 dim) default flip. +- 기존 user TOML 이 small 명시 시 그대로 유지 (backwards-compat). +- fb-23 incremental ingest 가 embedding_version mismatch 감지 → 자동 re-embed. +- 0.5 → 0.6 minor bump 트리거 (design §9 cascade rule, current Cargo.toml = 0.5.0). + +## 구현 항목 + +1. **config defaults flip** — `[models.embedding] model = "multilingual-e5-large"`, `dimensions = 1024`. +2. **fastembed e5-large resolution** — `kebab-embed-local` 의 `resolve_model()` 에 e5-large arm 추가. +3. **fixture sweep** — 모든 unit/integration 테스트의 default embedding 모델 확인. Config 에서 명시하지 않으면 새 default 따름 (`provider = "none"` 테스트 제외). +4. **design contract update** — design §5 (storage example) + §9 (versioning table) 의 embedding_model.id + dimensions 갱신. +5. **HOTFIXES entry** — 사용자 재 ingest 절차 + backwards-compat 동작 명시. +6. **README update** — `[models.embedding]` 섹션의 기본값 + `dimensions` 필드 설명 갱신. +7. **SMOKE.md append** — 스모크 테스트 중 embedding 업그레이드 검증 절차 (reset → config 갱신 → ingest → eval). +8. **tasks/INDEX.md append** — p9-fb-39b row 추가 (p9-fb-39 sibling). + +## Allowed dependencies + +- `kebab-embed-local` — fastembed crate + `kebab-core` +- `kebab-config` — toml crate +- `kebab-store-vector` — lancedb crate (table naming 로직만 영향) +- `kebab-app` — 와이어링만 (API 변경 없음) + +## Forbidden dependencies + +- parse-* crate (parser 무관) +- llm-* crate (embedding 과 무관) +- search crate (검색 로직은 adapter pattern 으로 이미 generic) + +## Test + +- `cargo test -p kebab-embed-local -- e5_large` (새 arm 테스트) +- `cargo test -p kebab-config -- embedding_defaults` (config defaults) +- `cargo test --workspace --no-fail-fast -j 1` (full regression) +- Smoke: `kebab --config /tmp/smoke.toml doctor | grep embedding` → `multilingual-e5-large (1024d)` +- Smoke: `kebab --config /tmp/smoke.toml ingest` → embedding 진행 표시 + dimension check +- P@k eval: `kebab eval run` (fb-39 의 golden set) small vs large 비교 + +## Backward compat notes + +- Pre-fb-39b user 가 config 에서 명시하지 않은 embedding → new default (large) 자동 적용. TOML 에 `model = "multilingual-e5-small"` 명시하면 유지. +- `kebab-config` 의 `EmbeddingCfg.model` 은 String 필드 — TOML 에 명시한 값이 default 를 override (serde 기본 동작). +- Orphan LanceDB table (`chunk_embeddings_multilingual-e5-small_384`) 은 다음 `kebab ingest` 실행 후 stale 취급 — 사용자가 수동 `kebab reset --vector-only` 로 정리 가능. + +## Binary version bump + +- 0.5.0 → 0.6.0 (current Cargo.toml = 0.5.0; embedding_version cascade triggers minor bump per design §9). +- Release notes: `embedding default: multilingual-e5-small (384d) → multilingual-e5-large (1024d), P@k metric ↑`. + +## Post-merge deviation + +- **`embedding_dim_mismatch` ErrorV1 dropped**: design spec §Migration policy 가 `LanceVectorStore::open` 안 dim mismatch 감지 + 신규 `error.v1.code = "embedding_dim_mismatch"` 를 명시했으나 구현에서 제외. 이유: LanceDB tables 가 `(model, dim)` namespaced (`crates/kebab-store-vector/src/paths.rs:21`) — 신규 model 변경 시 새 table 자동 생성, 옛 table orphan. dim mismatch 가 hard error 되지 않고 검색 결과 0건 (silent precision loss) 으로 surface. HOTFIXES 항목이 documentation source. 명시 error 가 의미 있으려면 별도 startup health check 필요 — fb-39c 후보 또는 v0.7.0 의 doctor 확장. +- 영향: 사용자가 model 변경 후 `kebab ingest` 안 하면 검색 결과 0건. README + SMOKE walkthrough 가 reset --vector-only && ingest 시퀀스 안내.