refactor(config): OCR 중복 제거 — 공유 [ingest.ocr] 엔진 블록 + v4→v5 마이그레이션

Phase 2 Unit 1. OcrCfg(image) 13필드가 PdfOcrCfg(pdf) 와 전부 중복(image 고유
0, pdf 고유 4) + apply_env 에 KEBAB_IMAGE_OCR_*/KEBAB_PDF_OCR_* 27 arm 복제를
제거한다.

- 신규 SharedOcrEngineCfg(13 공유 필드, 전부 Option, default None) = [ingest.ocr].
  엔진 설정 단일 출처. image/pdf 블록은 on/off 토글 + override.
- load-time resolution(Config::resolve_ocr, from_file 호출): 공유 필드가 Some 이고
  미디어 블록이 그 키 미명시면 concrete OcrCfg/PdfOcrCfg 로 overlay(presence 는
  toml::Value 로 판정; 미디어 > 공유 > 내장 default). struct 필드는 그대로 두고
  엔진 필드에 #[serde(default)] 만 추가(slim 블록 파싱) → image(gemma4:e4b/1600)
  vs pdf(qwen2.5vl:3b/2048) 미디어별 기본값 보존.
- resolver Config::image_ocr()/pdf_ocr() 추가. consumer(kebab-parse-image,
  kebab-app build_*_ocr_engine·ingest gate·pdf_ocr_apply·ingest_config_signature)가
  전부 경유 → god-struct 직접 read 제거.
- apply_env: 27 arm → 공유 KEBAB_OCR_* 12 arm(image+pdf 동시) + pdf 고유 4 arm +
  미디어별 KEBAB_IMAGE_OCR_ENABLED/KEBAB_PDF_OCR_ENABLED.
- step_4_to_5: [ingest.image.ocr] 12 엔진 키를 [ingest.ocr] 로 move_table(enabled
  제외). pdf 블록 무손상(reconcile 이 채워 공유 overlay 오염 X). annotated_default
  도 동일 통합으로 v5 canonical 형상. CURRENT_SCHEMA_VERSION=5.
- v4→v5 round-trip 테스트(비-default image engine 보존 + pdf 오염 X + 멱등). effective
  OCR 바이트 동일 → ingest_config_signature 불변 → 강제 재색인 없음.

검증: clippy --workspace --all-targets 0 / kebab-config·kebab-parse-image·
kebab-parse-pdf·kebab-app 테스트 pass. surface: README [ingest.ocr] 절 + SMOKE
config 블록 + DOGFOOD env + HOTFIXES dated entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Mc6W1fgsrbFKTsqA6P8La
This commit is contained in:
2026-06-24 11:03:39 +00:00
parent c9a9832f7f
commit da323af87b
15 changed files with 747 additions and 174 deletions

View File

@@ -166,8 +166,9 @@ nli_threshold = 0.0 # >0 (예: 0.5) 면 mDeBERTa XNLI groundedn
- **`[ingest.chunking]`** — 청크 크기·오버랩·heading 존중. `chunker_version` 기본 `"md-heading-v2"` (v0.30.0). **`max_chunk_tokens`** (default 4000, byte/3 토큰) — 이 값을 넘는 청크는 줄(→UTF-8 char) 경계로 분할해 각 조각이 예산 이하가 되게 한다. 거대 list/code/log 덤프가 한 청크로 임베더 컨텍스트를 초과하던 문제를 막는다(미분할 청크는 v0.29.0 `md-heading-v1` 과 출력 동일). 이 값을 바꾸면 markdown 자산이 자동 재청크된다.
- **파생물 캐시** — embedding 결과를 내용 해시로 자동 캐싱한다 (위 「핵심 기능」 참고). 설정 항목 없음.
- **`[ingest.code]`** — code ingest 의 skip 정책 (`skip_generated_header`, `max_file_bytes`, `extra_skip_globs`). `.gitignore` 자동 honor, `.kebabignore` 는 추가 layer.
- **`[ingest.image.ocr]`** — 이미지 OCR (default off / opt-in). `engine` 으로 백엔드 선택: `"ollama-vision"` (default, 원격 vision LM) 또는 `"paddle-onnx"` (PP-OCRv5 ONNX 를 in-process 로 실행, Python 런타임 불필요, 큰 페이지 CPU <4초, 오프라인). `paddle-onnx` 는 워크스페이스에 번들된 모델을 쓰며 `det_model`/`rec_model`/`dict` 로 경로 override, `score_thresh`(0.3)/`unclip_ratio`(1.5)/`max_boxes`(1000) 로 검출 튜닝 가능 (`KEBAB_IMAGE_OCR_*` env 동일 지원 — env 이름은 v3 에서도 불변). engine 또는 모델을 바꾸면 영향 이미지가 자동 재색인된다.
- **`[ingest.pdf.ocr]`** — scanned PDF 의 page-단위 OCR (default off / opt-in, page 당 ~수십 초 cost). `engine``[ingest.image.ocr]` 과 동일하게 `"ollama-vision"`/`"paddle-onnx"` 선택. v3 에서 paddle 모델 경로 키(`det_model`/`rec_model`/`dict`/`score_thresh`/`unclip_ratio`/`max_boxes`)를 PDF 자체적으로 가질 수 있다(`KEBAB_PDF_OCR_*` env 동일). 활성화 후 옛 색인분은 `kebab ingest --force-reingest` 로 재처리.
- **`[ingest.ocr]`** (config schema v5) — image/pdf OCR 가 공유하는 **엔진** 설정의 단일 출처 (`engine`/`model`/`endpoint`/`languages`/`max_pixels`/`request_timeout_secs` + paddle 모델 경로·튜닝 키). 여기에 한 번 적어 두면 image·pdf 양쪽에 적용되고, 각 미디어 블록(`[ingest.image.ocr]`/`[ingest.pdf.ocr]`)이 자기 키로 override 한다 (우선순위: 미디어 블록 > `[ingest.ocr]` > 내장 기본값). 옛 v4 `config.toml` 의 image/pdf 에 중복돼 있던 OCR 엔진 키는 로드 시 자동으로 이 블록으로 통합된다 (effective 값 불변, 자동 재색인 없음). env override 도 `KEBAB_OCR_*` 하나로 통합 (양쪽 미디어에 적용).
- **`[ingest.image.ocr]`** — 이미지 OCR. on/off 토글(`enabled`, default off / opt-in)은 미디어별이며, 엔진 설정은 `[ingest.ocr]` 에서 상속하되 이 블록에서 override 할 수 있다. `engine` 으로 백엔드 선택: `"ollama-vision"` (default, 원격 vision LM) 또는 `"paddle-onnx"` (PP-OCRv5 ONNX 를 in-process 로 실행, Python 런타임 불필요, 큰 페이지 CPU <4초, 오프라인). `paddle-onnx` 는 워크스페이스에 번들된 모델을 쓰며 `det_model`/`rec_model`/`dict` 로 경로 override, `score_thresh`(0.3)/`unclip_ratio`(1.5)/`max_boxes`(1000) 로 검출 튜닝 가능. engine 또는 모델을 바꾸면 영향 이미지가 자동 재색인된다.
- **`[ingest.pdf.ocr]`** — scanned PDF 의 page-단위 OCR (default off / opt-in, page 당 ~수십 초 cost). on/off 토글(`enabled`/`always_on`)과 PDF 고유 키(`valid_ratio_threshold`/`min_char_count`/`lang_hint`)는 미디어별이고, 엔진 설정은 `[ingest.ocr]` 에서 상속하되 이 블록에서 override 한다(PDF 기본 모델은 `qwen2.5vl:3b`, 이미지의 `gemma4:e4b` 와 다름 — 미디어별 기본값 보존). 활성화 후 옛 색인분은 `kebab ingest --force-reingest` 로 재처리.
- **`--config <path>`** — 임시 워크스페이스 / 격리 테스트용 (CLI honor).
- **`kebab config migrate`** — 새 버전에서 추가된 config 섹션을 기존 `config.toml` 에 설명 주석과 함께 채워 넣는다 (사용자가 손본 값·주석·순서는 보존, 멱등, 변경 시 자동 `.bak` 백업). `--dry-run` 으로 변경 미리보기. `kebab doctor` 가 갱신 필요 시 안내한다. `kebab init` 으로 새로 생성되는 config.toml 도 섹션별 주석을 포함한다.
- **`KEBAB_*` env** — 일부 키 override (`KEBAB_RAG_SCORE_GATE`, `KEBAB_EVAL_GOLDEN` 등).

View File

@@ -407,7 +407,7 @@ pub fn ingest_with_config_opts(
// loop is correct and cheap. Construction failure (e.g. invalid
// endpoint) aborts ingest fail-fast — better than silently disabling
// OCR/caption mid-run.
let ocr_engine: Option<Box<dyn OcrEngine>> = if app.config.ingest.image.ocr.enabled {
let ocr_engine: Option<Box<dyn OcrEngine>> = if app.config.image_ocr().enabled {
Some(build_image_ocr_engine(&app.config).context("kb-app::ingest: build image OCR engine")?)
} else {
None
@@ -427,7 +427,7 @@ pub fn ingest_with_config_opts(
// p10 / v0.20 sub-item 1: PDF OCR engine eager init (H-5 resolution).
// image OCR pattern mirror — per-ingest 1회 build, fallible → fail-fast.
let pdf_ocr_engine: Option<Box<dyn OcrEngine>> =
if app.config.ingest.pdf.ocr.enabled || app.config.ingest.pdf.ocr.always_on {
if app.config.pdf_ocr().enabled || app.config.pdf_ocr().always_on {
Some(
build_pdf_ocr_engine(&app.config)
.context("kb-app::ingest: build pdf OCR engine")?,
@@ -892,7 +892,7 @@ type SqliteStoreAlias = kebab_store_sqlite::SqliteStore;
fn build_image_ocr_engine(
config: &kebab_config::Config,
) -> anyhow::Result<Box<dyn OcrEngine>> {
match config.ingest.image.ocr.engine.as_str() {
match config.image_ocr().engine.as_str() {
OLLAMA_VISION_ENGINE => Ok(Box::new(
OllamaVisionOcr::new(config).context("build OllamaVisionOcr")?,
)),
@@ -906,29 +906,26 @@ fn build_image_ocr_engine(
}
}
/// v0.27.0 (T8): build the PDF OCR engine selected by
/// `config.ingest.pdf.ocr.engine`. The ollama-vision arm uses the PDF-specific
/// `model` / `languages` / `max_pixels` / `request_timeout_secs` knobs (and
/// endpoint fallback to `models.llm.endpoint`). The paddle-onnx arm shares
/// the same bundled ONNX models as image OCR (resolved from `image.ocr`
/// overrides) — PaddleOCR is page-agnostic and carries no per-engine prompt.
/// v0.27.0 (T8): build the PDF OCR engine selected by `pdf.ocr.engine`. The
/// ollama-vision arm uses the resolved PDF OCR knobs (`model` / `languages` /
/// `max_pixels` / `request_timeout_secs`, endpoint fallback to
/// `models.llm.endpoint`) from [`Config::pdf_ocr`].
///
/// # Paddle-ONNX asymmetry
/// # Paddle-ONNX assets (v5)
///
/// When `pdf.ocr.engine = "paddle-onnx"`, the model paths and tuning knobs
/// (`det_model`, `rec_model`, `dict`, `score_thresh`, `unclip_ratio`,
/// `max_boxes`, `max_pixels`) are read from **`[image.ocr]`**, not
/// `[pdf.ocr]`. PaddleOCR has no PDF-specific prompt or page-level config;
/// `[pdf.ocr]` fields other than `engine` / `enabled` / `always_on` /
/// `valid_ratio_threshold` / `min_char_count` / `lang_hint` are effectively
/// ignored for the paddle path. This asymmetry is intentional — one set of
/// tuned ONNX knobs serves both image and PDF pages.
/// The paddle-onnx arm still builds via `OnnxPaddleOcr::new(config)`, which
/// resolves its ONNX asset paths from the image OCR block
/// ([`Config::image_ocr`]). After the v5 `[ingest.ocr]` consolidation both
/// mediums inherit the same shared engine defaults, so image and PDF paddle
/// resolve to one identical set of tuned ONNX knobs — the historical
/// "PDF borrows image's paddle assets" behaviour, now expressed as a single
/// shared block rather than a cross-medium read.
fn build_pdf_ocr_engine(
config: &kebab_config::Config,
) -> anyhow::Result<Box<dyn OcrEngine>> {
match config.ingest.pdf.ocr.engine.as_str() {
match config.pdf_ocr().engine.as_str() {
OLLAMA_VISION_ENGINE => {
let cfg = &config.ingest.pdf.ocr;
let cfg = config.pdf_ocr();
let endpoint = match cfg.endpoint.as_deref() {
Some(s) if !s.is_empty() => s.to_string(),
_ => config.models.llm.endpoint.clone(),
@@ -2304,16 +2301,17 @@ fn ingest_one_pdf_asset(
// v0.20 sub-item 1: post-extract OCR enrichment (PR #187 registry
// dispatch invariant 보존 — extract_for 가 normal entry).
let (pdf_ocr_pages, pdf_ocr_ms_total): (Option<u32>, Option<u64>) =
if app.config.ingest.pdf.ocr.enabled || app.config.ingest.pdf.ocr.always_on {
let (pdf_ocr_pages, pdf_ocr_ms_total): (Option<u32>, Option<u64>) = {
let pdf_ocr = app.config.pdf_ocr();
if pdf_ocr.enabled || pdf_ocr.always_on {
match pdf_ocr_engine {
Some(engine) => {
let ocr_opts = crate::pdf_ocr_apply::PdfOcrOpts {
enabled: app.config.ingest.pdf.ocr.enabled || app.config.ingest.pdf.ocr.always_on,
always_on: app.config.ingest.pdf.ocr.always_on,
valid_ratio_threshold: app.config.ingest.pdf.ocr.valid_ratio_threshold,
min_char_count: app.config.ingest.pdf.ocr.min_char_count,
lang_hint: app.config.ingest.pdf.ocr.lang_hint.clone().map(kebab_core::Lang),
enabled: pdf_ocr.enabled || pdf_ocr.always_on,
always_on: pdf_ocr.always_on,
valid_ratio_threshold: pdf_ocr.valid_ratio_threshold,
min_char_count: pdf_ocr.min_char_count,
lang_hint: pdf_ocr.lang_hint.clone().map(kebab_core::Lang),
cancel: cancel.cloned(),
};
// v0.20.x Hook 2: pre-clone Arcs for capture by OCR closure.
@@ -2429,7 +2427,8 @@ fn ingest_one_pdf_asset(
}
} else {
(None, None)
};
}
};
// Per-medium chunker selection: PDF docs always use pdf-page-v1
// regardless of `config.ingest.chunking.chunker_version`. The chunker
@@ -3292,7 +3291,7 @@ fn ingest_config_signature(config: &kebab_config::Config, media: &MediaType) ->
// OCR / caption only affect output when their `enabled` flag is
// on; the model / prompt version matters only then. Off ↔ off is
// a stable empty token so re-running the same config skips.
let ocr = &config.ingest.image.ocr;
let ocr = config.image_ocr();
if ocr.enabled {
// v0.27.0 (T9): engine + engine_version so switching engine
// (ollama-vision ↔ paddle-onnx) OR changing the model/assets
@@ -3321,7 +3320,7 @@ fn ingest_config_signature(config: &kebab_config::Config, media: &MediaType) ->
MediaType::Pdf => {
// PDF OCR is active when EITHER `enabled` or `always_on` is set
// (mirrors the ingest gate). `model` only matters when active.
let ocr = &config.ingest.pdf.ocr;
let ocr = config.pdf_ocr();
if ocr.enabled || ocr.always_on {
// v0.27.0 (T9): engine + engine_version (same cascade rule as
// image OCR above) alongside the enabled/always_on gate.

View File

@@ -2,7 +2,7 @@
//!
//! Tests 1 and 2 require a live Ollama endpoint — `#[ignore]` by default.
//! Manual invoke:
//! KEBAB_PDF_OCR_ENDPOINT=http://192.168.0.47:11434 \
//! KEBAB_OCR_ENDPOINT=http://192.168.0.47:11434 \
//! cargo test -p kebab-app --test ingest_pdf_ocr_smoke --ignored -j 4
//!
//! Test 3 (cancel) uses a dummy endpoint + pre-set cancel — runs by default
@@ -17,7 +17,8 @@ use std::sync::atomic::AtomicBool;
use common::TestEnv;
fn ollama_endpoint() -> String {
std::env::var("KEBAB_PDF_OCR_ENDPOINT").unwrap_or_else(|_| "http://localhost:11434".to_string())
// v5: shared KEBAB_OCR_* env (manual harness reads it directly).
std::env::var("KEBAB_OCR_ENDPOINT").unwrap_or_else(|_| "http://localhost:11434".to_string())
}
fn make_ocr_env_real() -> TestEnv {

View File

@@ -172,7 +172,7 @@ fn dropped_receiver_does_not_panic_or_fail_ingest() {
/// Manual invoke:
/// ```
/// KEBAB_PDF_OCR_ENABLED=true \
/// KEBAB_PDF_OCR_ENDPOINT=http://192.168.0.47:11434 \
/// KEBAB_OCR_ENDPOINT=http://192.168.0.47:11434 \
/// cargo test -p kebab-app --test ingest_progress \
/// --ignored pdf_ocr_progress_emits_started_finished_events
/// ```
@@ -197,7 +197,8 @@ fn pdf_ocr_progress_emits_started_finished_events() {
config.models.embedding.provider = "none".to_string();
config.models.embedding.dimensions = 0;
config.ingest.pdf.ocr.enabled = true;
if let Ok(endpoint) = std::env::var("KEBAB_PDF_OCR_ENDPOINT") {
// v5: shared KEBAB_OCR_* env (manual harness reads it directly).
if let Ok(endpoint) = std::env::var("KEBAB_OCR_ENDPOINT") {
config.ingest.pdf.ocr.endpoint = Some(endpoint);
}

View File

@@ -406,6 +406,78 @@ fn default_nli_threshold() -> f32 {
0.0
}
/// v5: workspace-wide OCR **engine** defaults shared by image and PDF OCR.
///
/// Before v5, the 13 engine-level OCR knobs (engine, model, endpoint,
/// languages, max_pixels, request_timeout_secs + the 6 paddle-onnx asset /
/// tuning fields) were duplicated verbatim across `[ingest.image.ocr]` and
/// `[ingest.pdf.ocr]`. This block holds them ONCE under `[ingest.ocr]`; the
/// per-medium blocks (`[ingest.image.ocr]` / `[ingest.pdf.ocr]`) keep only
/// their `enabled` toggle + any medium-specific override / unique field.
///
/// Resolution happens at load (`Config::resolve_ocr`, called from
/// `from_file`): for every shared key present here but **absent** from a
/// medium's own block, the shared value is merged down into that medium's
/// concrete [`OcrCfg`] / [`PdfOcrCfg`]. A key the user wrote explicitly in
/// the per-medium block always wins (precedence: medium override > shared >
/// medium hardcoded default). Image and PDF therefore keep their distinct
/// hardcoded defaults (`gemma4:e4b`/1600 vs `qwen2.5vl:3b`/2048) when neither
/// block sets the field.
///
/// Every field is `Option` so "unset in `[ingest.ocr]`" is distinguishable
/// from "set to the type's zero value". Default = all `None` (no shared
/// override → each medium uses its own block / hardcoded default), so a
/// `Config` built without a `[ingest.ocr]` section behaves exactly as before.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct SharedOcrEngineCfg {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub enabled: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub engine: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub model: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub endpoint: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub languages: Option<Vec<String>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_pixels: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub request_timeout_secs: Option<u64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub det_model: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub rec_model: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub dict: Option<String>,
#[serde(
default,
skip_serializing_if = "Option::is_none",
serialize_with = "ser_opt_f32_clean"
)]
pub score_thresh: Option<f32>,
#[serde(
default,
skip_serializing_if = "Option::is_none",
serialize_with = "ser_opt_f32_clean"
)]
pub unclip_ratio: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_boxes: Option<usize>,
}
/// `Option<f32>` 직렬화 시 `Some` 값을 [`ser_f32_clean`] 과 같은 shortest
/// round-trip 으로 출력한다(`None` 은 `skip_serializing_if` 가 처리).
fn ser_opt_f32_clean<S>(v: &Option<f32>, s: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match v {
Some(f) => ser_f32_clean(f, s),
None => s.serialize_none(),
}
}
/// Settings for the image ingest pipeline (P6). `ocr` controls OCR
/// behaviour (P6-2); `caption` controls vision-LM captioning (P6-3).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -434,11 +506,19 @@ impl ImageCfg {
pub struct OcrCfg {
/// Run OCR on every image during ingest. Default `false` because
/// OCR adds one model call per asset.
///
/// v5: all engine-level fields carry `#[serde(default)]` so a slim
/// `[ingest.image.ocr]` block (e.g. only `enabled`, with the engine
/// knobs hoisted to the shared `[ingest.ocr]`) still deserializes. The
/// per-field defaults match [`OcrCfg::defaults`] (image medium).
#[serde(default)]
pub enabled: bool,
/// Engine identifier. v1 only ships `"ollama-vision"`.
#[serde(default = "default_ocr_engine")]
pub engine: String,
/// Model id passed to the engine (e.g. `"gemma4:e4b"` for
/// Ollama-vision).
#[serde(default = "default_image_ocr_model")]
pub model: String,
/// HTTP endpoint for the OCR engine. `None` (or a missing key in
/// TOML) means "fall back to `models.llm.endpoint`" — convenient
@@ -447,9 +527,11 @@ pub struct OcrCfg {
pub endpoint: Option<String>,
/// BCP-47 language hints (e.g. `["eng", "kor"]`). The adapter
/// renders them into the prompt; the LLM honours them probabilistically.
#[serde(default = "default_ocr_languages")]
pub languages: Vec<String>,
/// Cap the long edge of the image (in pixels) before sending. Larger
/// images bloat prompt cost. Default `1600`.
#[serde(default = "default_image_ocr_max_pixels")]
pub max_pixels: u32,
/// v0.17.2 post-dogfood: Hard ceiling on a single HTTP exchange to
/// the OCR endpoint. Sister knob to [`LlmCfg::request_timeout_secs`]
@@ -519,6 +601,22 @@ impl OcrCfg {
}
}
/// v5: shared OCR engine-field defaults (image medium), so a slim
/// `[ingest.image.ocr]` block with the engine knobs hoisted to
/// `[ingest.ocr]` still deserializes. Mirror [`OcrCfg::defaults`].
fn default_ocr_engine() -> String {
"ollama-vision".to_string()
}
fn default_ocr_languages() -> Vec<String> {
vec!["eng".to_string(), "kor".to_string()]
}
fn default_image_ocr_model() -> String {
"gemma4:e4b".to_string()
}
fn default_image_ocr_max_pixels() -> u32 {
1600
}
/// paddle-onnx DBNet box score threshold default. See [`OcrCfg::score_thresh`].
fn default_ocr_score_thresh() -> f32 {
0.3
@@ -534,7 +632,8 @@ fn default_ocr_max_boxes() -> usize {
/// v0.17.2 post-dogfood: matches the legacy hard-coded ceiling so
/// existing configs that omit the field keep behaving identically.
/// Overridable per config / `KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS`.
/// Overridable per config / `KEBAB_OCR_REQUEST_TIMEOUT_SECS` (v5: shared
/// engine env, applies to both image and pdf OCR).
fn default_ocr_request_timeout_secs() -> u64 {
300
}
@@ -646,24 +745,37 @@ impl Default for LoggingCfg {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct PdfOcrCfg {
/// Run OCR on scanned PDF pages. Default `false` (opt-in).
///
/// v5: the shared engine fields (engine/model/languages/max_pixels +
/// the request-timeout & paddle knobs) carry `#[serde(default)]` so a
/// slim `[ingest.pdf.ocr]` block (engine knobs hoisted to the shared
/// `[ingest.ocr]`) still deserializes; per-field defaults match the PDF
/// medium ([`PdfOcrCfg::defaults`]).
#[serde(default)]
pub enabled: bool,
/// `false` (default) — text-detect first + vision fallback on
/// scanned pages only. `true` — vision LLM 호출 on every page
/// (vector PDF 의 dual-text confidence boost — doubles chunk count).
#[serde(default)]
pub always_on: bool,
/// Engine identifier: `"ollama-vision"` or `"paddle-onnx"`. When set to
/// `"paddle-onnx"`, model paths and tuning knobs are read from
/// `[image.ocr]`, not `[pdf.ocr]` — PaddleOCR has no PDF-specific tuning.
/// `"paddle-onnx"`, model paths and tuning knobs are read from the shared
/// `[ingest.ocr]` (resolved via the image OCR block) — PaddleOCR has no
/// PDF-specific tuning.
#[serde(default = "default_ocr_engine")]
pub engine: String,
/// Vision model id. Default `"qwen2.5vl:3b"` per PoC (§3.5 family
/// asymmetry vs image OCR's gemma4:e4b is acknowledged).
#[serde(default = "default_pdf_ocr_model")]
pub model: String,
/// HTTP endpoint. `None` → fall back to `models.llm.endpoint`.
#[serde(default)]
pub endpoint: Option<String>,
/// BCP-47 language hints rendered into prompt.
#[serde(default = "default_ocr_languages")]
pub languages: Vec<String>,
/// Long-edge cap (px). Larger images bloat prompt cost.
#[serde(default = "default_pdf_ocr_max_pixels")]
pub max_pixels: u32,
/// HTTP request timeout (sec). Same `0` = "fail immediately"
/// semantics as `image.ocr.request_timeout_secs` (NOT a disable
@@ -757,6 +869,15 @@ fn default_pdf_ocr_min_char_count() -> u32 {
fn default_pdf_ocr_lang_hint() -> Option<String> {
Some("kor".to_string())
}
/// v5: PDF-medium engine-field defaults (distinct from image: qwen2.5vl:3b /
/// 2048 px), so a slim `[ingest.pdf.ocr]` block deserializes. Mirror
/// [`PdfOcrCfg::defaults`].
fn default_pdf_ocr_model() -> String {
"qwen2.5vl:3b".to_string()
}
fn default_pdf_ocr_max_pixels() -> u32 {
2048
}
/// p9-fb-14: TUI-only configuration. Currently a single `theme`
/// selector (`"dark"` / `"light"`); future fields (custom role
@@ -795,6 +916,11 @@ pub struct IngestCfg {
pub chunking: ChunkingCfg,
#[serde(default)]
pub code: IngestCodeCfg,
/// v5: shared OCR engine defaults (`[ingest.ocr]`). Merged down into
/// `image.ocr` / `pdf.ocr` at load by [`Config::resolve_ocr`]. Empty
/// (all `None`) by default — see [`SharedOcrEngineCfg`].
#[serde(default)]
pub ocr: SharedOcrEngineCfg,
#[serde(default = "ImageCfg::defaults")]
pub image: ImageCfg,
#[serde(default = "PdfCfg::defaults")]
@@ -809,6 +935,7 @@ impl Default for IngestCfg {
watch_filesystem: false,
chunking: ChunkingCfg::defaults(),
code: IngestCodeCfg::default(),
ocr: SharedOcrEngineCfg::default(),
image: ImageCfg::defaults(),
pdf: PdfCfg::defaults(),
}
@@ -916,6 +1043,7 @@ impl Config {
watch_filesystem: false,
chunking: ChunkingCfg::defaults(),
code: IngestCodeCfg::default(),
ocr: SharedOcrEngineCfg::default(),
image: ImageCfg::defaults(),
pdf: PdfCfg::defaults(),
},
@@ -1174,6 +1302,13 @@ impl Config {
cause: format!("parse_failed: {e}"),
})
})?;
// v5: merge `[ingest.ocr]` shared engine defaults down into the
// per-medium concrete OCR blocks. Driven by toml-level presence
// (a key explicitly written in `[ingest.image.ocr]` /
// `[ingest.pdf.ocr]` wins over the shared block) — so we hand the
// parsed `toml::Value` (which preserves presence) to the resolver.
let parsed_value = toml::from_str::<toml::Value>(&parse_text).ok();
cfg.resolve_ocr(parsed_value.as_ref());
cfg.validate_sources().map_err(|cause| {
anyhow::Error::new(ConfigInvalid {
path: path.to_path_buf(),
@@ -1261,6 +1396,192 @@ impl Config {
Ok(())
}
/// v5: merge the shared `[ingest.ocr]` engine block down into the
/// per-medium concrete OCR structs (`ingest.image.ocr` /
/// `ingest.pdf.ocr`). For each shared field that is `Some`, the value is
/// written into a medium's block **only if** that medium did not set the
/// field explicitly in its own table. Presence is read from `parsed`
/// (the raw `toml::Value` of the loaded file) because the typed struct
/// has already absorbed serde defaults and can no longer distinguish
/// "user wrote the default" from "omitted".
///
/// `parsed = None` (programmatic config, no source text) → only the
/// "explicitly set" guard is unavailable, but in that path
/// `ingest.ocr` is whatever the caller built; the common case
/// (`SharedOcrEngineCfg::default()` = all `None`) is a no-op. Idempotent.
pub(crate) fn resolve_ocr(&mut self, parsed: Option<&toml::Value>) {
// No shared overrides → nothing to merge (the overwhelming common
// case: configs without an `[ingest.ocr]` block).
if self.ingest.ocr == SharedOcrEngineCfg::default() {
return;
}
let shared = self.ingest.ocr.clone();
// Helper: was `key` explicitly present in `ingest.<medium>.ocr`?
let medium_has = |medium: &str, key: &str| -> bool {
parsed
.and_then(|v| v.get("ingest"))
.and_then(|v| v.get(medium))
.and_then(|v| v.get("ocr"))
.and_then(|v| v.get(key))
.is_some()
};
// ── image OCR ──────────────────────────────────────────────────
let img = &mut self.ingest.image.ocr;
if let Some(v) = &shared.enabled {
if !medium_has("image", "enabled") {
img.enabled = *v;
}
}
if let Some(v) = &shared.engine {
if !medium_has("image", "engine") {
img.engine = v.clone();
}
}
if let Some(v) = &shared.model {
if !medium_has("image", "model") {
img.model = v.clone();
}
}
if let Some(v) = &shared.endpoint {
if !medium_has("image", "endpoint") {
img.endpoint = Some(v.clone());
}
}
if let Some(v) = &shared.languages {
if !medium_has("image", "languages") {
img.languages = v.clone();
}
}
if let Some(v) = &shared.max_pixels {
if !medium_has("image", "max_pixels") {
img.max_pixels = *v;
}
}
if let Some(v) = &shared.request_timeout_secs {
if !medium_has("image", "request_timeout_secs") {
img.request_timeout_secs = *v;
}
}
if let Some(v) = &shared.det_model {
if !medium_has("image", "det_model") {
img.det_model = Some(v.clone());
}
}
if let Some(v) = &shared.rec_model {
if !medium_has("image", "rec_model") {
img.rec_model = Some(v.clone());
}
}
if let Some(v) = &shared.dict {
if !medium_has("image", "dict") {
img.dict = Some(v.clone());
}
}
if let Some(v) = &shared.score_thresh {
if !medium_has("image", "score_thresh") {
img.score_thresh = *v;
}
}
if let Some(v) = &shared.unclip_ratio {
if !medium_has("image", "unclip_ratio") {
img.unclip_ratio = *v;
}
}
if let Some(v) = &shared.max_boxes {
if !medium_has("image", "max_boxes") {
img.max_boxes = *v;
}
}
// ── pdf OCR (same shared fields; the 4 pdf-unique fields are never
// touched by the shared block) ─────────────────────────────────
let pdf = &mut self.ingest.pdf.ocr;
if let Some(v) = &shared.enabled {
if !medium_has("pdf", "enabled") {
pdf.enabled = *v;
}
}
if let Some(v) = &shared.engine {
if !medium_has("pdf", "engine") {
pdf.engine = v.clone();
}
}
if let Some(v) = &shared.model {
if !medium_has("pdf", "model") {
pdf.model = v.clone();
}
}
if let Some(v) = &shared.endpoint {
if !medium_has("pdf", "endpoint") {
pdf.endpoint = Some(v.clone());
}
}
if let Some(v) = &shared.languages {
if !medium_has("pdf", "languages") {
pdf.languages = v.clone();
}
}
if let Some(v) = &shared.max_pixels {
if !medium_has("pdf", "max_pixels") {
pdf.max_pixels = *v;
}
}
if let Some(v) = &shared.request_timeout_secs {
if !medium_has("pdf", "request_timeout_secs") {
pdf.request_timeout_secs = *v;
}
}
if let Some(v) = &shared.det_model {
if !medium_has("pdf", "det_model") {
pdf.det_model = Some(v.clone());
}
}
if let Some(v) = &shared.rec_model {
if !medium_has("pdf", "rec_model") {
pdf.rec_model = Some(v.clone());
}
}
if let Some(v) = &shared.dict {
if !medium_has("pdf", "dict") {
pdf.dict = Some(v.clone());
}
}
if let Some(v) = &shared.score_thresh {
if !medium_has("pdf", "score_thresh") {
pdf.score_thresh = *v;
}
}
if let Some(v) = &shared.unclip_ratio {
if !medium_has("pdf", "unclip_ratio") {
pdf.unclip_ratio = *v;
}
}
if let Some(v) = &shared.max_boxes {
if !medium_has("pdf", "max_boxes") {
pdf.max_boxes = *v;
}
}
}
/// Effective image OCR settings (`[ingest.ocr]` shared block merged with
/// the `[ingest.image.ocr]` override block). After [`Config::resolve_ocr`]
/// has run at load, the concrete `ingest.image.ocr` already holds the
/// resolved values, so this is the canonical read handle for OCR
/// consumers. (Decouples them from the merge mechanics — they ask the
/// `Config` for "image OCR" rather than reaching into the struct path.)
pub fn image_ocr(&self) -> &OcrCfg {
&self.ingest.image.ocr
}
/// Effective PDF OCR settings (shared `[ingest.ocr]` merged with the
/// `[ingest.pdf.ocr]` override + 4 pdf-unique fields). See
/// [`Config::image_ocr`].
pub fn pdf_ocr(&self) -> &PdfOcrCfg {
&self.ingest.pdf.ocr
}
/// Apply `KEBAB_<SECTION>_<KEY>` env overrides. Unknown keys are ignored.
///
/// The mapping is an explicit grep-friendly whitelist — one match arm
@@ -1456,64 +1777,93 @@ impl Config {
),
},
// image.ocr
"KEBAB_IMAGE_OCR_ENABLED" => {
self.ingest.image.ocr.enabled = parse_bool(v);
// ── shared OCR engine (v5: KEBAB_OCR_*) ──────────────────
// The 13 engine-level knobs collapsed from the v4
// KEBAB_IMAGE_OCR_* / KEBAB_PDF_OCR_* duplicate sets. Each
// arm writes BOTH the image and pdf concrete blocks (env is a
// deliberate "set the OCR engine for the whole workspace"
// override — applied after load-time `resolve_ocr`, with no
// toml presence to consult). Per-medium `enabled` stays
// separately addressable below (KEBAB_IMAGE_OCR_ENABLED /
// KEBAB_PDF_OCR_ENABLED) so a user can turn image OCR on
// without forcing pdf OCR on.
"KEBAB_OCR_ENGINE" => {
self.ingest.image.ocr.engine = v.clone();
self.ingest.pdf.ocr.engine = v.clone();
}
"KEBAB_IMAGE_OCR_ENGINE" => self.ingest.image.ocr.engine = v.clone(),
"KEBAB_IMAGE_OCR_MODEL" => self.ingest.image.ocr.model = v.clone(),
"KEBAB_IMAGE_OCR_ENDPOINT" => {
// Empty env value is treated the same as "fall back
// to models.llm.endpoint" — i.e. set None.
self.ingest.image.ocr.endpoint = if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_MODEL" => {
self.ingest.image.ocr.model = v.clone();
self.ingest.pdf.ocr.model = v.clone();
}
"KEBAB_IMAGE_OCR_LANGUAGES" => {
"KEBAB_OCR_ENDPOINT" => {
// Empty env value → None (= fall back to models.llm.endpoint).
let e = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.endpoint = e.clone();
self.ingest.pdf.ocr.endpoint = e;
}
"KEBAB_OCR_LANGUAGES" => {
// Comma-separated list, e.g. "eng,kor".
self.ingest.image.ocr.languages = v
let langs: Vec<String> = v
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
self.ingest.image.ocr.languages = langs.clone();
self.ingest.pdf.ocr.languages = langs;
}
"KEBAB_IMAGE_OCR_MAX_PIXELS" => {
"KEBAB_OCR_MAX_PIXELS" => {
if let Ok(n) = v.parse::<u32>() {
self.ingest.image.ocr.max_pixels = n;
self.ingest.pdf.ocr.max_pixels = n;
}
}
"KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS" => {
"KEBAB_OCR_REQUEST_TIMEOUT_SECS" => {
if let Ok(n) = v.parse::<u64>() {
self.ingest.image.ocr.request_timeout_secs = n;
self.ingest.pdf.ocr.request_timeout_secs = n;
}
}
// paddle-onnx engine overrides (v0.27.0). Empty string → None
// paddle-onnx engine overrides. Empty string → None
// (fall back to bundled / KEBAB_IMAGE_OCR_MODEL_DIR).
"KEBAB_IMAGE_OCR_DET_MODEL" => {
self.ingest.image.ocr.det_model =
if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_DET_MODEL" => {
let m = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.det_model = m.clone();
self.ingest.pdf.ocr.det_model = m;
}
"KEBAB_IMAGE_OCR_REC_MODEL" => {
self.ingest.image.ocr.rec_model =
if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_REC_MODEL" => {
let m = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.rec_model = m.clone();
self.ingest.pdf.ocr.rec_model = m;
}
"KEBAB_IMAGE_OCR_DICT" => {
self.ingest.image.ocr.dict = if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_DICT" => {
let m = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.dict = m.clone();
self.ingest.pdf.ocr.dict = m;
}
"KEBAB_IMAGE_OCR_SCORE_THRESH" => {
"KEBAB_OCR_SCORE_THRESH" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.image.ocr.score_thresh = f;
self.ingest.pdf.ocr.score_thresh = f;
}
}
"KEBAB_IMAGE_OCR_UNCLIP_RATIO" => {
"KEBAB_OCR_UNCLIP_RATIO" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.image.ocr.unclip_ratio = f;
self.ingest.pdf.ocr.unclip_ratio = f;
}
}
"KEBAB_IMAGE_OCR_MAX_BOXES" => {
"KEBAB_OCR_MAX_BOXES" => {
if let Ok(n) = v.parse::<usize>() {
self.ingest.image.ocr.max_boxes = n;
self.ingest.pdf.ocr.max_boxes = n;
}
}
// image OCR enabled toggle (kept per-medium addressable).
"KEBAB_IMAGE_OCR_ENABLED" => {
self.ingest.image.ocr.enabled = parse_bool(v);
}
// image.caption (P6-3)
"KEBAB_IMAGE_CAPTION_ENABLED" => {
self.ingest.image.caption.enabled = parse_bool(v);
@@ -1527,31 +1877,14 @@ impl Config {
self.ingest.image.caption.prompt_template_version = v.clone();
}
// pdf.ocr (v0.20.0 sub-item 1)
// ── pdf-only OCR knobs (v5: the 4 fields with no image
// counterpart + the per-medium enabled toggle) ──────────
// The engine-level pdf knobs (engine/model/endpoint/languages/
// max_pixels/request_timeout_secs + the 6 paddle fields) moved
// to the shared KEBAB_OCR_* set above. These five have no
// image analogue, so they stay pdf-specific.
"KEBAB_PDF_OCR_ENABLED" => self.ingest.pdf.ocr.enabled = parse_bool(v),
"KEBAB_PDF_OCR_ALWAYS_ON" => self.ingest.pdf.ocr.always_on = parse_bool(v),
"KEBAB_PDF_OCR_ENGINE" => self.ingest.pdf.ocr.engine = v.clone(),
"KEBAB_PDF_OCR_MODEL" => self.ingest.pdf.ocr.model = v.clone(),
"KEBAB_PDF_OCR_ENDPOINT" => {
self.ingest.pdf.ocr.endpoint = if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_LANGUAGES" => {
self.ingest.pdf.ocr.languages = v
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
}
"KEBAB_PDF_OCR_MAX_PIXELS" => {
if let Ok(n) = v.parse::<u32>() {
self.ingest.pdf.ocr.max_pixels = n;
}
}
"KEBAB_PDF_OCR_REQUEST_TIMEOUT_SECS" => {
if let Ok(n) = v.parse::<u64>() {
self.ingest.pdf.ocr.request_timeout_secs = n;
}
}
"KEBAB_PDF_OCR_VALID_RATIO_THRESHOLD" => {
if let Ok(n) = v.parse::<f32>() {
self.ingest.pdf.ocr.valid_ratio_threshold = n.clamp(0.0, 1.0);
@@ -1565,34 +1898,6 @@ impl Config {
"KEBAB_PDF_OCR_LANG_HINT" => {
self.ingest.pdf.ocr.lang_hint = if v.is_empty() { None } else { Some(v.clone()) };
}
// pdf paddle-onnx engine overrides (v3). image.ocr paddle 패턴 복제.
// Empty string → None (fall back to bundled / KEBAB_IMAGE_OCR_MODEL_DIR).
"KEBAB_PDF_OCR_DET_MODEL" => {
self.ingest.pdf.ocr.det_model =
if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_REC_MODEL" => {
self.ingest.pdf.ocr.rec_model =
if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_DICT" => {
self.ingest.pdf.ocr.dict = if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_SCORE_THRESH" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.pdf.ocr.score_thresh = f;
}
}
"KEBAB_PDF_OCR_UNCLIP_RATIO" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.pdf.ocr.unclip_ratio = f;
}
}
"KEBAB_PDF_OCR_MAX_BOXES" => {
if let Ok(n) = v.parse::<usize>() {
self.ingest.pdf.ocr.max_boxes = n;
}
}
// Unknown KEBAB_* keys are silently ignored — see
// `env_unknown_key_is_ignored` test.
@@ -1874,24 +2179,31 @@ max_pixels = 1600
env.insert("KEBAB_CHUNKING_TARGET_TOKENS".into(), "640".into());
env.insert("KEBAB_INDEXING_MAX_PARALLEL_EXTRACTORS".into(), "6".into());
env.insert("KEBAB_IMAGE_OCR_ENABLED".into(), "true".into());
env.insert("KEBAB_PDF_OCR_ENGINE".into(), "paddle-onnx".into());
// v5: shared engine knob — sets BOTH image and pdf OCR engine.
env.insert("KEBAB_OCR_ENGINE".into(), "paddle-onnx".into());
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.chunking.target_tokens, 640);
assert_eq!(c.ingest.max_parallel_extractors, 6);
assert!(c.ingest.image.ocr.enabled);
assert_eq!(c.ingest.image.ocr.engine, "paddle-onnx");
assert_eq!(c.ingest.pdf.ocr.engine, "paddle-onnx");
}
/// v5: the paddle engine overrides moved to the shared `KEBAB_OCR_*` set
/// and apply to BOTH mediums in one shot.
#[test]
fn env_pdf_paddle_symmetric_overrides() {
fn env_shared_ocr_paddle_overrides_both_mediums() {
let mut env = HashMap::new();
env.insert("KEBAB_PDF_OCR_DET_MODEL".into(), "/d.onnx".into());
env.insert("KEBAB_PDF_OCR_SCORE_THRESH".into(), "0.4".into());
env.insert("KEBAB_PDF_OCR_MAX_BOXES".into(), "500".into());
env.insert("KEBAB_OCR_DET_MODEL".into(), "/d.onnx".into());
env.insert("KEBAB_OCR_SCORE_THRESH".into(), "0.4".into());
env.insert("KEBAB_OCR_MAX_BOXES".into(), "500".into());
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.pdf.ocr.det_model.as_deref(), Some("/d.onnx"));
assert!((c.ingest.pdf.ocr.score_thresh - 0.4).abs() < 1e-6);
assert_eq!(c.ingest.pdf.ocr.max_boxes, 500);
assert_eq!(c.ingest.image.ocr.det_model.as_deref(), Some("/d.onnx"));
assert!((c.ingest.image.ocr.score_thresh - 0.4).abs() < 1e-6);
assert_eq!(c.ingest.image.ocr.max_boxes, 500);
}
#[test]
@@ -1989,12 +2301,14 @@ max_pixels = 1600
#[test]
fn env_overrides_image_ocr_request_timeout_secs() {
let mut env = HashMap::new();
// v5: shared KEBAB_OCR_REQUEST_TIMEOUT_SECS sets both mediums.
env.insert(
"KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS".to_string(),
"KEBAB_OCR_REQUEST_TIMEOUT_SECS".to_string(),
"900".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.image.ocr.request_timeout_secs, 900);
assert_eq!(c.ingest.pdf.ocr.request_timeout_secs, 900);
}
/// post-v0.17.1 dogfood: a config file written before the OCR
@@ -2136,25 +2450,24 @@ max_pixels = 1600
);
}
/// v5: the engine-level OCR knobs come from the shared `KEBAB_OCR_*` set
/// (sets image AND pdf); only `enabled` stays per-medium.
#[test]
fn image_ocr_env_overrides() {
let mut env = HashMap::new();
env.insert("KEBAB_IMAGE_OCR_ENABLED".to_string(), "true".to_string());
env.insert("KEBAB_OCR_MODEL".to_string(), "gemma4:31b".to_string());
env.insert(
"KEBAB_IMAGE_OCR_MODEL".to_string(),
"gemma4:31b".to_string(),
);
env.insert(
"KEBAB_IMAGE_OCR_ENDPOINT".to_string(),
"KEBAB_OCR_ENDPOINT".to_string(),
"http://192.168.0.47:11434".to_string(),
);
// Empty env value should map to None (= fall back to llm.endpoint).
// We exercise that branch in a separate test.
env.insert(
"KEBAB_IMAGE_OCR_LANGUAGES".to_string(),
"KEBAB_OCR_LANGUAGES".to_string(),
"eng, kor, jpn".to_string(),
);
env.insert("KEBAB_IMAGE_OCR_MAX_PIXELS".to_string(), "2048".to_string());
env.insert("KEBAB_OCR_MAX_PIXELS".to_string(), "2048".to_string());
let c = Config::defaults().apply_env(&env);
assert!(c.ingest.image.ocr.enabled);
assert_eq!(c.ingest.image.ocr.model, "gemma4:31b");
@@ -2164,6 +2477,9 @@ max_pixels = 1600
);
assert_eq!(c.ingest.image.ocr.languages, vec!["eng", "kor", "jpn"]);
assert_eq!(c.ingest.image.ocr.max_pixels, 2048);
// shared knob also reached the pdf block.
assert_eq!(c.ingest.pdf.ocr.model, "gemma4:31b");
assert_eq!(c.ingest.pdf.ocr.max_pixels, 2048);
}
/// Pre-P6 config files don't have an `[image]` section. The
@@ -2198,15 +2514,17 @@ max_pixels = 1600
assert_eq!(c.ingest.image.caption.prompt_template_version, "caption-v2");
}
/// `KEBAB_IMAGE_OCR_ENDPOINT=""` (empty value) should map to `None`
/// v5: `KEBAB_OCR_ENDPOINT=""` (empty value) should map to `None`
/// rather than to `Some("")` so the fallback to `models.llm.endpoint`
/// kicks in. Covers the env-equivalent of a missing TOML key.
/// kicks in (for both mediums). Covers the env-equivalent of a missing
/// TOML key.
#[test]
fn image_ocr_endpoint_empty_env_value_is_none() {
let mut env = HashMap::new();
env.insert("KEBAB_IMAGE_OCR_ENDPOINT".to_string(), String::new());
env.insert("KEBAB_OCR_ENDPOINT".to_string(), String::new());
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.image.ocr.endpoint, None);
assert_eq!(c.ingest.pdf.ocr.endpoint, None);
}
#[test]

View File

@@ -9,7 +9,7 @@ use toml_edit::{DocumentMut, Item};
/// 현재 바이너리가 이해하는 config 스키마 버전. 마이그레이션 완료 시
/// 사용자 파일의 `schema_version` 을 이 값으로 stamp 한다.
pub const CURRENT_SCHEMA_VERSION: u32 = 4;
pub const CURRENT_SCHEMA_VERSION: u32 = 5;
/// 한 번의 마이그레이션에서 발생한 개별 변경.
#[derive(Clone, Debug, PartialEq, serde::Serialize)]
@@ -80,6 +80,7 @@ fn section_comment(path: &str) -> Option<&'static str> {
"rag" => "# 답변 생성: prompt 템플릿·score gate·NLI.",
"ui" => "# TUI 팔레트·role 스타일.",
"ingest" => "# 모든 형식 ingest 우산: 병렬도 + chunking/code/image/pdf.",
"ingest.ocr" => "# 공유 OCR 엔진 설정(image/pdf 공통). 각 미디어 블록이 override.",
"ingest.chunking" => "# 청크 크기·오버랩·heading 존중(전 형식 공통).",
"ingest.code" => "# code ingest skip 정책(.gitignore 자동 honor).",
"ingest.image" => "# 이미지 OCR + 캡션(기본 off, asset 당 모델 호출 비용).",
@@ -142,6 +143,16 @@ pub fn annotated_default_document() -> DocumentMut {
let pretty = toml::to_string_pretty(&defaults).expect("defaults serialize");
let mut doc: DocumentMut = pretty.parse().expect("defaults parse as toml_edit");
// v5: 직렬화된 defaults 는 `[ingest.image.ocr]` 에 12개 엔진 키를 그대로
// 담고 `[ingest.ocr]` 은 비어 있다(`SharedOcrEngineCfg::default()` = 전부
// None). 참조 문서를 v5 canonical 형상으로 맞추기 위해 같은 통합을 적용한다
// — 그러지 않으면 reconcile 이 마이그레이션으로 끌어올린 키를 image 블록에
// 다시 추가해 통합을 되돌린다(그리고 image 의 effective engine 을 default 로
// 덮어써 동작을 바꾼다). pdf 블록은 그대로 둔다(자기 default 가 image 와
// 달라 override 로 유지).
let mut discard = Vec::new();
step_4_to_5(&mut doc, &mut discard);
// 헤더: 첫 최상위 항목의 prefix 로.
if let Some((mut first_key, _)) = doc.as_table_mut().iter_mut().next() {
first_key.leaf_decor_mut().set_prefix(format!("{HEADER}\n"));
@@ -413,6 +424,87 @@ pub fn step_3_to_4(doc: &mut DocumentMut, changes: &mut Vec<MigrationChange>) {
});
}
/// v5: `[ingest.image.ocr]` 의 12개 **엔진** 키를 새 공유 블록 `[ingest.ocr]`
/// 로 끌어올린다(`enabled` 은 미디어별 토글이라 제외 — 끌어올리면 공유 블록이
/// pdf 의 `enabled` 까지 켜버려 동작이 바뀐다). image 는 unique 필드가 없으므로
/// 공유 블록의 canonical source 로 삼는다. `[ingest.pdf.ocr]` 은 손대지 않는다
/// — reconcile 이 pdf 의 모든 키를 default 로 채워 명시 상태가 되므로(아래
/// `resolve_ocr` 의 "미디어가 명시한 키는 공유 overlay 가 덮지 않음" 규칙에 의해)
/// pdf 의 effective 값은 마이그레이션 전후 불변. 멱등: 키가 이미 옮겨졌으면 no-op.
///
/// 결과적으로 image 의 effective OCR 엔진 설정은 `[ingest.ocr]` 에서, pdf 는
/// 자신의 (reconcile 로 완전 채워진) 블록에서 그대로 resolve 되어 양쪽 모두
/// 마이그레이션 전 값을 유지한다 — `ingest_config_signature` 바이트도 불변.
const SHARED_OCR_ENGINE_KEYS: [&str; 12] = [
"engine",
"model",
"endpoint",
"languages",
"max_pixels",
"request_timeout_secs",
"det_model",
"rec_model",
"dict",
"score_thresh",
"unclip_ratio",
"max_boxes",
];
pub fn step_4_to_5(doc: &mut DocumentMut, changes: &mut Vec<MigrationChange>) {
// image OCR 블록이 없으면 끌어올릴 게 없음(reconcile 이 빈 `[ingest.ocr]`
// 를 추가). 멱등 진입점.
let img_present = doc
.get("ingest")
.and_then(|i| i.get("image"))
.and_then(|i| i.get("ocr"))
.and_then(Item::as_table)
.is_some();
if !img_present {
return;
}
let mut lifted_any = false;
for key in SHARED_OCR_ENGINE_KEYS {
// image.ocr 에 key 가 있고, ingest.ocr 에 아직 없으면 통째(decor 포함) 이동.
let has_in_image = doc
.get("ingest")
.and_then(|i| i.get("image"))
.and_then(|i| i.get("ocr"))
.and_then(Item::as_table)
.is_some_and(|t| t.contains_key(key));
if !has_in_image {
continue;
}
let already_in_shared = doc
.get("ingest")
.and_then(|i| i.get("ocr"))
.and_then(Item::as_table)
.is_some_and(|t| t.contains_key(key));
if already_in_shared {
// 공유 블록에 이미 있으면 image 쪽 중복 키는 그냥 제거(공유가 우선).
if let Some(img) = doc["ingest"]["image"]["ocr"].as_table_mut() {
img.remove(key);
}
continue;
}
move_table(
doc,
&["ingest", "image", "ocr", key],
&["ingest", "ocr", key],
changes,
);
lifted_any = true;
}
if lifted_any {
changes.push(MigrationChange {
kind: ChangeKind::AddedSection,
path: "ingest.ocr".to_string(),
detail: "OCR 엔진 키를 공유 [ingest.ocr] 로 통합(image/pdf 중복 제거)".to_string(),
});
}
}
/// 파일의 schema_version(없으면 1) 부터 CURRENT 까지 step 적용.
fn run_steps(doc: &mut DocumentMut, from: u32, changes: &mut Vec<MigrationChange>) {
if from < 2 {
@@ -424,6 +516,9 @@ fn run_steps(doc: &mut DocumentMut, from: u32, changes: &mut Vec<MigrationChange
if from < 4 {
step_3_to_4(doc, changes);
}
if from < 5 {
step_4_to_5(doc, changes);
}
}
/// 사용자 config.toml 텍스트를 받아 step 체인 + reconciliation + version
@@ -477,6 +572,21 @@ pub fn migrate_document(text: &str) -> MigrationOutcome {
mod tests {
use super::*;
/// v5: parse a config text and run the shared-OCR resolution the way
/// `Config::from_file` does (overlay `[ingest.ocr]` down into the
/// per-medium concrete blocks), then clear the now-applied shared block.
/// The result is the canonical *effective* config — comparable against
/// `Config::defaults()` regardless of whether the engine knobs live in
/// the shared block (annotated default doc) or the per-medium blocks
/// (in-memory `defaults()`).
fn parse_effective(text: &str) -> crate::Config {
let parsed = toml::from_str::<toml::Value>(text).ok();
let mut cfg: crate::Config = toml::from_str(text).expect("parse config");
cfg.resolve_ocr(parsed.as_ref());
cfg.ingest.ocr = crate::SharedOcrEngineCfg::default();
cfg
}
#[test]
fn annotated_default_has_per_key_comments() {
let text = annotated_default_document().to_string();
@@ -487,9 +597,9 @@ mod tests {
text.contains("paddle-onnx 는 번들 모델"),
"ocr.model 주석 누락:\n{text}"
);
// 주석 추가가 파싱을 깨지 않는다.
let back: crate::Config = toml::from_str(&text).expect("parse annotated default");
assert_eq!(back, crate::Config::defaults());
// 주석 추가가 파싱을 깨지 않고, v5 OCR resolution 후 effective 값이
// defaults 와 동일.
assert_eq!(parse_effective(&text), crate::Config::defaults());
}
#[test]
@@ -498,10 +608,11 @@ mod tests {
let text = doc.to_string();
// v3: 미디어 형식 섹션이 전부 `[ingest.*]` 하위로 통합됐다. IngestCfg
// 는 스칼라(병렬도) 필드가 있어 bare `[ingest]` + 하위 테이블이 함께
// 직렬화된다.
// 직렬화된다. v5: 공유 `[ingest.ocr]` 엔진 블록이 추가됐다.
for section in [
"[workspace]",
"[ingest]",
"[ingest.ocr]",
"[ingest.chunking]",
"[ingest.code]",
"[ingest.image.ocr]",
@@ -512,8 +623,8 @@ mod tests {
assert!(text.contains(section), "missing {section}:\n{text}");
}
assert!(text.contains("# "), "no comments attached");
let back: crate::Config = toml::from_str(&text).expect("parse annotated default");
assert_eq!(back, crate::Config::defaults());
// v5: effective 값(공유 OCR resolution 후)이 defaults 와 동일.
assert_eq!(parse_effective(&text), crate::Config::defaults());
}
#[test]
@@ -739,7 +850,7 @@ root = \"/my/notes\"
}
#[test]
fn migrate_document_v3_to_v4_adds_sources_and_is_idempotent() {
fn migrate_document_v3_to_current_adds_sources_and_is_idempotent() {
let v3 = "\
schema_version = 3
@@ -749,10 +860,10 @@ exclude = []
";
let outcome = migrate_document(v3);
assert_eq!(outcome.from_schema_version, 3);
assert_eq!(outcome.to_schema_version, 4);
assert_eq!(outcome.to_schema_version, CURRENT_SCHEMA_VERSION);
assert!(outcome.changed());
assert!(outcome.new_text.contains("[[workspace.sources]]"));
assert_eq!(read_schema_version(&outcome.new_text), 4);
assert_eq!(read_schema_version(&outcome.new_text), CURRENT_SCHEMA_VERSION);
let again = migrate_document(&outcome.new_text);
assert!(!again.changed(), "not idempotent: {:?}", again.changes);
assert_eq!(again.new_text, outcome.new_text);
@@ -765,4 +876,98 @@ exclude = []
assert_eq!(outcome.from_schema_version, 1);
assert_eq!(read_schema_version(&outcome.new_text), CURRENT_SCHEMA_VERSION);
}
/// v4 → v5 무손실 라운드트립: `[ingest.image.ocr]` / `[ingest.pdf.ocr]` 이
/// 채워진 v4 config 을 마이그레이션 → from_file 로 로드(공유 OCR resolution
/// 포함) → image/pdf OCR 의 effective 값이 마이그레이션 전과 정확히 동일해야
/// 한다. image 의 비-default engine(paddle-onnx) 이 공유 블록으로 끌어올려진
/// 뒤에도 보존되는지(이전 버그) + pdf 의 고유 값(qwen 모델·2048px)이 공유
/// overlay 에 오염되지 않는지를 함께 검증한다.
#[test]
fn migrate_v4_to_v5_preserves_effective_ocr() {
let v4 = "\
schema_version = 4
[workspace]
root = \"/my/notes\"
exclude = []
[[workspace.sources]]
id = \"default\"
root = \"/my/notes\"
[ingest.image.ocr]
enabled = true
engine = \"paddle-onnx\"
model = \"gemma4:e4b\"
languages = [\"eng\", \"kor\"]
max_pixels = 1280
request_timeout_secs = 450
det_model = \"/custom/det.onnx\"
score_thresh = 0.45
[ingest.pdf.ocr]
enabled = true
always_on = false
engine = \"ollama-vision\"
model = \"qwen2.5vl:7b\"
languages = [\"eng\", \"kor\"]
max_pixels = 2048
request_timeout_secs = 240
valid_ratio_threshold = 0.6
min_char_count = 25
lang_hint = \"kor\"
";
// pre-migration effective values, loaded the v4 way (no shared block).
let dir = std::env::temp_dir().join(format!("kebab_v5_rt_{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let p4 = dir.join("v4.toml");
std::fs::write(&p4, v4).unwrap();
let before = crate::Config::from_file(&p4).expect("load v4");
// migrate → load the v5 text via from_file (runs resolve_ocr).
let outcome = migrate_document(v4);
assert_eq!(outcome.from_schema_version, 4);
assert_eq!(outcome.to_schema_version, 5);
assert!(outcome.changed());
assert!(
outcome.new_text.contains("[ingest.ocr]"),
"shared block missing:\n{}",
outcome.new_text
);
let p5 = dir.join("v5.toml");
std::fs::write(&p5, &outcome.new_text).unwrap();
let after = crate::Config::from_file(&p5).expect("load v5");
// image OCR effective 값 전부 보존(특히 비-default engine paddle-onnx).
assert_eq!(after.image_ocr().enabled, before.image_ocr().enabled);
assert_eq!(after.image_ocr().engine, "paddle-onnx");
assert_eq!(after.image_ocr().engine, before.image_ocr().engine);
assert_eq!(after.image_ocr().model, before.image_ocr().model);
assert_eq!(after.image_ocr().languages, before.image_ocr().languages);
assert_eq!(after.image_ocr().max_pixels, 1280);
assert_eq!(after.image_ocr().max_pixels, before.image_ocr().max_pixels);
assert_eq!(
after.image_ocr().request_timeout_secs,
before.image_ocr().request_timeout_secs
);
assert_eq!(
after.image_ocr().det_model.as_deref(),
Some("/custom/det.onnx")
);
assert_eq!(after.image_ocr().det_model, before.image_ocr().det_model);
assert!((after.image_ocr().score_thresh - 0.45).abs() < 1e-6);
assert_eq!(after.image_ocr(), before.image_ocr());
// pdf OCR effective 값 전부 보존(공유 overlay 가 image 값으로 오염 X).
assert_eq!(after.pdf_ocr().engine, "ollama-vision");
assert_eq!(after.pdf_ocr().model, "qwen2.5vl:7b");
assert_eq!(after.pdf_ocr().max_pixels, 2048);
assert_eq!(after.pdf_ocr(), before.pdf_ocr());
// 멱등.
let again = migrate_document(&outcome.new_text);
assert!(!again.changed(), "v5 재실행 변경: {:?}", again.changes);
assert_eq!(again.new_text, outcome.new_text);
}
}

View File

@@ -11,9 +11,13 @@ const USER_V2: &str = include_str!("fixtures/user_v2_config.toml");
fn user_v2_migrates_losslessly() {
let out = migrate_document(USER_V2);
assert_eq!(out.from_schema_version, 2);
// v2 → CURRENT(=4): v3 의 [ingest.*] relocation 에 더해 v4 의
// [[workspace.sources]] default source 미러링까지 적용된다.
assert_eq!(out.to_schema_version, 4);
// v2 → CURRENT(=5): v3 의 [ingest.*] relocation, v4 의
// [[workspace.sources]] default source 미러링, v5 의 공유 [ingest.ocr]
// 통합까지 적용된다.
assert_eq!(
out.to_schema_version,
kebab_config::migrate::CURRENT_SCHEMA_VERSION
);
let t = &out.new_text;
// 사용자 값 보존.
@@ -36,15 +40,23 @@ fn user_v2_migrates_losslessly() {
assert!(!t.contains("\n[image.ocr]"));
assert!(!t.contains("\n[indexing]"));
// v3 Config 로 parse + 값 동일.
let cfg: kebab_config::Config = toml::from_str(t).expect("v3 parse");
assert!(cfg.ingest.image.ocr.enabled);
assert_eq!(cfg.ingest.image.ocr.engine, "paddle-onnx");
// v5: 공유 [ingest.ocr] 통합 후 image 엔진 키는 공유 블록에 산다.
assert!(t.contains("[ingest.ocr]"), "공유 OCR 블록 누락:\n{t}");
// effective 값은 from_file(공유 OCR resolution 포함)로 검증한다 —
// image 의 engine=paddle-onnx 가 공유 블록으로 끌어올려진 뒤에도 보존.
let dir = std::env::temp_dir().join(format!("kebab_mv3_{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let p = dir.join("config.toml");
std::fs::write(&p, t).unwrap();
let cfg = kebab_config::Config::from_file(&p).expect("v5 from_file");
assert!(cfg.image_ocr().enabled);
assert_eq!(cfg.image_ocr().engine, "paddle-onnx");
assert_eq!(cfg.models.embedding.model, "snowflake-arctic-embed2");
assert_eq!(cfg.models.llm.endpoint, "http://192.168.0.2:11943");
// pdf paddle 값 보존(v2 비대칭 → pdf 대칭 키로 복사). user 의 pdf.ocr 는
// engine=paddle-onnx 이고 자체 det_model 없으므로 번들(None) 유지.
assert_eq!(cfg.ingest.pdf.ocr.engine, "paddle-onnx");
assert_eq!(cfg.pdf_ocr().engine, "paddle-onnx");
// 멱등.
let again = migrate_document(t);

View File

@@ -63,15 +63,14 @@ fn pdf_ocr_defaults_off_with_qwen_3b() {
assert_eq!(cfg.ingest.pdf.ocr.lang_hint.as_deref(), Some("kor"));
}
// Test 3: env var override — 4 keys 의 typical override case.
// Test 3: env var override — pdf-only keys + shared engine knob.
// v5: `model` moved to the shared `KEBAB_OCR_MODEL` (sets both mediums);
// `enabled`/`always_on`/`valid_ratio_threshold` stay pdf-specific.
#[test]
fn pdf_ocr_env_overrides() {
let mut env: HashMap<String, String> = HashMap::new();
env.insert("KEBAB_PDF_OCR_ENABLED".to_string(), "true".to_string());
env.insert(
"KEBAB_PDF_OCR_MODEL".to_string(),
"qwen2.5vl:7b".to_string(),
);
env.insert("KEBAB_OCR_MODEL".to_string(), "qwen2.5vl:7b".to_string());
env.insert("KEBAB_PDF_OCR_ALWAYS_ON".to_string(), "true".to_string());
env.insert(
"KEBAB_PDF_OCR_VALID_RATIO_THRESHOLD".to_string(),

View File

@@ -133,7 +133,7 @@ impl OllamaVisionOcr {
/// Construction does NOT touch the network — the first HTTP call
/// happens inside [`OcrEngine::recognize`].
pub fn new(config: &kebab_config::Config) -> Result<Self> {
let ocr = &config.ingest.image.ocr;
let ocr = config.image_ocr();
let endpoint = match ocr.endpoint.as_deref() {
Some(s) if !s.is_empty() => s.to_string(),
_ => config.models.llm.endpoint.clone(),

View File

@@ -122,7 +122,7 @@ impl ModelPaths {
/// [`from_default_dir`]: ModelPaths::from_default_dir
pub fn from_config(config: &kebab_config::Config) -> Self {
let defaults = Self::from_default_dir();
let ocr = &config.ingest.image.ocr;
let ocr = config.image_ocr();
Self {
det: ocr.det_model.as_ref().map(PathBuf::from).unwrap_or(defaults.det),
rec: ocr.rec_model.as_ref().map(PathBuf::from).unwrap_or(defaults.rec),
@@ -138,7 +138,7 @@ impl OnnxPaddleOcr {
/// here are fail-fast (matches the Ollama adapter's construction contract).
pub fn new(config: &kebab_config::Config) -> Result<Self> {
let paths = ModelPaths::from_config(config);
let ocr = &config.ingest.image.ocr;
let ocr = config.image_ocr();
Self::from_paths(
&paths,
ocr.score_thresh,

View File

@@ -354,15 +354,16 @@ fn from_parts_clamps_max_pixels_into_legal_range() {
/// Run with:
///
/// ```sh
/// KEBAB_IMAGE_OCR_ENDPOINT=http://192.168.0.47:11434 \
/// KEBAB_OCR_ENDPOINT=http://192.168.0.47:11434 \
/// cargo test -p kebab-parse-image --test ocr ocr_integration -- --ignored
/// ```
#[tokio::test]
#[ignore = "hits a real Ollama daemon; opt in via `cargo test -- --ignored`"]
async fn ocr_integration_real_ollama_transcribes_text() {
let endpoint = std::env::var("KEBAB_IMAGE_OCR_ENDPOINT")
// v5: shared KEBAB_OCR_* env (manual harness reads it directly).
let endpoint = std::env::var("KEBAB_OCR_ENDPOINT")
.unwrap_or_else(|_| "http://192.168.0.47:11434".to_string());
let model = std::env::var("KEBAB_IMAGE_OCR_MODEL").unwrap_or_else(|_| "gemma4:e4b".to_string());
let model = std::env::var("KEBAB_OCR_MODEL").unwrap_or_else(|_| "gemma4:e4b".to_string());
// Generate a fixture with known text. If the DejaVu font is
// missing from this dev box, skip rather than crash.

View File

@@ -2,7 +2,7 @@
// F1 ≥ 0.85, F2 ≥ 0.70. real Ollama 의존 — `#[ignore]` default.
//
// Manual invoke:
// KEBAB_PDF_OCR_ENDPOINT=http://192.168.0.47:11434 \
// KEBAB_OCR_ENDPOINT=http://192.168.0.47:11434 \
// cargo test -p kebab-parse-pdf --test ocr_e2e --ignored -j 4
use kebab_core::Lang;
@@ -11,7 +11,8 @@ use kebab_parse_pdf::extract_dctdecode_page_image;
use lopdf::Document;
fn run_real_ollama_ocr(pdf: &[u8], page: u32) -> anyhow::Result<String> {
let endpoint = std::env::var("KEBAB_PDF_OCR_ENDPOINT")
// v5: shared KEBAB_OCR_* env (manual harness reads it directly).
let endpoint = std::env::var("KEBAB_OCR_ENDPOINT")
.unwrap_or_else(|_| "http://localhost:11434".to_string());
let doc = Document::load_mem(pdf)?;
let jpeg = extract_dctdecode_page_image(&doc, page)?

View File

@@ -803,13 +803,13 @@ Cross-link: `tasks/HOTFIXES.md` (2026-05-29 — 검색 품질 baseline entry), `
```bash
KEBAB_PDF_OCR_ENABLED=true \
KEBAB_PDF_OCR_MODEL=qwen2.5vl:7b \
KEBAB_OCR_MODEL=qwen2.5vl:7b \
"$RELEASE_BIN" ingest --config "$DOGFOOD/config.toml"
```
**verify per env**:
- `KEBAB_PDF_OCR_*` (11 env, v0.20.0).
- `KEBAB_IMAGE_OCR_*` (P6).
- `KEBAB_OCR_*` (config schema v5: 공유 OCR 엔진 env — image·pdf 양쪽 적용).
- `KEBAB_IMAGE_OCR_ENABLED` / `KEBAB_PDF_OCR_ENABLED` (미디어별 on/off 토글) + PDF 고유 `KEBAB_PDF_OCR_{ALWAYS_ON,VALID_RATIO_THRESHOLD,MIN_CHAR_COUNT,LANG_HINT}`.
- `KEBAB_MODELS_LLM_*`, `KEBAB_MODELS_EMBEDDING_*`.
- `KEBAB_READONLY` (write-path subcommand 차단).

View File

@@ -347,14 +347,18 @@ MCP tool 동등:
[workspace]
include = ["**/*.md", "**/*.png", "**/*.jpg"]
[ingest.image.ocr]
enabled = true # vision LM 으로 이미지 안 텍스트 전사
# config schema v5: OCR 엔진 설정은 공유 [ingest.ocr] 에 한 번. 각 미디어
# 블록은 on/off 토글 + 필요한 override 만. 미디어 블록이 같은 키를 적으면 우선.
[ingest.ocr]
engine = "ollama-vision"
model = "gemma4:e4b" # 사용자 환경의 비전 모델
model = "gemma4:e4b" # 사용자 환경의 비전 모델 (image 기본)
endpoint = "http://192.168.0.47:11434" # 비우면 models.llm.endpoint fallback
languages = ["eng", "kor"]
max_pixels = 1600 # long-edge cap
[ingest.image.ocr]
enabled = true # vision LM 으로 이미지 안 텍스트 전사 (미디어별 토글)
[ingest.image.caption]
enabled = true # vision LM 으로 한 문장 객관 설명 생성
max_pixels = 768
@@ -363,17 +367,16 @@ prompt_template_version = "caption-v1"
[ingest.pdf.ocr]
enabled = true # smoke test 의 OCR path 활성화 (manual invoke)
always_on = false
engine = "ollama-vision"
model = "qwen2.5vl:3b"
# endpoint = "http://192.168.0.47:11434" # 사용자 dogfood host
languages = ["eng", "kor"]
max_pixels = 2048
model = "qwen2.5vl:3b" # PDF 는 다른 비전 모델로 override ([ingest.ocr] 의 gemma4 대신)
max_pixels = 2048 # PDF 페이지는 더 큰 long-edge
request_timeout_secs = 600
valid_ratio_threshold = 0.5
valid_ratio_threshold = 0.5 # PDF 고유 키 (image 에 없음)
min_char_count = 20
lang_hint = "kor"
```
> env override: 엔진 설정은 `KEBAB_OCR_*` (예: `KEBAB_OCR_ENDPOINT`, `KEBAB_OCR_MODEL`) 하나로 image·pdf 양쪽에 적용. on/off 는 `KEBAB_IMAGE_OCR_ENABLED` / `KEBAB_PDF_OCR_ENABLED` 로 미디어별. (config schema v5 — 옛 `KEBAB_IMAGE_OCR_*` / `KEBAB_PDF_OCR_*` 엔진 키는 `KEBAB_OCR_*` 로 통합.)
이미지 자산 한 장당 OCR 1 호출 + Caption 1 호출 → ~3-6초 (`gemma4:e4b` 기준). 다이어그램 / 카메라 사진 / 스크린샷 위주 워크스페이스에 권장. 책 / 스캔본은 P7 PDF 라인으로.
**v0.27.0 — paddle-onnx 엔진 (오프라인, Ollama 불필요).** `[ingest.image.ocr] engine = "paddle-onnx"` 로 바꾸면 PP-OCRv5 ONNX 를 in-process 로 실행한다 (원격 vision LM 불필요, 큰 페이지 CPU <4초). embedding 까지 끄려면 `[models.embedding] provider = "none"` (lexical-only) 로 두면 Ollama 없이 OCR→FTS5 검색 전체 경로를 스모크할 수 있다:

View File

@@ -14,6 +14,38 @@ historical contract that was implemented; this file accumulates the
deltas so phase 5+ readers can find the live behavior without diffing
git history.
## 2026-06-24 — spine-rewrite Phase 2 Unit 1: OCR 중복 제거 — 공유 `[ingest.ocr]` + config v4→v5
척추 단순화 Phase 2 Unit 1 = OCR config 중복 제거. v4 까지 `OcrCfg`(image) 13필드가
`PdfOcrCfg`(pdf) 와 전부 중복(image 고유 필드 0, pdf 고유 4: `always_on`/`valid_ratio_threshold`/
`min_char_count`/`lang_hint`)이었고, `apply_env``KEBAB_IMAGE_OCR_*`/`KEBAB_PDF_OCR_*` 27 arm 이
복제돼 있었다.
- **신규 `SharedOcrEngineCfg`** = 13 공유 필드(전부 `Option`, default 전부 `None`) → `[ingest.ocr]`.
엔진 설정의 단일 출처. image/pdf 블록은 on/off 토글 + override 만.
- **load-time resolution** (`Config::resolve_ocr`, `from_file` 에서 호출): 각 공유 필드가
`Some` 이고 해당 미디어 블록이 그 키를 **명시 안 했으면** concrete `OcrCfg`/`PdfOcrCfg`
overlay. presence 는 parse 한 `toml::Value` 로 판정(미디어 명시 > 공유 > 내장 default).
`OcrCfg`/`PdfOcrCfg` 의 struct 필드는 그대로(75 mutation site 무영향) — 엔진 필드에
`#[serde(default)]` 만 추가해 slim 블록도 파싱. image(gemma4:e4b/1600) vs pdf(qwen2.5vl:3b/2048)
**미디어별 기본값 보존**.
- **resolver method** `Config::image_ocr()`/`pdf_ocr()` → resolved 블록 반환. consumer
(`kebab-parse-image` ocr/paddle, `kebab-app` build_*_ocr_engine·ingest gate·pdf_ocr_apply·
**ingest_config_signature**)가 전부 이걸 경유 → god-struct 직접 read 제거.
- **`apply_env` 통합**: 27 arm → 공유 `KEBAB_OCR_*` 12 arm(image+pdf 동시 set) + pdf 고유 4 arm
+ 미디어별 `KEBAB_IMAGE_OCR_ENABLED`/`KEBAB_PDF_OCR_ENABLED` 토글.
- **마이그레이션 `step_4_to_5`**: `[ingest.image.ocr]` 의 12 엔진 키를 `[ingest.ocr]`
`move_table`(`enabled` 은 미디어별이라 제외 — 끌어올리면 pdf 까지 켜짐). pdf 블록은
무손상(reconcile 이 pdf 모든 키를 default 로 채워 명시 상태 → 공유 overlay 오염 X).
`annotated_default_document` 도 같은 통합을 적용해 v5 canonical 형상으로(안 그러면 reconcile 이
끌어올린 키를 image 에 재추가 → image engine 을 default 로 덮어쓰는 회귀). `CURRENT_SCHEMA_VERSION=5`.
- **불변식 검증**: v4→v5 round-trip 테스트(image engine=paddle-onnx 비-default 보존 + pdf
qwen/2048 오염 X + 멱등) green. `from_file` 이 effective image/pdf OCR 를 v4 와 바이트 동일하게
resolve → `ingest_config_signature` 도 입력 불변 → **강제 재색인 없음**. `clippy --workspace
--all-targets` 0, `kebab-config`/`kebab-parse-image`/`kebab-parse-pdf`/`kebab-app` 테스트 pass.
- 브랜치 `refactor/spine-cuts`. surface 동기화: README `[ingest.ocr]` 절 + SMOKE config 블록 +
DOGFOOD env 표.
## 2026-06-24 — spine-rewrite Phase 1: 5건 삭제 (cache/templates/candle/sessions/tui) — 코어 출력 불변
척추 단순화 Phase 1 = 순수 삭제 5건. **OMC-style worktree 격리 병렬 teammate** 5명이 각자