altair823-org/kebab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 46 Wiki Activity

refactor(config): OCR 중복 제거 — 공유 [ingest.ocr] 엔진 블록 + v4→v5 마이그레이션

Browse Source 
Phase 2 Unit 1. OcrCfg(image) 13필드가 PdfOcrCfg(pdf) 와 전부 중복(image 고유
0, pdf 고유 4) + apply_env 에 KEBAB_IMAGE_OCR_*/KEBAB_PDF_OCR_* 27 arm 복제를
제거한다.

- 신규 SharedOcrEngineCfg(13 공유 필드, 전부 Option, default None) = [ingest.ocr].
  엔진 설정 단일 출처. image/pdf 블록은 on/off 토글 + override.
- load-time resolution(Config::resolve_ocr, from_file 호출): 공유 필드가 Some 이고
  미디어 블록이 그 키 미명시면 concrete OcrCfg/PdfOcrCfg 로 overlay(presence 는
  toml::Value 로 판정; 미디어 > 공유 > 내장 default). struct 필드는 그대로 두고
  엔진 필드에 #[serde(default)] 만 추가(slim 블록 파싱) → image(gemma4:e4b/1600)
  vs pdf(qwen2.5vl:3b/2048) 미디어별 기본값 보존.
- resolver Config::image_ocr()/pdf_ocr() 추가. consumer(kebab-parse-image,
  kebab-app build_*_ocr_engine·ingest gate·pdf_ocr_apply·ingest_config_signature)가
  전부 경유 → god-struct 직접 read 제거.
- apply_env: 27 arm → 공유 KEBAB_OCR_* 12 arm(image+pdf 동시) + pdf 고유 4 arm +
  미디어별 KEBAB_IMAGE_OCR_ENABLED/KEBAB_PDF_OCR_ENABLED.
- step_4_to_5: [ingest.image.ocr] 12 엔진 키를 [ingest.ocr] 로 move_table(enabled
  제외). pdf 블록 무손상(reconcile 이 채워 공유 overlay 오염 X). annotated_default
  도 동일 통합으로 v5 canonical 형상. CURRENT_SCHEMA_VERSION=5.
- v4→v5 round-trip 테스트(비-default image engine 보존 + pdf 오염 X + 멱등). effective
  OCR 바이트 동일 → ingest_config_signature 불변 → 강제 재색인 없음.

검증: clippy --workspace --all-targets 0 / kebab-config·kebab-parse-image·
kebab-parse-pdf·kebab-app 테스트 pass. surface: README [ingest.ocr] 절 + SMOKE
config 블록 + DOGFOOD env + HOTFIXES dated entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Mc6W1fgsrbFKTsqA6P8La
This commit is contained in:
altair823
2026-06-24 11:03:39 +00:00
parent c9a9832f7f
commit da323af87b
15 changed files with 747 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

508
crates/kebab-config/src/lib.rs
View File

@@ -406,6 +406,78 @@ fn default_nli_threshold() -> f32 {
0.0
}
/// v5: workspace-wide OCR **engine** defaults shared by image and PDF OCR.
///
/// Before v5, the 13 engine-level OCR knobs (engine, model, endpoint,
/// languages, max_pixels, request_timeout_secs + the 6 paddle-onnx asset /
/// tuning fields) were duplicated verbatim across `[ingest.image.ocr]` and
/// `[ingest.pdf.ocr]`. This block holds them ONCE under `[ingest.ocr]`; the
/// per-medium blocks (`[ingest.image.ocr]` / `[ingest.pdf.ocr]`) keep only
/// their `enabled` toggle + any medium-specific override / unique field.
///
/// Resolution happens at load (`Config::resolve_ocr`, called from
/// `from_file`): for every shared key present here but **absent** from a
/// medium's own block, the shared value is merged down into that medium's
/// concrete [`OcrCfg`] / [`PdfOcrCfg`]. A key the user wrote explicitly in
/// the per-medium block always wins (precedence: medium override > shared >
/// medium hardcoded default). Image and PDF therefore keep their distinct
/// hardcoded defaults (`gemma4:e4b`/1600 vs `qwen2.5vl:3b`/2048) when neither
/// block sets the field.
///
/// Every field is `Option` so "unset in `[ingest.ocr]`" is distinguishable
/// from "set to the type's zero value". Default = all `None` (no shared
/// override → each medium uses its own block / hardcoded default), so a
/// `Config` built without a `[ingest.ocr]` section behaves exactly as before.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct SharedOcrEngineCfg {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub enabled: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub engine: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub model: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub endpoint: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub languages: Option<Vec<String>>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_pixels: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub request_timeout_secs: Option<u64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub det_model: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub rec_model: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub dict: Option<String>,
#[serde(
default,
skip_serializing_if = "Option::is_none",
serialize_with = "ser_opt_f32_clean"
)]
pub score_thresh: Option<f32>,
#[serde(
default,
skip_serializing_if = "Option::is_none",
serialize_with = "ser_opt_f32_clean"
)]
pub unclip_ratio: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max_boxes: Option<usize>,
}
/// `Option<f32>` 직렬화 시 `Some` 값을 [`ser_f32_clean`] 과 같은 shortest
/// round-trip 으로 출력한다(`None` 은 `skip_serializing_if` 가 처리).
fn ser_opt_f32_clean<S>(v: &Option<f32>, s: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
match v {
Some(f) => ser_f32_clean(f, s),
None => s.serialize_none(),
}
}
/// Settings for the image ingest pipeline (P6). `ocr` controls OCR
/// behaviour (P6-2); `caption` controls vision-LM captioning (P6-3).
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -434,11 +506,19 @@ impl ImageCfg {
pub struct OcrCfg {
/// Run OCR on every image during ingest. Default `false` because
/// OCR adds one model call per asset.
///
/// v5: all engine-level fields carry `#[serde(default)]` so a slim
/// `[ingest.image.ocr]` block (e.g. only `enabled`, with the engine
/// knobs hoisted to the shared `[ingest.ocr]`) still deserializes. The
/// per-field defaults match [`OcrCfg::defaults`] (image medium).
#[serde(default)]
pub enabled: bool,
/// Engine identifier. v1 only ships `"ollama-vision"`.
#[serde(default = "default_ocr_engine")]
pub engine: String,
/// Model id passed to the engine (e.g. `"gemma4:e4b"` for
/// Ollama-vision).
#[serde(default = "default_image_ocr_model")]
pub model: String,
/// HTTP endpoint for the OCR engine. `None` (or a missing key in
/// TOML) means "fall back to `models.llm.endpoint`" — convenient
@@ -447,9 +527,11 @@ pub struct OcrCfg {
pub endpoint: Option<String>,
/// BCP-47 language hints (e.g. `["eng", "kor"]`). The adapter
/// renders them into the prompt; the LLM honours them probabilistically.
#[serde(default = "default_ocr_languages")]
pub languages: Vec<String>,
/// Cap the long edge of the image (in pixels) before sending. Larger
/// images bloat prompt cost. Default `1600`.
#[serde(default = "default_image_ocr_max_pixels")]
pub max_pixels: u32,
/// v0.17.2 post-dogfood: Hard ceiling on a single HTTP exchange to
/// the OCR endpoint. Sister knob to [`LlmCfg::request_timeout_secs`]
@@ -519,6 +601,22 @@ impl OcrCfg {
}
}
/// v5: shared OCR engine-field defaults (image medium), so a slim
/// `[ingest.image.ocr]` block with the engine knobs hoisted to
/// `[ingest.ocr]` still deserializes. Mirror [`OcrCfg::defaults`].
fn default_ocr_engine() -> String {
"ollama-vision".to_string()
}
fn default_ocr_languages() -> Vec<String> {
vec!["eng".to_string(), "kor".to_string()]
}
fn default_image_ocr_model() -> String {
"gemma4:e4b".to_string()
}
fn default_image_ocr_max_pixels() -> u32 {
1600
}
/// paddle-onnx DBNet box score threshold default. See [`OcrCfg::score_thresh`].
fn default_ocr_score_thresh() -> f32 {
0.3
@@ -534,7 +632,8 @@ fn default_ocr_max_boxes() -> usize {
/// v0.17.2 post-dogfood: matches the legacy hard-coded ceiling so
/// existing configs that omit the field keep behaving identically.
/// Overridable per config / `KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS`.
/// Overridable per config / `KEBAB_OCR_REQUEST_TIMEOUT_SECS` (v5: shared
/// engine env, applies to both image and pdf OCR).
fn default_ocr_request_timeout_secs() -> u64 {
300
}
@@ -646,24 +745,37 @@ impl Default for LoggingCfg {
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct PdfOcrCfg {
/// Run OCR on scanned PDF pages. Default `false` (opt-in).
///
/// v5: the shared engine fields (engine/model/languages/max_pixels +
/// the request-timeout & paddle knobs) carry `#[serde(default)]` so a
/// slim `[ingest.pdf.ocr]` block (engine knobs hoisted to the shared
/// `[ingest.ocr]`) still deserializes; per-field defaults match the PDF
/// medium ([`PdfOcrCfg::defaults`]).
#[serde(default)]
pub enabled: bool,
/// `false` (default) — text-detect first + vision fallback on
/// scanned pages only. `true` — vision LLM 호출 on every page
/// (vector PDF 의 dual-text confidence boost — doubles chunk count).
#[serde(default)]
pub always_on: bool,
/// Engine identifier: `"ollama-vision"` or `"paddle-onnx"`. When set to
/// `"paddle-onnx"`, model paths and tuning knobs are read from
/// `[image.ocr]`, not `[pdf.ocr]` — PaddleOCR has no PDF-specific tuning.
/// `"paddle-onnx"`, model paths and tuning knobs are read from the shared
/// `[ingest.ocr]` (resolved via the image OCR block) — PaddleOCR has no
/// PDF-specific tuning.
#[serde(default = "default_ocr_engine")]
pub engine: String,
/// Vision model id. Default `"qwen2.5vl:3b"` per PoC (§3.5 family
/// asymmetry vs image OCR's gemma4:e4b is acknowledged).
#[serde(default = "default_pdf_ocr_model")]
pub model: String,
/// HTTP endpoint. `None` → fall back to `models.llm.endpoint`.
#[serde(default)]
pub endpoint: Option<String>,
/// BCP-47 language hints rendered into prompt.
#[serde(default = "default_ocr_languages")]
pub languages: Vec<String>,
/// Long-edge cap (px). Larger images bloat prompt cost.
#[serde(default = "default_pdf_ocr_max_pixels")]
pub max_pixels: u32,
/// HTTP request timeout (sec). Same `0` = "fail immediately"
/// semantics as `image.ocr.request_timeout_secs` (NOT a disable
@@ -757,6 +869,15 @@ fn default_pdf_ocr_min_char_count() -> u32 {
fn default_pdf_ocr_lang_hint() -> Option<String> {
Some("kor".to_string())
}
/// v5: PDF-medium engine-field defaults (distinct from image: qwen2.5vl:3b /
/// 2048 px), so a slim `[ingest.pdf.ocr]` block deserializes. Mirror
/// [`PdfOcrCfg::defaults`].
fn default_pdf_ocr_model() -> String {
"qwen2.5vl:3b".to_string()
}
fn default_pdf_ocr_max_pixels() -> u32 {
2048
}
/// p9-fb-14: TUI-only configuration. Currently a single `theme`
/// selector (`"dark"` / `"light"`); future fields (custom role
@@ -795,6 +916,11 @@ pub struct IngestCfg {
pub chunking: ChunkingCfg,
#[serde(default)]
pub code: IngestCodeCfg,
/// v5: shared OCR engine defaults (`[ingest.ocr]`). Merged down into
/// `image.ocr` / `pdf.ocr` at load by [`Config::resolve_ocr`]. Empty
/// (all `None`) by default — see [`SharedOcrEngineCfg`].
#[serde(default)]
pub ocr: SharedOcrEngineCfg,
#[serde(default = "ImageCfg::defaults")]
pub image: ImageCfg,
#[serde(default = "PdfCfg::defaults")]
@@ -809,6 +935,7 @@ impl Default for IngestCfg {
watch_filesystem: false,
chunking: ChunkingCfg::defaults(),
code: IngestCodeCfg::default(),
ocr: SharedOcrEngineCfg::default(),
image: ImageCfg::defaults(),
pdf: PdfCfg::defaults(),
}
@@ -916,6 +1043,7 @@ impl Config {
watch_filesystem: false,
chunking: ChunkingCfg::defaults(),
code: IngestCodeCfg::default(),
ocr: SharedOcrEngineCfg::default(),
image: ImageCfg::defaults(),
pdf: PdfCfg::defaults(),
},
@@ -1174,6 +1302,13 @@ impl Config {
cause: format!("parse_failed: {e}"),
})
})?;
// v5: merge `[ingest.ocr]` shared engine defaults down into the
// per-medium concrete OCR blocks. Driven by toml-level presence
// (a key explicitly written in `[ingest.image.ocr]` /
// `[ingest.pdf.ocr]` wins over the shared block) — so we hand the
// parsed `toml::Value` (which preserves presence) to the resolver.
let parsed_value = toml::from_str::<toml::Value>(&parse_text).ok();
cfg.resolve_ocr(parsed_value.as_ref());
cfg.validate_sources().map_err(|cause| {
anyhow::Error::new(ConfigInvalid {
path: path.to_path_buf(),
@@ -1261,6 +1396,192 @@ impl Config {
Ok(())
}
/// v5: merge the shared `[ingest.ocr]` engine block down into the
/// per-medium concrete OCR structs (`ingest.image.ocr` /
/// `ingest.pdf.ocr`). For each shared field that is `Some`, the value is
/// written into a medium's block **only if** that medium did not set the
/// field explicitly in its own table. Presence is read from `parsed`
/// (the raw `toml::Value` of the loaded file) because the typed struct
/// has already absorbed serde defaults and can no longer distinguish
/// "user wrote the default" from "omitted".
///
/// `parsed = None` (programmatic config, no source text) → only the
/// "explicitly set" guard is unavailable, but in that path
/// `ingest.ocr` is whatever the caller built; the common case
/// (`SharedOcrEngineCfg::default()` = all `None`) is a no-op. Idempotent.
pub(crate) fn resolve_ocr(&mut self, parsed: Option<&toml::Value>) {
// No shared overrides → nothing to merge (the overwhelming common
// case: configs without an `[ingest.ocr]` block).
if self.ingest.ocr == SharedOcrEngineCfg::default() {
return;
}
let shared = self.ingest.ocr.clone();
// Helper: was `key` explicitly present in `ingest.<medium>.ocr`?
let medium_has = |medium: &str, key: &str| -> bool {
parsed
.and_then(|v| v.get("ingest"))
.and_then(|v| v.get(medium))
.and_then(|v| v.get("ocr"))
.and_then(|v| v.get(key))
.is_some()
};
// ── image OCR ──────────────────────────────────────────────────
let img = &mut self.ingest.image.ocr;
if let Some(v) = &shared.enabled {
if !medium_has("image", "enabled") {
img.enabled = *v;
}
}
if let Some(v) = &shared.engine {
if !medium_has("image", "engine") {
img.engine = v.clone();
}
}
if let Some(v) = &shared.model {
if !medium_has("image", "model") {
img.model = v.clone();
}
}
if let Some(v) = &shared.endpoint {
if !medium_has("image", "endpoint") {
img.endpoint = Some(v.clone());
}
}
if let Some(v) = &shared.languages {
if !medium_has("image", "languages") {
img.languages = v.clone();
}
}
if let Some(v) = &shared.max_pixels {
if !medium_has("image", "max_pixels") {
img.max_pixels = *v;
}
}
if let Some(v) = &shared.request_timeout_secs {
if !medium_has("image", "request_timeout_secs") {
img.request_timeout_secs = *v;
}
}
if let Some(v) = &shared.det_model {
if !medium_has("image", "det_model") {
img.det_model = Some(v.clone());
}
}
if let Some(v) = &shared.rec_model {
if !medium_has("image", "rec_model") {
img.rec_model = Some(v.clone());
}
}
if let Some(v) = &shared.dict {
if !medium_has("image", "dict") {
img.dict = Some(v.clone());
}
}
if let Some(v) = &shared.score_thresh {
if !medium_has("image", "score_thresh") {
img.score_thresh = *v;
}
}
if let Some(v) = &shared.unclip_ratio {
if !medium_has("image", "unclip_ratio") {
img.unclip_ratio = *v;
}
}
if let Some(v) = &shared.max_boxes {
if !medium_has("image", "max_boxes") {
img.max_boxes = *v;
}
}
// ── pdf OCR (same shared fields; the 4 pdf-unique fields are never
// touched by the shared block) ─────────────────────────────────
let pdf = &mut self.ingest.pdf.ocr;
if let Some(v) = &shared.enabled {
if !medium_has("pdf", "enabled") {
pdf.enabled = *v;
}
}
if let Some(v) = &shared.engine {
if !medium_has("pdf", "engine") {
pdf.engine = v.clone();
}
}
if let Some(v) = &shared.model {
if !medium_has("pdf", "model") {
pdf.model = v.clone();
}
}
if let Some(v) = &shared.endpoint {
if !medium_has("pdf", "endpoint") {
pdf.endpoint = Some(v.clone());
}
}
if let Some(v) = &shared.languages {
if !medium_has("pdf", "languages") {
pdf.languages = v.clone();
}
}
if let Some(v) = &shared.max_pixels {
if !medium_has("pdf", "max_pixels") {
pdf.max_pixels = *v;
}
}
if let Some(v) = &shared.request_timeout_secs {
if !medium_has("pdf", "request_timeout_secs") {
pdf.request_timeout_secs = *v;
}
}
if let Some(v) = &shared.det_model {
if !medium_has("pdf", "det_model") {
pdf.det_model = Some(v.clone());
}
}
if let Some(v) = &shared.rec_model {
if !medium_has("pdf", "rec_model") {
pdf.rec_model = Some(v.clone());
}
}
if let Some(v) = &shared.dict {
if !medium_has("pdf", "dict") {
pdf.dict = Some(v.clone());
}
}
if let Some(v) = &shared.score_thresh {
if !medium_has("pdf", "score_thresh") {
pdf.score_thresh = *v;
}
}
if let Some(v) = &shared.unclip_ratio {
if !medium_has("pdf", "unclip_ratio") {
pdf.unclip_ratio = *v;
}
}
if let Some(v) = &shared.max_boxes {
if !medium_has("pdf", "max_boxes") {
pdf.max_boxes = *v;
}
}
}
/// Effective image OCR settings (`[ingest.ocr]` shared block merged with
/// the `[ingest.image.ocr]` override block). After [`Config::resolve_ocr`]
/// has run at load, the concrete `ingest.image.ocr` already holds the
/// resolved values, so this is the canonical read handle for OCR
/// consumers. (Decouples them from the merge mechanics — they ask the
/// `Config` for "image OCR" rather than reaching into the struct path.)
pub fn image_ocr(&self) -> &OcrCfg {
&self.ingest.image.ocr
}
/// Effective PDF OCR settings (shared `[ingest.ocr]` merged with the
/// `[ingest.pdf.ocr]` override + 4 pdf-unique fields). See
/// [`Config::image_ocr`].
pub fn pdf_ocr(&self) -> &PdfOcrCfg {
&self.ingest.pdf.ocr
}
/// Apply `KEBAB_<SECTION>_<KEY>` env overrides. Unknown keys are ignored.
///
/// The mapping is an explicit grep-friendly whitelist — one match arm
@@ -1456,64 +1777,93 @@ impl Config {
),
},
// image.ocr
"KEBAB_IMAGE_OCR_ENABLED" => {
self.ingest.image.ocr.enabled = parse_bool(v);
// ── shared OCR engine (v5: KEBAB_OCR_*) ──────────────────
// The 13 engine-level knobs collapsed from the v4
// KEBAB_IMAGE_OCR_* / KEBAB_PDF_OCR_* duplicate sets. Each
// arm writes BOTH the image and pdf concrete blocks (env is a
// deliberate "set the OCR engine for the whole workspace"
// override — applied after load-time `resolve_ocr`, with no
// toml presence to consult). Per-medium `enabled` stays
// separately addressable below (KEBAB_IMAGE_OCR_ENABLED /
// KEBAB_PDF_OCR_ENABLED) so a user can turn image OCR on
// without forcing pdf OCR on.
"KEBAB_OCR_ENGINE" => {
self.ingest.image.ocr.engine = v.clone();
self.ingest.pdf.ocr.engine = v.clone();
}
"KEBAB_IMAGE_OCR_ENGINE" => self.ingest.image.ocr.engine = v.clone(),
"KEBAB_IMAGE_OCR_MODEL" => self.ingest.image.ocr.model = v.clone(),
"KEBAB_IMAGE_OCR_ENDPOINT" => {
// Empty env value is treated the same as "fall back
// to models.llm.endpoint" — i.e. set None.
self.ingest.image.ocr.endpoint = if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_MODEL" => {
self.ingest.image.ocr.model = v.clone();
self.ingest.pdf.ocr.model = v.clone();
}
"KEBAB_IMAGE_OCR_LANGUAGES" => {
"KEBAB_OCR_ENDPOINT" => {
// Empty env value → None (= fall back to models.llm.endpoint).
let e = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.endpoint = e.clone();
self.ingest.pdf.ocr.endpoint = e;
}
"KEBAB_OCR_LANGUAGES" => {
// Comma-separated list, e.g. "eng,kor".
self.ingest.image.ocr.languages = v
let langs: Vec<String> = v
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
self.ingest.image.ocr.languages = langs.clone();
self.ingest.pdf.ocr.languages = langs;
}
"KEBAB_IMAGE_OCR_MAX_PIXELS" => {
"KEBAB_OCR_MAX_PIXELS" => {
if let Ok(n) = v.parse::<u32>() {
self.ingest.image.ocr.max_pixels = n;
self.ingest.pdf.ocr.max_pixels = n;
}
}
"KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS" => {
"KEBAB_OCR_REQUEST_TIMEOUT_SECS" => {
if let Ok(n) = v.parse::<u64>() {
self.ingest.image.ocr.request_timeout_secs = n;
self.ingest.pdf.ocr.request_timeout_secs = n;
}
}
// paddle-onnx engine overrides (v0.27.0). Empty string → None
// paddle-onnx engine overrides. Empty string → None
// (fall back to bundled / KEBAB_IMAGE_OCR_MODEL_DIR).
"KEBAB_IMAGE_OCR_DET_MODEL" => {
self.ingest.image.ocr.det_model =
if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_DET_MODEL" => {
let m = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.det_model = m.clone();
self.ingest.pdf.ocr.det_model = m;
}
"KEBAB_IMAGE_OCR_REC_MODEL" => {
self.ingest.image.ocr.rec_model =
if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_REC_MODEL" => {
let m = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.rec_model = m.clone();
self.ingest.pdf.ocr.rec_model = m;
}
"KEBAB_IMAGE_OCR_DICT" => {
self.ingest.image.ocr.dict = if v.is_empty() { None } else { Some(v.clone()) };
"KEBAB_OCR_DICT" => {
let m = if v.is_empty() { None } else { Some(v.clone()) };
self.ingest.image.ocr.dict = m.clone();
self.ingest.pdf.ocr.dict = m;
}
"KEBAB_IMAGE_OCR_SCORE_THRESH" => {
"KEBAB_OCR_SCORE_THRESH" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.image.ocr.score_thresh = f;
self.ingest.pdf.ocr.score_thresh = f;
}
}
"KEBAB_IMAGE_OCR_UNCLIP_RATIO" => {
"KEBAB_OCR_UNCLIP_RATIO" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.image.ocr.unclip_ratio = f;
self.ingest.pdf.ocr.unclip_ratio = f;
}
}
"KEBAB_IMAGE_OCR_MAX_BOXES" => {
"KEBAB_OCR_MAX_BOXES" => {
if let Ok(n) = v.parse::<usize>() {
self.ingest.image.ocr.max_boxes = n;
self.ingest.pdf.ocr.max_boxes = n;
}
}
// image OCR enabled toggle (kept per-medium addressable).
"KEBAB_IMAGE_OCR_ENABLED" => {
self.ingest.image.ocr.enabled = parse_bool(v);
}
// image.caption (P6-3)
"KEBAB_IMAGE_CAPTION_ENABLED" => {
self.ingest.image.caption.enabled = parse_bool(v);
@@ -1527,31 +1877,14 @@ impl Config {
self.ingest.image.caption.prompt_template_version = v.clone();
}
// pdf.ocr (v0.20.0 sub-item 1)
// ── pdf-only OCR knobs (v5: the 4 fields with no image
// counterpart + the per-medium enabled toggle) ──────────
// The engine-level pdf knobs (engine/model/endpoint/languages/
// max_pixels/request_timeout_secs + the 6 paddle fields) moved
// to the shared KEBAB_OCR_* set above. These five have no
// image analogue, so they stay pdf-specific.
"KEBAB_PDF_OCR_ENABLED" => self.ingest.pdf.ocr.enabled = parse_bool(v),
"KEBAB_PDF_OCR_ALWAYS_ON" => self.ingest.pdf.ocr.always_on = parse_bool(v),
"KEBAB_PDF_OCR_ENGINE" => self.ingest.pdf.ocr.engine = v.clone(),
"KEBAB_PDF_OCR_MODEL" => self.ingest.pdf.ocr.model = v.clone(),
"KEBAB_PDF_OCR_ENDPOINT" => {
self.ingest.pdf.ocr.endpoint = if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_LANGUAGES" => {
self.ingest.pdf.ocr.languages = v
.split(',')
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
}
"KEBAB_PDF_OCR_MAX_PIXELS" => {
if let Ok(n) = v.parse::<u32>() {
self.ingest.pdf.ocr.max_pixels = n;
}
}
"KEBAB_PDF_OCR_REQUEST_TIMEOUT_SECS" => {
if let Ok(n) = v.parse::<u64>() {
self.ingest.pdf.ocr.request_timeout_secs = n;
}
}
"KEBAB_PDF_OCR_VALID_RATIO_THRESHOLD" => {
if let Ok(n) = v.parse::<f32>() {
self.ingest.pdf.ocr.valid_ratio_threshold = n.clamp(0.0, 1.0);
@@ -1565,34 +1898,6 @@ impl Config {
"KEBAB_PDF_OCR_LANG_HINT" => {
self.ingest.pdf.ocr.lang_hint = if v.is_empty() { None } else { Some(v.clone()) };
}
// pdf paddle-onnx engine overrides (v3). image.ocr paddle 패턴 복제.
// Empty string → None (fall back to bundled / KEBAB_IMAGE_OCR_MODEL_DIR).
"KEBAB_PDF_OCR_DET_MODEL" => {
self.ingest.pdf.ocr.det_model =
if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_REC_MODEL" => {
self.ingest.pdf.ocr.rec_model =
if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_DICT" => {
self.ingest.pdf.ocr.dict = if v.is_empty() { None } else { Some(v.clone()) };
}
"KEBAB_PDF_OCR_SCORE_THRESH" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.pdf.ocr.score_thresh = f;
}
}
"KEBAB_PDF_OCR_UNCLIP_RATIO" => {
if let Ok(f) = v.parse::<f32>() {
self.ingest.pdf.ocr.unclip_ratio = f;
}
}
"KEBAB_PDF_OCR_MAX_BOXES" => {
if let Ok(n) = v.parse::<usize>() {
self.ingest.pdf.ocr.max_boxes = n;
}
}
// Unknown KEBAB_* keys are silently ignored — see
// `env_unknown_key_is_ignored` test.
@@ -1874,24 +2179,31 @@ max_pixels = 1600
env.insert("KEBAB_CHUNKING_TARGET_TOKENS".into(), "640".into());
env.insert("KEBAB_INDEXING_MAX_PARALLEL_EXTRACTORS".into(), "6".into());
env.insert("KEBAB_IMAGE_OCR_ENABLED".into(), "true".into());
env.insert("KEBAB_PDF_OCR_ENGINE".into(), "paddle-onnx".into());
// v5: shared engine knob — sets BOTH image and pdf OCR engine.
env.insert("KEBAB_OCR_ENGINE".into(), "paddle-onnx".into());
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.chunking.target_tokens, 640);
assert_eq!(c.ingest.max_parallel_extractors, 6);
assert!(c.ingest.image.ocr.enabled);
assert_eq!(c.ingest.image.ocr.engine, "paddle-onnx");
assert_eq!(c.ingest.pdf.ocr.engine, "paddle-onnx");
}
/// v5: the paddle engine overrides moved to the shared `KEBAB_OCR_*` set
/// and apply to BOTH mediums in one shot.
#[test]
fn env_pdf_paddle_symmetric_overrides() {
fn env_shared_ocr_paddle_overrides_both_mediums() {
let mut env = HashMap::new();
env.insert("KEBAB_PDF_OCR_DET_MODEL".into(), "/d.onnx".into());
env.insert("KEBAB_PDF_OCR_SCORE_THRESH".into(), "0.4".into());
env.insert("KEBAB_PDF_OCR_MAX_BOXES".into(), "500".into());
env.insert("KEBAB_OCR_DET_MODEL".into(), "/d.onnx".into());
env.insert("KEBAB_OCR_SCORE_THRESH".into(), "0.4".into());
env.insert("KEBAB_OCR_MAX_BOXES".into(), "500".into());
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.pdf.ocr.det_model.as_deref(), Some("/d.onnx"));
assert!((c.ingest.pdf.ocr.score_thresh - 0.4).abs() < 1e-6);
assert_eq!(c.ingest.pdf.ocr.max_boxes, 500);
assert_eq!(c.ingest.image.ocr.det_model.as_deref(), Some("/d.onnx"));
assert!((c.ingest.image.ocr.score_thresh - 0.4).abs() < 1e-6);
assert_eq!(c.ingest.image.ocr.max_boxes, 500);
}
#[test]
@@ -1989,12 +2301,14 @@ max_pixels = 1600
#[test]
fn env_overrides_image_ocr_request_timeout_secs() {
let mut env = HashMap::new();
// v5: shared KEBAB_OCR_REQUEST_TIMEOUT_SECS sets both mediums.
env.insert(
"KEBAB_IMAGE_OCR_REQUEST_TIMEOUT_SECS".to_string(),
"KEBAB_OCR_REQUEST_TIMEOUT_SECS".to_string(),
"900".to_string(),
);
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.image.ocr.request_timeout_secs, 900);
assert_eq!(c.ingest.pdf.ocr.request_timeout_secs, 900);
}
/// post-v0.17.1 dogfood: a config file written before the OCR
@@ -2136,25 +2450,24 @@ max_pixels = 1600
);
}
/// v5: the engine-level OCR knobs come from the shared `KEBAB_OCR_*` set
/// (sets image AND pdf); only `enabled` stays per-medium.
#[test]
fn image_ocr_env_overrides() {
let mut env = HashMap::new();
env.insert("KEBAB_IMAGE_OCR_ENABLED".to_string(), "true".to_string());
env.insert("KEBAB_OCR_MODEL".to_string(), "gemma4:31b".to_string());
env.insert(
"KEBAB_IMAGE_OCR_MODEL".to_string(),
"gemma4:31b".to_string(),
);
env.insert(
"KEBAB_IMAGE_OCR_ENDPOINT".to_string(),
"KEBAB_OCR_ENDPOINT".to_string(),
"http://192.168.0.47:11434".to_string(),
);
// Empty env value should map to None (= fall back to llm.endpoint).
// We exercise that branch in a separate test.
env.insert(
"KEBAB_IMAGE_OCR_LANGUAGES".to_string(),
"KEBAB_OCR_LANGUAGES".to_string(),
"eng, kor, jpn".to_string(),
);
env.insert("KEBAB_IMAGE_OCR_MAX_PIXELS".to_string(), "2048".to_string());
env.insert("KEBAB_OCR_MAX_PIXELS".to_string(), "2048".to_string());
let c = Config::defaults().apply_env(&env);
assert!(c.ingest.image.ocr.enabled);
assert_eq!(c.ingest.image.ocr.model, "gemma4:31b");
@@ -2164,6 +2477,9 @@ max_pixels = 1600
);
assert_eq!(c.ingest.image.ocr.languages, vec!["eng", "kor", "jpn"]);
assert_eq!(c.ingest.image.ocr.max_pixels, 2048);
// shared knob also reached the pdf block.
assert_eq!(c.ingest.pdf.ocr.model, "gemma4:31b");
assert_eq!(c.ingest.pdf.ocr.max_pixels, 2048);
}
/// Pre-P6 config files don't have an `[image]` section. The
@@ -2198,15 +2514,17 @@ max_pixels = 1600
assert_eq!(c.ingest.image.caption.prompt_template_version, "caption-v2");
}
/// `KEBAB_IMAGE_OCR_ENDPOINT=""` (empty value) should map to `None`
/// v5: `KEBAB_OCR_ENDPOINT=""` (empty value) should map to `None`
/// rather than to `Some("")` so the fallback to `models.llm.endpoint`
/// kicks in. Covers the env-equivalent of a missing TOML key.
/// kicks in (for both mediums). Covers the env-equivalent of a missing
/// TOML key.
#[test]
fn image_ocr_endpoint_empty_env_value_is_none() {
let mut env = HashMap::new();
env.insert("KEBAB_IMAGE_OCR_ENDPOINT".to_string(), String::new());
env.insert("KEBAB_OCR_ENDPOINT".to_string(), String::new());
let c = Config::defaults().apply_env(&env);
assert_eq!(c.ingest.image.ocr.endpoint, None);
assert_eq!(c.ingest.pdf.ocr.endpoint, None);
}
#[test]

225
crates/kebab-config/src/migrate.rs
View File

@@ -9,7 +9,7 @@ use toml_edit::{DocumentMut, Item};
/// 현재 바이너리가 이해하는 config 스키마 버전. 마이그레이션 완료 시
/// 사용자 파일의 `schema_version` 을 이 값으로 stamp 한다.
pub const CURRENT_SCHEMA_VERSION: u32 = 4;
pub const CURRENT_SCHEMA_VERSION: u32 = 5;
/// 한 번의 마이그레이션에서 발생한 개별 변경.
#[derive(Clone, Debug, PartialEq, serde::Serialize)]
@@ -80,6 +80,7 @@ fn section_comment(path: &str) -> Option<&'static str> {
"rag" => "# 답변 생성: prompt 템플릿·score gate·NLI.",
"ui" => "# TUI 팔레트·role 스타일.",
"ingest" => "# 모든 형식 ingest 우산: 병렬도 + chunking/code/image/pdf.",
"ingest.ocr" => "# 공유 OCR 엔진 설정(image/pdf 공통). 각 미디어 블록이 override.",
"ingest.chunking" => "# 청크 크기·오버랩·heading 존중(전 형식 공통).",
"ingest.code" => "# code ingest skip 정책(.gitignore 자동 honor).",
"ingest.image" => "# 이미지 OCR + 캡션(기본 off, asset 당 모델 호출 비용).",
@@ -142,6 +143,16 @@ pub fn annotated_default_document() -> DocumentMut {
let pretty = toml::to_string_pretty(&defaults).expect("defaults serialize");
let mut doc: DocumentMut = pretty.parse().expect("defaults parse as toml_edit");
// v5: 직렬화된 defaults 는 `[ingest.image.ocr]` 에 12개 엔진 키를 그대로
// 담고 `[ingest.ocr]` 은 비어 있다(`SharedOcrEngineCfg::default()` = 전부
// None). 참조 문서를 v5 canonical 형상으로 맞추기 위해 같은 통합을 적용한다
// — 그러지 않으면 reconcile 이 마이그레이션으로 끌어올린 키를 image 블록에
// 다시 추가해 통합을 되돌린다(그리고 image 의 effective engine 을 default 로
// 덮어써 동작을 바꾼다). pdf 블록은 그대로 둔다(자기 default 가 image 와
// 달라 override 로 유지).
let mut discard = Vec::new();
step_4_to_5(&mut doc, &mut discard);
// 헤더: 첫 최상위 항목의 prefix 로.
if let Some((mut first_key, _)) = doc.as_table_mut().iter_mut().next() {
first_key.leaf_decor_mut().set_prefix(format!("{HEADER}\n"));
@@ -413,6 +424,87 @@ pub fn step_3_to_4(doc: &mut DocumentMut, changes: &mut Vec<MigrationChange>) {
});
}
/// v5: `[ingest.image.ocr]` 의 12개 **엔진** 키를 새 공유 블록 `[ingest.ocr]`
/// 로 끌어올린다(`enabled` 은 미디어별 토글이라 제외 — 끌어올리면 공유 블록이
/// pdf 의 `enabled` 까지 켜버려 동작이 바뀐다). image 는 unique 필드가 없으므로
/// 공유 블록의 canonical source 로 삼는다. `[ingest.pdf.ocr]` 은 손대지 않는다
/// — reconcile 이 pdf 의 모든 키를 default 로 채워 명시 상태가 되므로(아래
/// `resolve_ocr` 의 "미디어가 명시한 키는 공유 overlay 가 덮지 않음" 규칙에 의해)
/// pdf 의 effective 값은 마이그레이션 전후 불변. 멱등: 키가 이미 옮겨졌으면 no-op.
///
/// 결과적으로 image 의 effective OCR 엔진 설정은 `[ingest.ocr]` 에서, pdf 는
/// 자신의 (reconcile 로 완전 채워진) 블록에서 그대로 resolve 되어 양쪽 모두
/// 마이그레이션 전 값을 유지한다 — `ingest_config_signature` 바이트도 불변.
const SHARED_OCR_ENGINE_KEYS: [&str; 12] = [
"engine",
"model",
"endpoint",
"languages",
"max_pixels",
"request_timeout_secs",
"det_model",
"rec_model",
"dict",
"score_thresh",
"unclip_ratio",
"max_boxes",
];
pub fn step_4_to_5(doc: &mut DocumentMut, changes: &mut Vec<MigrationChange>) {
// image OCR 블록이 없으면 끌어올릴 게 없음(reconcile 이 빈 `[ingest.ocr]`
// 를 추가). 멱등 진입점.
let img_present = doc
.get("ingest")
.and_then(|i| i.get("image"))
.and_then(|i| i.get("ocr"))
.and_then(Item::as_table)
.is_some();
if !img_present {
return;
}
let mut lifted_any = false;
for key in SHARED_OCR_ENGINE_KEYS {
// image.ocr 에 key 가 있고, ingest.ocr 에 아직 없으면 통째(decor 포함) 이동.
let has_in_image = doc
.get("ingest")
.and_then(|i| i.get("image"))
.and_then(|i| i.get("ocr"))
.and_then(Item::as_table)
.is_some_and(|t| t.contains_key(key));
if !has_in_image {
continue;
}
let already_in_shared = doc
.get("ingest")
.and_then(|i| i.get("ocr"))
.and_then(Item::as_table)
.is_some_and(|t| t.contains_key(key));
if already_in_shared {
// 공유 블록에 이미 있으면 image 쪽 중복 키는 그냥 제거(공유가 우선).
if let Some(img) = doc["ingest"]["image"]["ocr"].as_table_mut() {
img.remove(key);
}
continue;
}
move_table(
doc,
&["ingest", "image", "ocr", key],
&["ingest", "ocr", key],
changes,
);
lifted_any = true;
}
if lifted_any {
changes.push(MigrationChange {
kind: ChangeKind::AddedSection,
path: "ingest.ocr".to_string(),
detail: "OCR 엔진 키를 공유 [ingest.ocr] 로 통합(image/pdf 중복 제거)".to_string(),
});
}
}
/// 파일의 schema_version(없으면 1) 부터 CURRENT 까지 step 적용.
fn run_steps(doc: &mut DocumentMut, from: u32, changes: &mut Vec<MigrationChange>) {
if from < 2 {
@@ -424,6 +516,9 @@ fn run_steps(doc: &mut DocumentMut, from: u32, changes: &mut Vec<MigrationChange
if from < 4 {
step_3_to_4(doc, changes);
}
if from < 5 {
step_4_to_5(doc, changes);
}
}
/// 사용자 config.toml 텍스트를 받아 step 체인 + reconciliation + version
@@ -477,6 +572,21 @@ pub fn migrate_document(text: &str) -> MigrationOutcome {
mod tests {
use super::*;
/// v5: parse a config text and run the shared-OCR resolution the way
/// `Config::from_file` does (overlay `[ingest.ocr]` down into the
/// per-medium concrete blocks), then clear the now-applied shared block.
/// The result is the canonical *effective* config — comparable against
/// `Config::defaults()` regardless of whether the engine knobs live in
/// the shared block (annotated default doc) or the per-medium blocks
/// (in-memory `defaults()`).
fn parse_effective(text: &str) -> crate::Config {
let parsed = toml::from_str::<toml::Value>(text).ok();
let mut cfg: crate::Config = toml::from_str(text).expect("parse config");
cfg.resolve_ocr(parsed.as_ref());
cfg.ingest.ocr = crate::SharedOcrEngineCfg::default();
cfg
}
#[test]
fn annotated_default_has_per_key_comments() {
let text = annotated_default_document().to_string();
@@ -487,9 +597,9 @@ mod tests {
text.contains("paddle-onnx 는 번들 모델"),
"ocr.model 주석 누락:\n{text}"
);
// 주석 추가가 파싱을 깨지 않는다.
let back: crate::Config = toml::from_str(&text).expect("parse annotated default");
assert_eq!(back, crate::Config::defaults());
// 주석 추가가 파싱을 깨지 않고, v5 OCR resolution 후 effective 값이
// defaults 와 동일.
assert_eq!(parse_effective(&text), crate::Config::defaults());
}
#[test]
@@ -498,10 +608,11 @@ mod tests {
let text = doc.to_string();
// v3: 미디어 형식 섹션이 전부 `[ingest.*]` 하위로 통합됐다. IngestCfg
// 는 스칼라(병렬도) 필드가 있어 bare `[ingest]` + 하위 테이블이 함께
// 직렬화된다.
// 직렬화된다. v5: 공유 `[ingest.ocr]` 엔진 블록이 추가됐다.
for section in [
"[workspace]",
"[ingest]",
"[ingest.ocr]",
"[ingest.chunking]",
"[ingest.code]",
"[ingest.image.ocr]",
@@ -512,8 +623,8 @@ mod tests {
assert!(text.contains(section), "missing {section}:\n{text}");
}
assert!(text.contains("# "), "no comments attached");
let back: crate::Config = toml::from_str(&text).expect("parse annotated default");
assert_eq!(back, crate::Config::defaults());
// v5: effective 값(공유 OCR resolution 후)이 defaults 와 동일.
assert_eq!(parse_effective(&text), crate::Config::defaults());
}
#[test]
@@ -739,7 +850,7 @@ root = \"/my/notes\"
}
#[test]
fn migrate_document_v3_to_v4_adds_sources_and_is_idempotent() {
fn migrate_document_v3_to_current_adds_sources_and_is_idempotent() {
let v3 = "\
schema_version = 3
@@ -749,10 +860,10 @@ exclude = []
";
let outcome = migrate_document(v3);
assert_eq!(outcome.from_schema_version, 3);
assert_eq!(outcome.to_schema_version, 4);
assert_eq!(outcome.to_schema_version, CURRENT_SCHEMA_VERSION);
assert!(outcome.changed());
assert!(outcome.new_text.contains("[[workspace.sources]]"));
assert_eq!(read_schema_version(&outcome.new_text), 4);
assert_eq!(read_schema_version(&outcome.new_text), CURRENT_SCHEMA_VERSION);
let again = migrate_document(&outcome.new_text);
assert!(!again.changed(), "not idempotent: {:?}", again.changes);
assert_eq!(again.new_text, outcome.new_text);
@@ -765,4 +876,98 @@ exclude = []
assert_eq!(outcome.from_schema_version, 1);
assert_eq!(read_schema_version(&outcome.new_text), CURRENT_SCHEMA_VERSION);
}
/// v4 → v5 무손실 라운드트립: `[ingest.image.ocr]` / `[ingest.pdf.ocr]` 이
/// 채워진 v4 config 을 마이그레이션 → from_file 로 로드(공유 OCR resolution
/// 포함) → image/pdf OCR 의 effective 값이 마이그레이션 전과 정확히 동일해야
/// 한다. image 의 비-default engine(paddle-onnx) 이 공유 블록으로 끌어올려진
/// 뒤에도 보존되는지(이전 버그) + pdf 의 고유 값(qwen 모델·2048px)이 공유
/// overlay 에 오염되지 않는지를 함께 검증한다.
#[test]
fn migrate_v4_to_v5_preserves_effective_ocr() {
let v4 = "\
schema_version = 4
[workspace]
root = \"/my/notes\"
exclude = []
[[workspace.sources]]
id = \"default\"
root = \"/my/notes\"
[ingest.image.ocr]
enabled = true
engine = \"paddle-onnx\"
model = \"gemma4:e4b\"
languages = [\"eng\", \"kor\"]
max_pixels = 1280
request_timeout_secs = 450
det_model = \"/custom/det.onnx\"
score_thresh = 0.45
[ingest.pdf.ocr]
enabled = true
always_on = false
engine = \"ollama-vision\"
model = \"qwen2.5vl:7b\"
languages = [\"eng\", \"kor\"]
max_pixels = 2048
request_timeout_secs = 240
valid_ratio_threshold = 0.6
min_char_count = 25
lang_hint = \"kor\"
";
// pre-migration effective values, loaded the v4 way (no shared block).
let dir = std::env::temp_dir().join(format!("kebab_v5_rt_{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let p4 = dir.join("v4.toml");
std::fs::write(&p4, v4).unwrap();
let before = crate::Config::from_file(&p4).expect("load v4");
// migrate → load the v5 text via from_file (runs resolve_ocr).
let outcome = migrate_document(v4);
assert_eq!(outcome.from_schema_version, 4);
assert_eq!(outcome.to_schema_version, 5);
assert!(outcome.changed());
assert!(
outcome.new_text.contains("[ingest.ocr]"),
"shared block missing:\n{}",
outcome.new_text
);
let p5 = dir.join("v5.toml");
std::fs::write(&p5, &outcome.new_text).unwrap();
let after = crate::Config::from_file(&p5).expect("load v5");
// image OCR effective 값 전부 보존(특히 비-default engine paddle-onnx).
assert_eq!(after.image_ocr().enabled, before.image_ocr().enabled);
assert_eq!(after.image_ocr().engine, "paddle-onnx");
assert_eq!(after.image_ocr().engine, before.image_ocr().engine);
assert_eq!(after.image_ocr().model, before.image_ocr().model);
assert_eq!(after.image_ocr().languages, before.image_ocr().languages);
assert_eq!(after.image_ocr().max_pixels, 1280);
assert_eq!(after.image_ocr().max_pixels, before.image_ocr().max_pixels);
assert_eq!(
after.image_ocr().request_timeout_secs,
before.image_ocr().request_timeout_secs
);
assert_eq!(
after.image_ocr().det_model.as_deref(),
Some("/custom/det.onnx")
);
assert_eq!(after.image_ocr().det_model, before.image_ocr().det_model);
assert!((after.image_ocr().score_thresh - 0.45).abs() < 1e-6);
assert_eq!(after.image_ocr(), before.image_ocr());
// pdf OCR effective 값 전부 보존(공유 overlay 가 image 값으로 오염 X).
assert_eq!(after.pdf_ocr().engine, "ollama-vision");
assert_eq!(after.pdf_ocr().model, "qwen2.5vl:7b");
assert_eq!(after.pdf_ocr().max_pixels, 2048);
assert_eq!(after.pdf_ocr(), before.pdf_ocr());
// 멱등.
let again = migrate_document(&outcome.new_text);
assert!(!again.changed(), "v5 재실행 변경: {:?}", again.changes);
assert_eq!(again.new_text, outcome.new_text);
}
}

28
crates/kebab-config/tests/migrate_v3.rs
View File

@@ -11,9 +11,13 @@ const USER_V2: &str = include_str!("fixtures/user_v2_config.toml");
fn user_v2_migrates_losslessly() {
let out = migrate_document(USER_V2);
assert_eq!(out.from_schema_version, 2);
// v2 → CURRENT(=4): v3 의 [ingest.*] relocation 에 더해 v4 의
// [[workspace.sources]] default source 미러링까지 적용된다.
assert_eq!(out.to_schema_version, 4);
// v2 → CURRENT(=5): v3 의 [ingest.*] relocation, v4 의
// [[workspace.sources]] default source 미러링, v5 의 공유 [ingest.ocr]
// 통합까지 적용된다.
assert_eq!(
out.to_schema_version,
kebab_config::migrate::CURRENT_SCHEMA_VERSION
);
let t = &out.new_text;
// 사용자 값 보존.
@@ -36,15 +40,23 @@ fn user_v2_migrates_losslessly() {
assert!(!t.contains("\n[image.ocr]"));
assert!(!t.contains("\n[indexing]"));
// v3 Config 로 parse + 값 동일.
let cfg: kebab_config::Config = toml::from_str(t).expect("v3 parse");
assert!(cfg.ingest.image.ocr.enabled);
assert_eq!(cfg.ingest.image.ocr.engine, "paddle-onnx");
// v5: 공유 [ingest.ocr] 통합 후 image 엔진 키는 공유 블록에 산다.
assert!(t.contains("[ingest.ocr]"), "공유 OCR 블록 누락:\n{t}");
// effective 값은 from_file(공유 OCR resolution 포함)로 검증한다 —
// image 의 engine=paddle-onnx 가 공유 블록으로 끌어올려진 뒤에도 보존.
let dir = std::env::temp_dir().join(format!("kebab_mv3_{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let p = dir.join("config.toml");
std::fs::write(&p, t).unwrap();
let cfg = kebab_config::Config::from_file(&p).expect("v5 from_file");
assert!(cfg.image_ocr().enabled);
assert_eq!(cfg.image_ocr().engine, "paddle-onnx");
assert_eq!(cfg.models.embedding.model, "snowflake-arctic-embed2");
assert_eq!(cfg.models.llm.endpoint, "http://192.168.0.2:11943");
// pdf paddle 값 보존(v2 비대칭 → pdf 대칭 키로 복사). user 의 pdf.ocr 는
// engine=paddle-onnx 이고 자체 det_model 없으므로 번들(None) 유지.
assert_eq!(cfg.ingest.pdf.ocr.engine, "paddle-onnx");
assert_eq!(cfg.pdf_ocr().engine, "paddle-onnx");
// 멱등.
let again = migrate_document(t);

9
crates/kebab-config/tests/pdf_ocr.rs
View File

@@ -63,15 +63,14 @@ fn pdf_ocr_defaults_off_with_qwen_3b() {
assert_eq!(cfg.ingest.pdf.ocr.lang_hint.as_deref(), Some("kor"));
}
// Test 3: env var override — 4 keys 의 typical override case.
// Test 3: env var override — pdf-only keys + shared engine knob.
// v5: `model` moved to the shared `KEBAB_OCR_MODEL` (sets both mediums);
// `enabled`/`always_on`/`valid_ratio_threshold` stay pdf-specific.
#[test]
fn pdf_ocr_env_overrides() {
let mut env: HashMap<String, String> = HashMap::new();
env.insert("KEBAB_PDF_OCR_ENABLED".to_string(), "true".to_string());
env.insert(
"KEBAB_PDF_OCR_MODEL".to_string(),
"qwen2.5vl:7b".to_string(),
);
env.insert("KEBAB_OCR_MODEL".to_string(), "qwen2.5vl:7b".to_string());
env.insert("KEBAB_PDF_OCR_ALWAYS_ON".to_string(), "true".to_string());
env.insert(
"KEBAB_PDF_OCR_VALID_RATIO_THRESHOLD".to_string(),