feat(chunk): md-heading-v2 — 예산 초과 청크 일반 분할 (oversize-chunk split)

v1 의 "블록 미분할" 한계를 일반화. 거대 list/code/table/paragraph 가 한 청크로
임베더 컨텍스트를 초과해 임베딩이 통째로 실패하던 문제를 해소한다. v2 는 v1 과
모든 출력이 동일하되, 청크의 실제 임베드 text 크기(`text.len()/3`)가
`max_chunk_tokens`(신규 config, byte/3, default 4000)를 넘는 청크만 줄(`\n`)
경계로, 단일 거대 줄은 UTF-8 char 경계로 잘라 각 조각이 예산 이하가 되게 한다.

- 판정 기준은 저장 `token_estimate` 가 아니라 실제 `text` 길이: ImageRef/AudioRef
  청크는 image-only 규약으로 token_estimate=0 이지만 OCR/caption text 는 클 수
  있다(빽빽한 스크린샷). 도그푸딩 일치 재테스트에서 이 image-OCR 구멍 발견·수정.
- 분할 조각 chunk_id 는 동일 block_ids 를 공유하므로 id-input 해시에 `#seg{i}`
  접미사로 충돌 회피(저장 policy_hash 는 bare — pdf-page-v1 의 `#L` 레시피 동형).
- `max_chunk_tokens` 는 v2 의 policy_hash 에만 fold(공유 ChunkPolicy 미변경 →
  코드/PDF 청커 cascade 무영향). 값 변경 시 markdown 만 재청크.
- 미분할 청크는 v1 과 byte-identical. `chunker_version` v1→v2 → 다음 plain
  `kebab ingest` 에서 markdown 1회 자동 재청크(--force 불필요, 코드/PDF 무영향).

동기: strict 임베더(AMD Lemonade `/api/embed`)는 oversize 입력을 truncate 아닌
거부(`500 too large`) — ollama 가 조용히 truncate 하던 걸 청커가 애초에 안 만들게.

검증(실험 KB, arctic-embed-l-v2 @ Lemonade): v2 전 620 중 2 doc(거대 jira list
블록) 임베드 실패 → v2 후 620/620 errors=0, 7114 청크 전부 ≤ 4000. 사용자 실
config 일치 재테스트(이미지 OCR + PDF OCR paddle-onnx ON)에서 image-OCR 구멍
발견·수정 후 dense 이미지 OCR 텍스트가 budget 초과 시 분할(token_estimate=0 →
1청크였던 것이 실제 text 기준 다중 청크로) 실증. frozen 설계 doc / frozen p1-5
spec 미변경(설계 §9 가 md-heading-v2 라벨 bump 를 변경 메커니즘으로 명시).

known limitation: PDF 는 별도 청커 pdf-page-v1.1 이라 이 split 미적용(후속 후보).

Cargo.toml 0.29.0 → 0.30.0 (신규 config 키 + 청커 동작 변경 = pre-1.0 minor +
도그푸딩 트리거). docs cascade: HOTFIXES / release-notes-v0.30.0-draft /
plan(2026-06-24) / normalize-chunk README / README / SMOKE / HANDOFF.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Mc6W1fgsrbFKTsqA6P8La

crates/kebab-app/src/lib.rs

@@ -43,7 +43,7 @@ use kebab_chunk::{
     CodeCAstV1Chunker, CodeCppAstV1Chunker, CodeGoAstV1Chunker, CodeJavaAstV1Chunker,
     CodeJsAstV1Chunker, CodeKotlinAstV1Chunker, CodePythonAstV1Chunker, CodeRustAstV1Chunker,
     CodeTextParagraphV1Chunker, CodeTsAstV1Chunker, DockerfileFileV1Chunker,
-    K8sManifestResourceV1Chunker, ManifestFileV1Chunker, MdHeadingV1Chunker, PdfPageV1Chunker,
+    K8sManifestResourceV1Chunker, ManifestFileV1Chunker, MdHeadingV2Chunker, PdfPageV1Chunker,
 };
 use kebab_core::{
     Answer, Block, CanonicalDocument, Chunk, ChunkId, ChunkPolicy, Chunker, ChunkerVersion,
@@ -1388,7 +1388,7 @@ fn ingest_one_asset(
         app,
         asset,
         &eff_parser_version,
-        &MdHeadingV1Chunker.chunker_version(),
+        &md_chunker_from_config(&app.config).chunker_version(),
         embedder.map(|e| e.model_version()).as_ref(),
         force_reingest,
         None,
@@ -1441,9 +1441,9 @@ fn ingest_one_asset(
     let parse_ms = u64::try_from(t_parse.elapsed().as_millis()).unwrap_or(u64::MAX);
 
     let t_chunk = std::time::Instant::now();
-    let chunks = MdHeadingV1Chunker
+    let chunks = md_chunker_from_config(&app.config)
         .chunk(&canonical, chunk_policy)
-        .context("kb-chunk::MdHeadingV1Chunker::chunk")?;
+        .context("kb-chunk::MdHeadingV2Chunker::chunk")?;
     let chunk_ms = u64::try_from(t_chunk.elapsed().as_millis()).unwrap_or(u64::MAX);
 
     // v0.24.0: surface the chunk count immediately, before the (potentially
@@ -1465,7 +1465,7 @@ fn ingest_one_asset(
 
     // Stamp chunker + embedding versions so Task 7's skip detection has
     // data on the second run.
-    canonical.last_chunker_version = Some(MdHeadingV1Chunker.chunker_version());
+    canonical.last_chunker_version = Some(md_chunker_from_config(&app.config).chunker_version());
     if let Some(emb) = embedder {
         canonical.last_embedding_version = Some(emb.model_version());
     }
@@ -1607,7 +1607,7 @@ fn ingest_one_asset(
         block_count: u32::try_from(canonical.blocks.len()).ok(),
         chunk_count: u32::try_from(chunks.len()).ok(),
         parser_version: Some(parser_version.clone()),
-        chunker_version: Some(MdHeadingV1Chunker.chunker_version()),
+        chunker_version: Some(md_chunker_from_config(&app.config).chunker_version()),
         warnings: warning_notes,
         pdf_ocr_pages: None,
         pdf_ocr_ms_total: None,
@@ -1666,7 +1666,7 @@ fn ingest_one_image_asset(
     };
     // p9-fb-23 task 7: incremental-ingest early-skip for the image flow.
     // Image docs use the `image-meta-v1` parser_version + the same
-    // MdHeadingV1Chunker as the markdown flow (single-block doc). The
+    // MdHeadingV2Chunker as the markdown flow (single-block doc). The
     // embedding-version check matches the markdown path: when the
     // active embedder's model_version equals what was stamped on the
     // existing doc, the asset is Unchanged.
@@ -1679,7 +1679,7 @@ fn ingest_one_image_asset(
         app,
         asset,
         &eff_parser_version,
-        &MdHeadingV1Chunker.chunker_version(),
+        &md_chunker_from_config(&app.config).chunker_version(),
         embedder.map(|e| e.model_version()).as_ref(),
         force_reingest,
         None,
@@ -1828,14 +1828,17 @@ fn ingest_one_image_asset(
         }
     }
 
-    // 4. Chunk via the same `MdHeadingV1Chunker` markdown uses — its
+    // 4. Chunk via the same `MdHeadingV2Chunker` markdown uses — its
     //    `Block::ImageRef` arm already produces a single chunk per
-    //    image (P1-5). The chunk text now follows the (β) plain-concat
-    //    contract per the kebab-chunk render_block_text update.
+    //    image (P1-5). The chunk text follows the (β) plain-concat
+    //    contract per the kebab-chunk render_block_text update. Using v2
+    //    here keeps the markdown family consistent: a pathologically
+    //    large OCR text dump splits at line boundaries just like a giant
+    //    fenced code block would, instead of overflowing the embedder.
     let t_chunk = std::time::Instant::now();
-    let chunks = MdHeadingV1Chunker
+    let chunks = md_chunker_from_config(&app.config)
         .chunk(&canonical, chunk_policy)
-        .context("kb-chunk::MdHeadingV1Chunker::chunk (image)")?;
+        .context("kb-chunk::MdHeadingV2Chunker::chunk (image)")?;
     let chunk_ms = u64::try_from(t_chunk.elapsed().as_millis()).unwrap_or(u64::MAX);
 
     // v0.24.0: surface chunk count for the image path too.
@@ -1849,9 +1852,9 @@ fn ingest_one_image_asset(
     );
 
     // 5. Persist + embed — identical sequence to markdown.
-    // Stamp chunker + embedding versions (image uses MdHeadingV1Chunker
+    // Stamp chunker + embedding versions (image uses MdHeadingV2Chunker
     // for its single-block doc, so we record that version).
-    canonical.last_chunker_version = Some(MdHeadingV1Chunker.chunker_version());
+    canonical.last_chunker_version = Some(md_chunker_from_config(&app.config).chunker_version());
     if let Some(emb) = embedder {
         canonical.last_embedding_version = Some(emb.model_version());
     }
@@ -1956,7 +1959,7 @@ fn ingest_one_image_asset(
         block_count: u32::try_from(canonical.blocks.len()).ok(),
         chunk_count: u32::try_from(chunks.len()).ok(),
         parser_version: Some(canonical.parser_version.clone()),
-        chunker_version: Some(MdHeadingV1Chunker.chunker_version()),
+        chunker_version: Some(md_chunker_from_config(&app.config).chunker_version()),
         warnings: warning_notes,
         pdf_ocr_pages: None,
         pdf_ocr_ms_total: None,
@@ -3154,6 +3157,19 @@ fn chunk_policy_from_config(config: &kebab_config::Config) -> ChunkPolicy {
     }
 }
 
+/// Construct the markdown chunker (the hardcoded `md-heading-v2`) with the
+/// split budget threaded from config. Used by the markdown ingest path
+/// AND the image-OCR / caption path (which flows its synthetic
+/// `Block::ImageRef` text through the same chunker), so a giant OCR dump
+/// is split like any other oversize chunk. The PDF path stays pinned to
+/// `pdf-page-v1` and code paths keep their own AST chunkers — only the
+/// markdown-family default moved v1 → v2.
+fn md_chunker_from_config(config: &kebab_config::Config) -> MdHeadingV2Chunker {
+    MdHeadingV2Chunker {
+        max_chunk_tokens: config.ingest.chunking.max_chunk_tokens,
+    }
+}
+
 /// v0.26.2: deterministic signature of the **ingest-output-affecting**
 /// config for an asset's media type, folded into the effective
 /// `parser_version` (both the `try_skip_unchanged` compare field AND the
@@ -3240,9 +3256,20 @@ fn ingest_config_signature(config: &kebab_config::Config, media: &MediaType) ->
     // boundaries. `target_tokens` / `overlap_tokens` change re-chunking for
     // markdown / image / pdf / code alike, so a change re-indexes all types.
     let c = &config.ingest.chunking;
+    // `max_chunk_tokens` is appended as a 5th field: md-heading-v2
+    // splits any oversize chunk (list, code, paragraph, table) at this
+    // budget, so changing it moves markdown chunk boundaries and must
+    // re-index. It also folds into the v2 policy_hash, but the signature
+    // is what the no-`--force` skip-check compares, so it must be here
+    // too. Appended (not inserted) so the existing 4-field prefix
+    // `chunk:T:O:H:V` stays a stable substring for any existing golden.
     let mut sig = format!(
-        "chunk:{}:{}:{}:{}",
-        c.target_tokens, c.overlap_tokens, c.respect_markdown_headings, c.chunker_version
+        "chunk:{}:{}:{}:{}:{}",
+        c.target_tokens,
+        c.overlap_tokens,
+        c.respect_markdown_headings,
+        c.chunker_version,
+        c.max_chunk_tokens
     );
     match media {
         MediaType::Image(_) => {

crates/kebab-app/tests/config_invalidation.rs

@@ -160,8 +160,10 @@ fn ingest_signature_image_paddle_byte_stable() {
         &kebab_core::MediaType::Image(kebab_core::ImageType::Png),
     );
     // 골든: chunk:... |ocr:1:paddle-onnx:<engine_version> |cap:0
+    // md-heading-v2 가 markdown 기본값 + max_chunk_tokens(4000) 가
+    // signature 5번째 필드로 추가됐다 — budget 변경 시 자동 재색인용.
     assert!(
-        sig.starts_with("chunk:500:80:true:md-heading-v1"),
+        sig.starts_with("chunk:500:80:true:md-heading-v2:4000"),
         "chunk prefix drift: {sig}"
     );
     assert!(sig.contains("|ocr:1:paddle-onnx:"), "ocr token drift: {sig}");

crates/kebab-chunk/src/lib.rs

@@ -7,8 +7,15 @@
 //!
 //! * [`MdHeadingV1Chunker`] — heading-aware chunker for Markdown
 //!   `CanonicalDocument`s, emitting `chunker_version = "md-heading-v1"`.
+//! * [`MdHeadingV2Chunker`] — byte-identical to v1 in its chunking pass,
+//!   then applies a generic post-pass: any chunk whose byte/3 estimate
+//!   exceeds `max_chunk_tokens` is split at line (then UTF-8 char)
+//!   boundaries. Covers all block kinds (list, code, paragraph, table).
+//!   Emits `chunker_version = "md-heading-v2"`; the hardcoded markdown
+//!   default (design §9 label bump).
 //!
-//! Behavior contract is enumerated on [`MdHeadingV1Chunker`].
+//! Behavior contract is enumerated on [`MdHeadingV1Chunker`] (v2 inherits
+//! it; the divergence is the generic post-pass documented on [`MdHeadingV2Chunker`]).
 //!
 //! This crate must NOT depend on any parser implementation
 //! (`kb-parse-md`, `kb-parse-pdf`, …), the document/vector store, the
@@ -29,6 +36,7 @@ pub mod dockerfile_file_v1;
 pub mod k8s_manifest_resource_v1;
 pub mod manifest_file_v1;
 mod md_heading_v1;
+mod md_heading_v2;
 mod pdf_page_v1;
 mod tier2_shared;
 
@@ -46,6 +54,7 @@ pub use dockerfile_file_v1::DockerfileFileV1Chunker;
 pub use k8s_manifest_resource_v1::K8sManifestResourceV1Chunker;
 pub use manifest_file_v1::ManifestFileV1Chunker;
 pub use md_heading_v1::MdHeadingV1Chunker;
+pub use md_heading_v2::MdHeadingV2Chunker;
 pub use pdf_page_v1::PdfPageV1Chunker;
 
 // ── Korean morphological tokenizer ───────────────────────────────────────────

crates/kebab-config/src/lib.rs

@@ -175,6 +175,25 @@ pub struct ChunkingCfg {
     pub overlap_tokens: usize,
     pub respect_markdown_headings: bool,
     pub chunker_version: String,
+    /// Max byte/3 token estimate per emitted chunk (md-heading-v2).
+    /// After the v1-equivalent chunking pass, any chunk whose estimate
+    /// exceeds this value is split at line (then UTF-8 char) boundaries
+    /// into sub-pieces each ≤ budget. Covers all block kinds: list, code,
+    /// paragraph, table. The default (4000) is large enough to keep normal
+    /// content atomic while splitting pathological log / stacktrace / Jira
+    /// list dumps that would otherwise overflow an embedder context window.
+    /// `#[serde(default)]` so pre-v2 config files that predate the key
+    /// still load (migration injects it additively).
+    #[serde(default = "default_max_chunk_tokens")]
+    pub max_chunk_tokens: usize,
+}
+
+/// Default md-heading-v2 chunk split budget. 4000 byte/3 tokens
+/// (~12 KB) keeps ordinary source files and prose atomic while
+/// splitting the pathological 20k–76k token Jira log blocks that fail
+/// to embed on strict servers.
+fn default_max_chunk_tokens() -> usize {
+    4000
 }
 
 impl ChunkingCfg {
@@ -183,7 +202,12 @@ impl ChunkingCfg {
             target_tokens: 500,
             overlap_tokens: 80,
             respect_markdown_headings: true,
-            chunker_version: "md-heading-v1".to_string(),
+            // md-heading-v2 is the hardcoded markdown default (it splits
+            // oversize chunks of any block kind; v1 never did). Stamping it
+            // here means the lib.rs skip-check re-chunks md docs on next
+            // ingest via the version cascade (design §9).
+            chunker_version: "md-heading-v2".to_string(),
+            max_chunk_tokens: default_max_chunk_tokens(),
         }
     }
 }
@@ -1251,6 +1275,11 @@ impl Config {
                     self.ingest.chunking.respect_markdown_headings = parse_bool(v);
                 }
                 "KEBAB_CHUNKING_CHUNKER_VERSION" => self.ingest.chunking.chunker_version = v.clone(),
+                "KEBAB_CHUNKING_MAX_CHUNK_TOKENS" => {
+                    if let Ok(n) = v.parse::<usize>() {
+                        self.ingest.chunking.max_chunk_tokens = n;
+                    }
+                }
 
                 // models.embedding
                 "KEBAB_MODELS_EMBEDDING_PROVIDER" => self.models.embedding.provider = v.clone(),

crates/kebab-config/src/migrate.rs

@@ -108,6 +108,9 @@ fn key_comment(path: &str) -> Option<&'static str> {
         "ingest.max_parallel_embeddings" => "동시 임베딩 수.",
         "ingest.chunking.target_tokens" => "청크 목표 토큰(전 형식 공통).",
         "ingest.chunking.respect_markdown_headings" => "markdown heading 경계 존중.",
+        "ingest.chunking.max_chunk_tokens" => {
+            "md-heading-v2 청크 최대 토큰(byte/3). 초과 시 줄/문자 경계로 분할(list·code·단락 공통)."
+        }
         "ingest.image.ocr.enabled" => "이미지 OCR(기본 off, asset 당 비용).",
         "ingest.image.ocr.engine" => "ollama-vision | paddle-onnx.",
         "ingest.image.ocr.model" => "ollama-vision 전용. paddle-onnx 는 번들 모델 사용(이 값 무시).",