refactor(app): markdown을 extractor registry 경유로 통일 (extract stage 대칭화)

markdown ingest arm 이 그동안 유일하게 `App::extract_for` extractor
registry 를 우회하고 `kebab_parse_md::{parse_frontmatter, parse_blocks,
build_canonical_document}` free function 을 직접 호출했다 ("the single
biggest asymmetry"). 이를 image/pdf/code 와 동일하게 registry 경유로
통일.

- `kebab-parse-md` 에 `MarkdownExtractor` 신설 — 기존 free function 3종을
  동일 순서·동일 인자로 감싸 `bytes → CanonicalDocument` 생산만 담당.
  fm_span_end / count_lines_in / build_body_hints 헬퍼도 함께 이식.
- `App.extractors` registry 에 등록 (11 → 12 entry), markdown 이 `supports`
  로 발견되도록 첫 entry 로 배치.
- `ExtractContext` 에 `source_id` / `source_trust` 필드 추가 — markdown
  frontmatter 가 per-source trust 기본값을 override 하고 그 precedence 가
  `parse_frontmatter` *내부*에서 결정되므로 ctx 가 carry 해야 함. 다른
  extractor 는 None (post-extract 에서 source_id stamp 유지).
- 핸들러는 추출 stage 만 registry 로 이전 — version stamping / chunking /
  embedding / store 는 그대로. IngestItem.warnings 는 pdf/code 처럼
  `canonical.provenance` 의 Warning 이벤트에서 도출.

byte-identical 검증: parity gate-ingest (all-markdown 183 doc / 7676 chunk)
CHUNKS / SEARCH / ASK 모두 IDENTICAL. clippy 0, kebab-app + kebab-parse-md
test 전체 green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Mc6W1fgsrbFKTsqA6P8La
This commit is contained in:
2026-06-24 13:30:54 +00:00
parent 9f40c88722
commit 2bbe2f8ace
20 changed files with 247 additions and 79 deletions

View File

@@ -49,6 +49,7 @@ use kebab_parse_code::{
KotlinAstExtractor, PythonAstExtractor, RustAstExtractor, TypescriptAstExtractor,
};
use kebab_parse_image::ImageExtractor;
use kebab_parse_md::MarkdownExtractor;
use kebab_parse_pdf::PdfTextExtractor;
use kebab_rag::{AskOpts, RagPipeline};
use kebab_search::{HybridRetriever, LexicalRetriever, VectorRetriever};
@@ -99,9 +100,9 @@ pub struct App {
pub(crate) sqlite: Arc<SqliteStore>,
/// post-v0.18.0 extractor-dispatch-unification: polymorphic Extractor
/// registry. App init 시 1회 등록되어 `extract_for(...)` 가 lookup
/// 한다. 현재 11 entry (ImageExtractor + PdfTextExtractor + 9 AST).
/// MarkdownExtractor 는 별 PR 에서 추가 — markdown ingest path 는
/// 본 PR 에서 free-function 그대로 유지.
/// 한다. 현재 12 entry (MarkdownExtractor + ImageExtractor +
/// PdfTextExtractor + 9 AST). MarkdownExtractor 가 마지막으로 합류해
/// 모든 media 가 `extract_for` 경유로 통일됨 (extract-stage 대칭화).
pub(crate) extractors: Vec<Box<dyn Extractor + Send + Sync>>,
/// Memoized embedder — built lazily on first `embedder()` call when
/// embeddings are enabled. `OnceLock` keeps the struct `Sync` and
@@ -170,13 +171,16 @@ impl App {
"korean tokenizer backfill complete: {backfill_count} chunks updated"
);
}
// post-v0.18.0 extractor-dispatch-unification: build the 11-entry
// post-v0.18.0 extractor-dispatch-unification: build the 12-entry
// Extractor registry. All entries are state-less unit structs with
// zero-cost `new()`, so init cost is effectively 0 and side effects
// are 0 — `pipeline_verifier` fallible `?` below may bail but the
// already-constructed `extractors` Vec drops without cost. Markdown
// is NOT registered (see field doc).
// already-constructed `extractors` Vec drops without cost.
// MarkdownExtractor is registered first so markdown ingest flows
// through `extract_for` like every other media (extract-stage
// symmetry — previously the only free-function arm).
let extractors: Vec<Box<dyn Extractor + Send + Sync>> = vec![
Box::new(MarkdownExtractor::new()),
Box::new(ImageExtractor::new()),
Box::new(PdfTextExtractor::new()),
Box::new(RustAstExtractor::new()),
@@ -1138,7 +1142,7 @@ mod tests_trace {
/// are `pub(crate)` — integration tests cannot reach them.
///
/// Spec §5.1 + plan §2 Step 10 — 3 test class:
/// 1. registry length = 11 (image + pdf + 9 AST).
/// 1. registry length = 12 (markdown + image + pdf + 9 AST).
/// 2. mutually-exclusive `supports()` grid over 16 sample MediaTypes.
/// 3. `extract_for` returns `Err("no Extractor ...")` for registry-NOT-cover
/// MediaType (Audio).
@@ -1161,21 +1165,19 @@ mod tests_extractor_dispatch {
(dir, app)
}
/// Registry length invariant: 11 Extractor (image + pdf + 9 AST).
/// Markdown is NOT registered (free-function path — defer to a
/// separate PR per spec §3.4).
/// Registry length invariant: 12 Extractor (markdown + image + pdf +
/// 9 AST). Markdown 합류로 모든 media 가 `extract_for` 경유로 통일됨.
#[test]
fn registry_has_eleven_extractors() {
fn registry_has_twelve_extractors() {
let (_dir, app) = open_app_with_temp_dir();
assert_eq!(
app.extractors.len(),
11,
"registry must hold 11 Extractors (image + pdf + 9 AST). \
markdown 은 별 PR."
12,
"registry must hold 12 Extractors (markdown + image + pdf + 9 AST)."
);
}
/// 11 Extractor 의 `supports()` 가 16 sample MediaType 에 대해
/// 12 Extractor 의 `supports()` 가 16 sample MediaType 에 대해
/// mutually exclusive — 어떤 두 Extractor 도 동일 MediaType 에
/// 대해 true 반환 안 됨.
#[test]
@@ -1246,6 +1248,8 @@ mod tests_extractor_dispatch {
asset: &asset,
workspace_root: &workspace_root,
config: &cfg,
source_id: None,
source_trust: None,
};
let result = app.extract_for(&MediaType::Audio(AudioType::Wav), &ctx, &[]);
assert!(result.is_err(), "Audio 는 registry 미포함 → Err 기대");