altair823-org/kebab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 45 Wiki Activity

refactor(app): markdown을 extractor registry 경유로 통일 (extract stage 대칭화)

Browse Source 
markdown ingest arm 이 그동안 유일하게 `App::extract_for` extractor
registry 를 우회하고 `kebab_parse_md::{parse_frontmatter, parse_blocks,
build_canonical_document}` free function 을 직접 호출했다 ("the single
biggest asymmetry"). 이를 image/pdf/code 와 동일하게 registry 경유로
통일.

- `kebab-parse-md` 에 `MarkdownExtractor` 신설 — 기존 free function 3종을
  동일 순서·동일 인자로 감싸 `bytes → CanonicalDocument` 생산만 담당.
  fm_span_end / count_lines_in / build_body_hints 헬퍼도 함께 이식.
- `App.extractors` registry 에 등록 (11 → 12 entry), markdown 이 `supports`
  로 발견되도록 첫 entry 로 배치.
- `ExtractContext` 에 `source_id` / `source_trust` 필드 추가 — markdown
  frontmatter 가 per-source trust 기본값을 override 하고 그 precedence 가
  `parse_frontmatter` *내부*에서 결정되므로 ctx 가 carry 해야 함. 다른
  extractor 는 None (post-extract 에서 source_id stamp 유지).
- 핸들러는 추출 stage 만 registry 로 이전 — version stamping / chunking /
  embedding / store 는 그대로. IngestItem.warnings 는 pdf/code 처럼
  `canonical.provenance` 의 Warning 이벤트에서 도출.

byte-identical 검증: parity gate-ingest (all-markdown 183 doc / 7676 chunk)
CHUNKS / SEARCH / ASK 모두 IDENTICAL. clippy 0, kebab-app + kebab-parse-md
test 전체 green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Mc6W1fgsrbFKTsqA6P8La
This commit is contained in:
altair823
2026-06-24 13:30:54 +00:00
parent 9f40c88722
commit 2bbe2f8ace
20 changed files with 247 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
crates/kebab-app/src/app.rs
View File

@@ -49,6 +49,7 @@ use kebab_parse_code::{
KotlinAstExtractor, PythonAstExtractor, RustAstExtractor, TypescriptAstExtractor,
};
use kebab_parse_image::ImageExtractor;
use kebab_parse_md::MarkdownExtractor;
use kebab_parse_pdf::PdfTextExtractor;
use kebab_rag::{AskOpts, RagPipeline};
use kebab_search::{HybridRetriever, LexicalRetriever, VectorRetriever};
@@ -99,9 +100,9 @@ pub struct App {
pub(crate) sqlite: Arc<SqliteStore>,
/// post-v0.18.0 extractor-dispatch-unification: polymorphic Extractor
/// registry. App init 시 1회 등록되어 `extract_for(...)` 가 lookup
/// 한다. 현재 11 entry (ImageExtractor + PdfTextExtractor + 9 AST).
/// MarkdownExtractor 는 별 PR 에서 추가 — markdown ingest path 는
/// 본 PR 에서 free-function 그대로 유지.
/// 한다. 현재 12 entry (MarkdownExtractor + ImageExtractor +
/// PdfTextExtractor + 9 AST). MarkdownExtractor 가 마지막으로 합류해
/// 모든 media 가 `extract_for` 경유로 통일됨 (extract-stage 대칭화).
pub(crate) extractors: Vec<Box<dyn Extractor + Send + Sync>>,
/// Memoized embedder — built lazily on first `embedder()` call when
/// embeddings are enabled. `OnceLock` keeps the struct `Sync` and
@@ -170,13 +171,16 @@ impl App {
"korean tokenizer backfill complete: {backfill_count} chunks updated"
);
}
// post-v0.18.0 extractor-dispatch-unification: build the 11-entry
// post-v0.18.0 extractor-dispatch-unification: build the 12-entry
// Extractor registry. All entries are state-less unit structs with
// zero-cost `new()`, so init cost is effectively 0 and side effects
// are 0 — `pipeline_verifier` fallible `?` below may bail but the
// already-constructed `extractors` Vec drops without cost. Markdown
// is NOT registered (see field doc).
// already-constructed `extractors` Vec drops without cost.
// MarkdownExtractor is registered first so markdown ingest flows
// through `extract_for` like every other media (extract-stage
// symmetry — previously the only free-function arm).
let extractors: Vec<Box<dyn Extractor + Send + Sync>> = vec![
Box::new(MarkdownExtractor::new()),
Box::new(ImageExtractor::new()),
Box::new(PdfTextExtractor::new()),
Box::new(RustAstExtractor::new()),
@@ -1138,7 +1142,7 @@ mod tests_trace {
/// are `pub(crate)` — integration tests cannot reach them.
///
/// Spec §5.1 + plan §2 Step 10 — 3 test class:
/// 1. registry length = 11 (image + pdf + 9 AST).
/// 1. registry length = 12 (markdown + image + pdf + 9 AST).
/// 2. mutually-exclusive `supports()` grid over 16 sample MediaTypes.
/// 3. `extract_for` returns `Err("no Extractor ...")` for registry-NOT-cover
/// MediaType (Audio).
@@ -1161,21 +1165,19 @@ mod tests_extractor_dispatch {
(dir, app)
}
/// Registry length invariant: 11 Extractor (image + pdf + 9 AST).
/// Markdown is NOT registered (free-function path — defer to a
/// separate PR per spec §3.4).
/// Registry length invariant: 12 Extractor (markdown + image + pdf +
/// 9 AST). Markdown 합류로 모든 media 가 `extract_for` 경유로 통일됨.
#[test]
fn registry_has_eleven_extractors() {
fn registry_has_twelve_extractors() {
let (_dir, app) = open_app_with_temp_dir();
assert_eq!(
app.extractors.len(),
11,
"registry must hold 11 Extractors (image + pdf + 9 AST). \
markdown 은 별 PR."
12,
"registry must hold 12 Extractors (markdown + image + pdf + 9 AST)."
);
}
/// 11 Extractor 의 `supports()` 가 16 sample MediaType 에 대해
/// 12 Extractor 의 `supports()` 가 16 sample MediaType 에 대해
/// mutually exclusive — 어떤 두 Extractor 도 동일 MediaType 에
/// 대해 true 반환 안 됨.
#[test]
@@ -1246,6 +1248,8 @@ mod tests_extractor_dispatch {
asset: &asset,
workspace_root: &workspace_root,
config: &cfg,
source_id: None,
source_trust: None,
};
let result = app.extract_for(&MediaType::Audio(AudioType::Wav), &ctx, &[]);
assert!(result.is_err(), "Audio 는 registry 미포함 → Err 기대");

109
crates/kebab-app/src/lib.rs
View File

@@ -57,7 +57,6 @@ use kebab_parse_image::{
OLLAMA_VISION_ENGINE, OcrEngine, OllamaVisionOcr, OnnxPaddleOcr, PADDLE_ONNX_ENGINE,
apply_caption, apply_ocr, engine_version_for_paths,
};
use kebab_parse_md::{BodyHints, build_canonical_document, parse_blocks, parse_frontmatter};
use kebab_source_fs::FsSourceConnector;
mod app;
@@ -1394,41 +1393,47 @@ fn ingest_one_asset(
let bytes = std::fs::read(&path)
.with_context(|| format!("read asset bytes from {}", path.display()))?;
let body_hints = build_body_hints(asset, Some(source_id), source_trust);
// Frontmatter — `parse_frontmatter` returns Ok even on malformed
// frontmatter (warnings are surfaced through the `Vec<Warning>`).
let (metadata, fm_span, fm_warns) =
parse_frontmatter(&bytes, &body_hints).context("kb-parse-md::parse_frontmatter")?;
let body_offset_lines = match fm_span {
Some(span) => count_lines_in(&bytes[..span.end]),
None => 0,
// post-spine-cut: markdown extraction (bytes → CanonicalDocument) now
// flows through the `App.extractors` registry like pdf / image / code,
// instead of calling the `kebab_parse_md` free functions inline. The
// `MarkdownExtractor` runs the identical sequence (frontmatter parse →
// body-offset count → block parse → canonical lift, same args/order),
// so `doc_id` / `chunk_id` and the whole document stay byte-identical.
// `ExtractContext` carries `source_id` / `source_trust` because markdown
// frontmatter can override the per-source trust default and that
// precedence is resolved *inside* `parse_frontmatter`.
let extract_config = kebab_core::ExtractConfig::default();
// `~` / `${XDG_…}` expansion (HOTFIXES 2026-05-02 P9-4 follow-up).
// p9-fb-05: relative `workspace.root` resolves against the config
// file's directory (Config.source_dir), not the user's cwd.
let workspace_root = app.config.resolve_workspace_root();
let ctx = ExtractContext {
asset,
workspace_root: &workspace_root,
config: &extract_config,
source_id: Some(source_id),
source_trust,
};
let (parsed_blocks, blk_warns) =
parse_blocks(&bytes[fm_span_end(fm_span)..], body_offset_lines)
.context("kb-parse-md::parse_blocks")?;
let mut all_warnings = Vec::with_capacity(fm_warns.len() + blk_warns.len());
all_warnings.extend(fm_warns);
all_warnings.extend(blk_warns);
// Snapshot warning notes for the IngestItem before the vec is
// consumed by `build_canonical_document`.
let warning_notes: Vec<String> = all_warnings
.iter()
.map(|w| format!("{:?}: {}", w.kind, w.note))
.collect();
let mut canonical =
build_canonical_document(asset, metadata, parsed_blocks, parser_version, all_warnings)
.context("kb-parse-md::build_canonical_document")?;
let mut canonical = app
.extract_for(&asset.media_type, &ctx, &bytes)
.context("kb-app::extract_for (markdown)")?;
// v0.26.2: persist the composite parser_version (base|signature) so the
// next run's skip compare matches what was computed above. doc_id was
// already derived from the base version inside build_canonical_document.
canonical.parser_version = eff_parser_version.clone();
// Surface frontmatter / block warnings up to the IngestItem from the
// document's provenance (same shape pdf / code use). The extractor
// already encoded each upstream warning as a `Warning`-kind
// ProvenanceEvent with note `"{:?}: {}"` of `(kind, note)`.
let warning_notes: Vec<String> = canonical
.provenance
.events
.iter()
.filter(|e| e.kind == kebab_core::ProvenanceKind::Warning)
.filter_map(|e| e.note.clone())
.collect();
let parse_ms = u64::try_from(t_parse.elapsed().as_millis()).unwrap_or(u64::MAX);
let t_chunk = std::time::Instant::now();
@@ -1684,6 +1689,8 @@ fn ingest_one_image_asset(
asset,
workspace_root: &workspace_root,
config: &extract_config,
source_id: None,
source_trust: None,
};
let t_parse = std::time::Instant::now();
let mut canonical = app
@@ -2296,6 +2303,8 @@ fn ingest_one_pdf_asset(
asset,
workspace_root: &workspace_root,
config: &extract_config,
source_id: None,
source_trust: None,
};
let t_parse = std::time::Instant::now();
let mut canonical = app
@@ -2706,6 +2715,8 @@ fn ingest_one_code_asset(
asset,
workspace_root: &workspace_root,
config: &extract_config,
source_id: None,
source_trust: None,
};
// post-v0.18.0 extractor-dispatch-unification:
@@ -3102,40 +3113,10 @@ fn lang_hint_from_doc(doc: &CanonicalDocument) -> Option<Lang> {
}
}
/// Convenience: end byte of the frontmatter region (or 0 when absent).
fn fm_span_end(span: Option<kebab_parse_md::FrontmatterSpan>) -> usize {
span.map_or(0, |s| s.end)
}
/// Count `\n` in a byte prefix to convert frontmatter byte span to
/// the line-offset `parse_blocks` expects.
fn count_lines_in(bytes: &[u8]) -> u32 {
let n = bytes.iter().filter(|&&b| b == b'\n').count();
u32::try_from(n).unwrap_or(u32::MAX)
}
/// Build `BodyHints` from the asset alone. We use the asset's
/// `discovered_at` for both `fs_ctime` and `fs_mtime` because going
/// through the FS metadata API for every file would be a noticeable
/// overhead for large workspaces and the source-of-truth timestamps
/// are written into the document's frontmatter when the user wants
/// authoritative values.
fn build_body_hints(
asset: &RawAsset,
source_id: Option<&str>,
source_trust: Option<TrustLevel>,
) -> BodyHints {
BodyHints {
first_h1: None,
fs_ctime: asset.discovered_at,
fs_mtime: asset.discovered_at,
fallback_lang: None,
// `[[workspace.sources]]`: stamp the owning source id + inject the
// per-source default trust level (frontmatter still overrides it).
source_id: source_id.map(str::to_string),
fallback_trust_level: source_trust,
}
}
// `fm_span_end` / `count_lines_in` / `build_body_hints` moved into
// `kebab_parse_md::extractor` (the `MarkdownExtractor`) when the markdown
// ingest arm was unified onto the `App.extractors` registry — they were
// only ever the inline frontmatter→blocks→canonical plumbing.
/// Build a `ChunkPolicy` from the active config.
fn chunk_policy_from_config(config: &kebab_config::Config) -> ChunkPolicy {

2
crates/kebab-app/tests/multi_scanned_pdf_ingest_no_chunk_id_collision.rs
View File

@@ -52,6 +52,8 @@ fn extract_and_ocr(
asset: &asset,
workspace_root,
config: &config,
source_id: None,
source_trust: None,
};
let mut canonical = PdfTextExtractor::new().extract(&ctx, bytes).unwrap();
let opts = PdfOcrOpts {

2
crates/kebab-app/tests/pdf_ocr_apply.rs
View File

@@ -49,6 +49,8 @@ fn extract_canonical_from_bytes(bytes: &[u8]) -> CanonicalDocument {
asset: &asset,
workspace_root,
config: &config,
source_id: None,
source_trust: None,
};
PdfTextExtractor::new().extract(&ctx, bytes).unwrap()
}

2
crates/kebab-chunk/tests/code_cpp_ast_snapshot.rs
View File

@@ -171,6 +171,8 @@ fn extract_cpp_fixture() -> CanonicalDocument {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
CppAstExtractor::new()
.extract(&ctx, src.as_bytes())

13
crates/kebab-core/src/traits.rs
View File

@@ -39,6 +39,19 @@ pub struct ExtractContext<'a> {
pub asset: &'a RawAsset,
pub workspace_root: &'a Path,
pub config: &'a ExtractConfig,
/// `[[workspace.sources]]`: id of the source this asset is being
/// ingested from. The markdown extractor threads it into `BodyHints`
/// so `parse_frontmatter` stamps `Metadata.source_id` (frontmatter
/// does not override it). Other extractors (pdf / image / code) leave
/// it `None` and the kebab-app handler stamps `source_id` post-extract.
pub source_id: Option<&'a str>,
/// `[[workspace.sources]]`: per-source default `trust_level`. The
/// markdown extractor threads it into `BodyHints` so `parse_frontmatter`
/// can apply the precedence chain (frontmatter > this default >
/// hardcoded `Primary`) *inside* extraction — which is why it must be
/// carried here rather than stamped after. `None` for non-markdown
/// extractors (their frontmatter never carries a `trust_level`).
pub source_trust: Option<crate::metadata::TrustLevel>,
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]

2
crates/kebab-parse-code/src/c.rs
View File

@@ -438,6 +438,8 @@ pub(crate) mod tests_support {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
CAstExtractor::new().extract(&ctx, src.as_bytes()).unwrap()
}

2
crates/kebab-parse-code/src/cpp.rs
View File

@@ -646,6 +646,8 @@ pub(crate) mod tests_support {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
CppAstExtractor::new()
.extract(&ctx, src.as_bytes())

2
crates/kebab-parse-code/src/go.rs
View File

@@ -392,6 +392,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
GoAstExtractor::new().extract(&ctx, &bytes).unwrap()
}

2
crates/kebab-parse-code/src/java.rs
View File

@@ -454,6 +454,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
JavaAstExtractor::new().extract(&ctx, &bytes).unwrap()
}

6
crates/kebab-parse-code/src/javascript.rs
View File

@@ -460,6 +460,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
JavascriptAstExtractor::new().extract(&ctx, &bytes).unwrap()
}
@@ -510,6 +512,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
let doc = JavascriptAstExtractor::new().extract(&ctx, bytes).unwrap();
let syms = symbols(&doc);
@@ -542,6 +546,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
let doc = JavascriptAstExtractor::new().extract(&ctx, bytes).unwrap();

2
crates/kebab-parse-code/src/kotlin.rs
View File

@@ -532,6 +532,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
KotlinAstExtractor::new().extract(&ctx, &bytes).unwrap()
}

2
crates/kebab-parse-code/src/python.rs
View File

@@ -397,6 +397,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
PythonAstExtractor::new().extract(&ctx, &bytes).unwrap()
}

4
crates/kebab-parse-code/src/rust.rs
View File

@@ -400,6 +400,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
RustAstExtractor::new().extract(&ctx, &bytes).unwrap()
}
@@ -463,6 +465,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
let doc = RustAstExtractor::new()
.extract(&ctx, source.as_bytes())

6
crates/kebab-parse-code/src/typescript.rs
View File

@@ -501,6 +501,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
TypescriptAstExtractor::new().extract(&ctx, &bytes).unwrap()
}
@@ -587,6 +589,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
let doc = TypescriptAstExtractor::new().extract(&ctx, bytes).unwrap();
@@ -640,6 +644,8 @@ mod tests {
asset: &asset,
workspace_root: &root,
config: &cfg,
source_id: None,
source_trust: None,
};
let doc = TypescriptAstExtractor::new().extract(&ctx, bytes).unwrap();

2
crates/kebab-parse-image/tests/common/mod.rs
View File

@@ -282,6 +282,8 @@ impl ImageFixture {
asset: &self.asset,
workspace_root: &self.workspace_root,
config: &self.config,
source_id: None,
source_trust: None,
}
}
}

122
crates/kebab-parse-md/src/extractor.rs Normal file
View File

@@ -0,0 +1,122 @@
//! `kb-parse-md::extractor` — the [`Extractor`] trait impl that wraps the
//! crate's free functions (`parse_frontmatter` + `parse_blocks` +
//! `build_canonical_document`) so markdown ingest flows through the same
//! `App.extractors` registry + `App::extract_for` polymorphic dispatch
//! that pdf / image / code already use.
//!
//! This is a pure structural unification: the byte sequence it runs is
//! identical to the inline arm `kebab-app::ingest_one_asset` used before
//! (frontmatter parse → body-offset count → block parse → canonical lift,
//! same args, same order), so the produced `CanonicalDocument` — and thus
//! `doc_id` / `chunk_id` — is byte-for-byte the same.
//!
//! The one piece of context the inline arm read that the other extractors
//! do not is the per-source `source_id` + `trust_level`: markdown
//! frontmatter can *override* the per-source trust default, and that
//! precedence is resolved *inside* `parse_frontmatter` via [`BodyHints`].
//! [`ExtractContext`] carries both so the resolution stays identical.
use kebab_core::{CanonicalDocument, ExtractContext, Extractor, MediaType, ParserVersion, RawAsset};
use crate::PARSER_VERSION;
use crate::frontmatter::{BodyHints, FrontmatterSpan, parse_frontmatter};
use crate::{build_canonical_document, parse_blocks};
/// Markdown extractor — wraps the crate's free functions behind the
/// [`Extractor`] trait.
pub struct MarkdownExtractor;
impl MarkdownExtractor {
pub fn new() -> Self {
Self
}
}
impl Default for MarkdownExtractor {
fn default() -> Self {
Self::new()
}
}
impl Extractor for MarkdownExtractor {
fn supports(&self, m: &MediaType) -> bool {
matches!(m, MediaType::Markdown)
}
fn parser_version(&self) -> ParserVersion {
ParserVersion(PARSER_VERSION.to_string())
}
fn extract(
&self,
ctx: &ExtractContext<'_>,
bytes: &[u8],
) -> anyhow::Result<CanonicalDocument> {
let asset = ctx.asset;
let parser_version = self.parser_version();
// `[[workspace.sources]]`: stamp the owning source id + inject the
// per-source default trust level (frontmatter still overrides it).
// Mirrors the old inline `build_body_hints` exactly.
let body_hints = build_body_hints(asset, ctx.source_id, ctx.source_trust);
// Frontmatter — `parse_frontmatter` returns Ok even on malformed
// frontmatter (warnings are surfaced through the `Vec<Warning>`).
use anyhow::Context as _;
let (metadata, fm_span, fm_warns) =
parse_frontmatter(bytes, &body_hints).context("kb-parse-md::parse_frontmatter")?;
let body_offset_lines = match fm_span {
Some(span) => count_lines_in(&bytes[..span.end]),
None => 0,
};
let (parsed_blocks, blk_warns) =
parse_blocks(&bytes[fm_span_end(fm_span)..], body_offset_lines)
.context("kb-parse-md::parse_blocks")?;
let mut all_warnings = Vec::with_capacity(fm_warns.len() + blk_warns.len());
all_warnings.extend(fm_warns);
all_warnings.extend(blk_warns);
let canonical =
build_canonical_document(asset, metadata, parsed_blocks, &parser_version, all_warnings)
.context("kb-parse-md::build_canonical_document")?;
Ok(canonical)
}
}
/// Build `BodyHints` from the asset alone. We use the asset's
/// `discovered_at` for both `fs_ctime` and `fs_mtime` because going
/// through the FS metadata API for every file would be a noticeable
/// overhead for large workspaces and the source-of-truth timestamps
/// are written into the document's frontmatter when the user wants
/// authoritative values.
fn build_body_hints(
asset: &RawAsset,
source_id: Option<&str>,
source_trust: Option<kebab_core::TrustLevel>,
) -> BodyHints {
BodyHints {
first_h1: None,
fs_ctime: asset.discovered_at,
fs_mtime: asset.discovered_at,
fallback_lang: None,
// `[[workspace.sources]]`: stamp the owning source id + inject the
// per-source default trust level (frontmatter still overrides it).
source_id: source_id.map(str::to_string),
fallback_trust_level: source_trust,
}
}
/// Convenience: end byte of the frontmatter region (or 0 when absent).
fn fm_span_end(span: Option<FrontmatterSpan>) -> usize {
span.map_or(0, |s| s.end)
}
/// Count `\n` in a byte prefix to convert frontmatter byte span to
/// the line-offset `parse_blocks` expects.
fn count_lines_in(bytes: &[u8]) -> u32 {
let n = bytes.iter().filter(|&&b| b == b'\n').count();
u32::try_from(n).unwrap_or(u32::MAX)
}

6
crates/kebab-parse-md/src/lib.rs
View File

@@ -18,6 +18,10 @@
//! * [`build_canonical_document`] / [`derive_title`] — lift a parsed
//! markdown document into a `kebab_core::CanonicalDocument` (absorbed
//! from `kebab-normalize` — P1-4 / p9-fb-07 frozen API).
//! * [`MarkdownExtractor`] — the [`kebab_core::Extractor`] impl that wraps
//! the three free functions above so markdown ingest flows through the
//! `App.extractors` registry like pdf / image / code (extract-stage
//! symmetry).
//! * Parser intermediate types ([`ParsedBlock`], [`ParsedBlockKind`],
//! [`ParsedPayload`], [`Warning`], [`WarningKind`]) and 3 forward-declared
//! structs ([`ParsedImageRegion`], [`ParsedPdfPage`], [`ParsedAudioSegment`]) —
@@ -26,11 +30,13 @@
//! Anything else in this crate is `pub(crate)` and may change without notice.
pub mod blocks;
mod extractor;
pub mod frontmatter;
mod normalize;
mod types;
pub use blocks::parse_blocks;
pub use extractor::MarkdownExtractor;
pub use frontmatter::{BodyHints, FrontmatterSpan, parse_frontmatter};
// Spec §3.3 의 surface 보존 정책 — explicit (NOT glob) 으로 future addition leak 방지.

2
crates/kebab-parse-pdf/tests/common/mod.rs
View File

@@ -169,6 +169,8 @@ impl PdfFixture {
asset: &self.asset,
workspace_root: &self.workspace_root,
config: &self.config,
source_id: None,
source_trust: None,
}
}
}

4
crates/kebab-parse-pdf/tests/text_extractor_regression.rs
View File

@@ -47,6 +47,8 @@ fn vector_pdf_extract_byte_identical_to_baseline() {
asset: &asset,
workspace_root,
config: &config,
source_id: None,
source_trust: None,
};
let mut canonical = PdfTextExtractor::new()
@@ -96,6 +98,8 @@ fn pdf_text_extractor_on_mojibake_yields_one_block() {
asset: &asset,
workspace_root,
config: &config,
source_id: None,
source_trust: None,
};
let canonical = PdfTextExtractor::new()
.extract(&ctx, bytes)