diff --git a/tasks/p7/p7-1-pdf-text-extractor.md b/tasks/p7/p7-1-pdf-text-extractor.md new file mode 100644 index 0000000..1dfbcab --- /dev/null +++ b/tasks/p7/p7-1-pdf-text-extractor.md @@ -0,0 +1,121 @@ +--- +phase: P7 +component: kb-parse-pdf (text extractor) +task_id: p7-1 +title: "Text PDF extractor → CanonicalDocument with page-level blocks" +status: planned +depends_on: [p0-1, p1-6] +unblocks: [p7-2] +contract_source: ../../docs/superpowers/specs/2026-04-27-kb-final-form-design.md +contract_sections: [§3.4 SourceSpan::Page, §3.4 Block::Paragraph, §9.2 PDF text extraction, §9 versioning] +--- + +# p7-1 — PDF text extractor + +## Goal + +Implement `Extractor` for `MediaType::Pdf`. Extracts text page-by-page, emits one `Block::Paragraph` per page with `SourceSpan::Page`. Failed-text pages get an empty paragraph + `Provenance::Warning` so they can be picked up later by an OCR fallback pipeline. + +## Why now / why this size + +Strict scope: page text + page numbers. Layout reconstruction (multi-column merge, table extraction) is intentionally NOT in scope — it's its own engineering project. This task gets a usable PDF retrieval surface online with minimal moving parts. + +## Allowed dependencies + +- `kb-core` +- `kb-config` +- `pdf-extract = "0.7"` (or current stable) +- `lopdf = "0.32"` for page metadata (count, optional title from /Info) +- `serde`, `serde_json` +- `time` +- `tracing` +- `thiserror` + +## Forbidden dependencies + +- `kb-source-fs`, `kb-parse-md`, `kb-normalize`, `kb-chunk`, `kb-store-*`, `kb-embed*`, `kb-search`, `kb-llm*`, `kb-rag`, `kb-tui`, `kb-desktop`, OCR libraries (OCR fallback is a separate task, not this one) + +## Inputs + +| input | type | source | +|-------|------|--------| +| `RawAsset` | `kb_core::RawAsset` | `kb-source-fs` | +| PDF bytes | `&[u8]` | filesystem | + +## Outputs + +| output | type | downstream | +|--------|------|------------| +| `CanonicalDocument` | `kb_core::CanonicalDocument` | `kb-chunk` (`pdf-page-v1` chunker in p7-2) | + +## Public surface (signatures only — no new types) + +```rust +pub struct PdfTextExtractor; + +impl kb_core::Extractor for PdfTextExtractor { + fn supports(&self, m: &kb_core::MediaType) -> bool { matches!(m, kb_core::MediaType::Pdf) } + fn parser_version(&self) -> kb_core::ParserVersion { kb_core::ParserVersion("pdf-text-v1".into()) } + fn extract(&self, ctx: &kb_core::ExtractContext, bytes: &[u8]) -> anyhow::Result; +} +``` + +## Behavior contract + +- Page count obtained via `lopdf::Document::load_mem`; iterate `1..=n`. +- For each page: + - Try `pdf-extract::extract_text_from_mem_by_pages(bytes)` (or equivalent) to get a `Vec` aligned with pages. + - If extraction returns text for page i: produce `Block::Paragraph(TextBlock { common, text, inlines: vec![Inline::Text(text)] })` with `common.source_span = SourceSpan::Page { page: i, char_start: Some(0), char_end: Some(text.len() as u32) }` and `common.heading_path = vec![]`. + - If text is empty or extraction errored: produce `Block::Paragraph` with `text: ""`, `Provenance::Warning { note: "page empty (scanned candidate)" }`. +- `title` precedence: `/Info/Title` from `lopdf` (when non-empty) → filename without extension. +- `lang = Lang("und")` (PDFs rarely declare; lingua detection over the body could be a future enhancement). +- `metadata.user["pdf"] = { "page_count": n, "producer": "...", "creator": "..." }` from `/Info`. +- `metadata.source_type = SourceType::Paper`; `trust_level = TrustLevel::Primary`. +- `provenance` events: `Discovered`, `Parsed` (per page text or warning). +- `block_id` per design §4.2 with `block_kind = "paragraph"`, `heading_path = []`, `ordinal = page - 1`, `source_span = SourceSpan::Page { page }`. +- Streaming: read PDF in memory only once; do not load `pdf-extract` per page (that re-parses N times). +- Failure modes: + - File not a PDF / corrupt header → `anyhow::Error`. + - Encrypted PDF → `anyhow::Error` with hint to remove encryption (no decryption attempt in v1). +- Determinism: identical bytes → identical doc/block IDs and text. + +## Storage / wire effects + +- None directly. + +## Test plan + +| kind | description | fixture / data | +|------|-------------|----------------| +| unit | 3-page PDF produces 3 paragraph blocks with `SourceSpan::Page { page: 1..=3 }` | `fixtures/pdf/three-page-en.pdf` | +| unit | PDF with image-only page 2 (no text) emits warning + empty text for page 2 | `fixtures/pdf/scanned-mixed.pdf` | +| unit | encrypted PDF returns error with helpful hint | `fixtures/pdf/encrypted.pdf` | +| unit | corrupt header PDF returns error | `fixtures/pdf/corrupt.pdf` | +| unit | `metadata.user.pdf.page_count` matches actual count | inline | +| unit | Korean text PDF preserved (CID mapping permitting) | `fixtures/pdf/korean.pdf` | +| determinism | identical bytes → identical CanonicalDocument JSON across two runs | inline | +| snapshot | CanonicalDocument JSON for fixture stable | `fixtures/pdf/three-page-en.pdf` | + +All tests under `cargo test -p kb-parse-pdf`. + +## Definition of Done + +- [ ] `cargo check -p kb-parse-pdf` passes +- [ ] `cargo test -p kb-parse-pdf` passes +- [ ] No OCR / LLM code present +- [ ] No imports outside Allowed dependencies +- [ ] PR links design §3.4 SourceSpan::Page, §9.2 + +## Out of scope + +- OCR for scanned PDFs (separate future task; reuses p6-2 OCR adapter). +- Layout reconstruction (multi-column reading order, tables). +- Math rendering / formula detection. +- Form-field extraction. +- Bookmark / outline ingestion (could become heading_path later — note for P+). + +## Risks / notes + +- `pdf-extract` text quality varies wildly. For broken-glyph PDFs, the text may be unicode noise; downstream embedding still works but quality is poor. Mark such pages with a confidence-style warning when feasible. +- Some PDFs have layered text (selectable text + scanned image overlay). v1 captures the selectable text only. +- For very large PDFs (> 1k pages), memory usage may spike. Document a soft limit (`config.pdf.max_pages` default 5000) and refuse beyond it. diff --git a/tasks/p7/p7-2-pdf-page-chunker.md b/tasks/p7/p7-2-pdf-page-chunker.md new file mode 100644 index 0000000..ff6c6e7 --- /dev/null +++ b/tasks/p7/p7-2-pdf-page-chunker.md @@ -0,0 +1,114 @@ +--- +phase: P7 +component: kb-chunk (pdf-page-v1) +task_id: p7-2 +title: "PDF page-aware chunker (pdf-page-v1)" +status: planned +depends_on: [p7-1] +unblocks: [] +contract_source: ../../docs/superpowers/specs/2026-04-27-kb-final-form-design.md +contract_sections: [§3.5 Chunk, §4.2 chunk_id recipe, §0 Q3 citation, §9 versioning] +--- + +# p7-2 — PDF page chunker + +## Goal + +Implement `Chunker` with `chunker_version = "pdf-page-v1"`. Honors page boundaries (no chunk crosses a page) and subdivides long pages by paragraph budget. Produces the same `Chunk` shape as `md-heading-v1` so retrieval is uniform. + +## Why now / why this size + +Per-medium chunkers must stay tiny and obvious. Page-aware logic is small but its `chunker_version` label is load-bearing for downstream embedding records. + +## Allowed dependencies + +- `kb-core` +- `kb-config` +- `serde`, `serde_json` +- `blake3` (policy_hash) +- `serde-json-canonicalizer` +- `thiserror` + +## Forbidden dependencies + +- `kb-source-fs`, `kb-parse-md`, `kb-parse-pdf` (consumes `CanonicalDocument` via `kb-core` only), `kb-normalize`, `kb-store-*`, `kb-embed*`, `kb-search`, `kb-llm*`, `kb-rag`, `kb-tui`, `kb-desktop` + +## Inputs + +| input | type | source | +|-------|------|--------| +| `CanonicalDocument` (produced by `pdf-text-v1`) | `kb_core::CanonicalDocument` | p7-1 | +| `ChunkPolicy` | `kb_core::ChunkPolicy` | `kb-app` | + +## Outputs + +| output | type | downstream | +|--------|------|------------| +| `Vec` | `kb_core::Chunk` | `kb-store-sqlite`, `kb-embed*` | + +## Public surface (signatures only — no new types) + +```rust +pub struct PdfPageV1Chunker; + +impl kb_core::Chunker for PdfPageV1Chunker { + fn chunker_version(&self) -> kb_core::ChunkerVersion { kb_core::ChunkerVersion("pdf-page-v1".into()) } + fn policy_hash(&self, policy: &kb_core::ChunkPolicy) -> String; + fn chunk(&self, doc: &kb_core::CanonicalDocument, policy: &kb_core::ChunkPolicy) -> anyhow::Result>; +} +``` + +`policy_hash` = `blake3(canonical_json(policy))` truncated to 16 hex chars. + +## Behavior contract + +- Only operates on documents whose blocks all carry `SourceSpan::Page` (i.e., from `kb-parse-pdf`). Other documents → return `anyhow::Error("PdfPageV1Chunker only handles PDF docs")`. +- For each page block (1 block per page after p7-1): + - If `text.len()` (byte estimate) ≤ `policy.target_tokens * 4` (proxy for tokens) → emit one chunk for the entire page. + - Else → split by paragraphs (split text on `\n\n` or sentence-ending punctuation followed by whitespace) and group adjacent paragraphs until the running byte total approaches `policy.target_tokens * 4`. Apply `policy.overlap_tokens * 4` bytes of trailing overlap into the next chunk's prefix. +- A chunk NEVER crosses a page boundary. +- Each chunk's `source_spans` contains exactly one `SourceSpan::Page { page: i, char_start: Some(start), char_end: Some(end) }` with `start`/`end` in characters within the page. +- `heading_path = []` (PDFs have no heading tree at v1). +- `block_ids = [page_block.block_id]` (one block per chunk). +- `text` = the chunk's slice of page text. If overlap is applied, the slice includes the overlap prefix from the previous chunk. +- `token_estimate = byte_len / 4` (matches `md-heading-v1` proxy). +- `chunk_id` per design §4.2 with `(doc_id, "pdf-page-v1", block_ids, policy_hash)`. +- Determinism: identical inputs + identical policy → identical chunk IDs and text slices. + +## Storage / wire effects + +- None. + +## Test plan + +| kind | description | fixture / data | +|------|-------------|----------------| +| unit | 3-page PDF where each page < target_tokens → 3 chunks, 1 per page | seeded `CanonicalDocument` | +| unit | 1-page PDF whose text >> target_tokens → multiple chunks all on page 1 with overlap honored | seeded | +| unit | chunk crossing page boundary never produced | property test (10 random docs) | +| unit | empty page block → 0 chunks for that page (skipped) | inline | +| unit | non-PDF doc returns error | inline (Markdown-style doc) | +| determinism | same input → same chunk_ids twice | inline | +| snapshot | `Vec` JSON for fixture stable | `fixtures/pdf/three-page-en.pdf` (chunked) | + +All tests under `cargo test -p kb-chunk pdf`. + +## Definition of Done + +- [ ] `cargo check -p kb-chunk` passes (existing `md-heading-v1` continues to pass) +- [ ] `cargo test -p kb-chunk pdf` passes +- [ ] Snapshot stable across two runs +- [ ] No imports outside Allowed dependencies +- [ ] PR links design §3.5, §0 Q3, §9 + +## Out of scope + +- Token-accurate splitting (real tokenizer integration is P+). +- Cross-page sentence merging (kept off; page citation simplicity wins). +- Section/heading inference from font metadata (P+). + +## Risks / notes + +- Byte-based proxy can over- or under-estimate. The chunker is intentionally crude; a proper tokenizer slot lives in P3+ and replaces this proxy across all chunkers in one PR. +- Sentence-splitting uses simple regex; languages without clear sentence punctuation (e.g., Japanese) may produce uneven chunks. Document this and accept for v1. +- Bumping `chunker_version` to `pdf-page-v2` invalidates downstream embedding records for all PDFs; treat as a versioning event per §9.