altair823-org/kebab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 40 Wiki Activity

tasks: add P7 component specs (pdf-extractor, pdf-chunker)

Browse Source
This commit is contained in:
kb
2026-04-27 12:07:56 +00:00
parent c84ab03404
commit d96d9cc56c
2 changed files with 235 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
tasks/p7/p7-1-pdf-text-extractor.md Normal file
View File

@@ -0,0 +1,121 @@
---
phase: P7
component: kb-parse-pdf (text extractor)
task_id: p7-1
title: "Text PDF extractor → CanonicalDocument with page-level blocks"
status: planned
depends_on: [p0-1, p1-6]
unblocks: [p7-2]
contract_source: ../../docs/superpowers/specs/2026-04-27-kb-final-form-design.md
contract_sections: [§3.4 SourceSpan::Page, §3.4 Block::Paragraph, §9.2 PDF text extraction, §9 versioning]
---
# p7-1 — PDF text extractor
## Goal
Implement `Extractor` for `MediaType::Pdf`. Extracts text page-by-page, emits one `Block::Paragraph` per page with `SourceSpan::Page`. Failed-text pages get an empty paragraph + `Provenance::Warning` so they can be picked up later by an OCR fallback pipeline.
## Why now / why this size
Strict scope: page text + page numbers. Layout reconstruction (multi-column merge, table extraction) is intentionally NOT in scope — it's its own engineering project. This task gets a usable PDF retrieval surface online with minimal moving parts.
## Allowed dependencies
- `kb-core`
- `kb-config`
- `pdf-extract = "0.7"` (or current stable)
- `lopdf = "0.32"` for page metadata (count, optional title from /Info)
- `serde`, `serde_json`
- `time`
- `tracing`
- `thiserror`
## Forbidden dependencies
- `kb-source-fs`, `kb-parse-md`, `kb-normalize`, `kb-chunk`, `kb-store-*`, `kb-embed*`, `kb-search`, `kb-llm*`, `kb-rag`, `kb-tui`, `kb-desktop`, OCR libraries (OCR fallback is a separate task, not this one)
## Inputs
| input | type | source |
|-------|------|--------|
| `RawAsset` | `kb_core::RawAsset` | `kb-source-fs` |
| PDF bytes | `&[u8]` | filesystem |
## Outputs
| output | type | downstream |
|--------|------|------------|
| `CanonicalDocument` | `kb_core::CanonicalDocument` | `kb-chunk` (`pdf-page-v1` chunker in p7-2) |
## Public surface (signatures only — no new types)
```rust
pub struct PdfTextExtractor;
impl kb_core::Extractor for PdfTextExtractor {
fn supports(&self, m: &kb_core::MediaType) -> bool { matches!(m, kb_core::MediaType::Pdf) }
fn parser_version(&self) -> kb_core::ParserVersion { kb_core::ParserVersion("pdf-text-v1".into()) }
fn extract(&self, ctx: &kb_core::ExtractContext, bytes: &[u8]) -> anyhow::Result<kb_core::CanonicalDocument>;
}
```
## Behavior contract
- Page count obtained via `lopdf::Document::load_mem`; iterate `1..=n`.
- For each page:
- Try `pdf-extract::extract_text_from_mem_by_pages(bytes)` (or equivalent) to get a `Vec<String>` aligned with pages.
- If extraction returns text for page i: produce `Block::Paragraph(TextBlock { common, text, inlines: vec![Inline::Text(text)] })` with `common.source_span = SourceSpan::Page { page: i, char_start: Some(0), char_end: Some(text.len() as u32) }` and `common.heading_path = vec![]`.
- If text is empty or extraction errored: produce `Block::Paragraph` with `text: ""`, `Provenance::Warning { note: "page<i> empty (scanned candidate)" }`.
- `title` precedence: `/Info/Title` from `lopdf` (when non-empty) → filename without extension.
- `lang = Lang("und")` (PDFs rarely declare; lingua detection over the body could be a future enhancement).
- `metadata.user["pdf"] = { "page_count": n, "producer": "...", "creator": "..." }` from `/Info`.
- `metadata.source_type = SourceType::Paper`; `trust_level = TrustLevel::Primary`.
- `provenance` events: `Discovered`, `Parsed` (per page text or warning).
- `block_id` per design §4.2 with `block_kind = "paragraph"`, `heading_path = []`, `ordinal = page - 1`, `source_span = SourceSpan::Page { page }`.
- Streaming: read PDF in memory only once; do not load `pdf-extract` per page (that re-parses N times).
- Failure modes:
- File not a PDF / corrupt header → `anyhow::Error`.
- Encrypted PDF → `anyhow::Error` with hint to remove encryption (no decryption attempt in v1).
- Determinism: identical bytes → identical doc/block IDs and text.
## Storage / wire effects
- None directly.
## Test plan
| kind | description | fixture / data |
|------|-------------|----------------|
| unit | 3-page PDF produces 3 paragraph blocks with `SourceSpan::Page { page: 1..=3 }` | `fixtures/pdf/three-page-en.pdf` |
| unit | PDF with image-only page 2 (no text) emits warning + empty text for page 2 | `fixtures/pdf/scanned-mixed.pdf` |
| unit | encrypted PDF returns error with helpful hint | `fixtures/pdf/encrypted.pdf` |
| unit | corrupt header PDF returns error | `fixtures/pdf/corrupt.pdf` |
| unit | `metadata.user.pdf.page_count` matches actual count | inline |
| unit | Korean text PDF preserved (CID mapping permitting) | `fixtures/pdf/korean.pdf` |
| determinism | identical bytes → identical CanonicalDocument JSON across two runs | inline |
| snapshot | CanonicalDocument JSON for fixture stable | `fixtures/pdf/three-page-en.pdf` |
All tests under `cargo test -p kb-parse-pdf`.
## Definition of Done
- [ ] `cargo check -p kb-parse-pdf` passes
- [ ] `cargo test -p kb-parse-pdf` passes
- [ ] No OCR / LLM code present
- [ ] No imports outside Allowed dependencies
- [ ] PR links design §3.4 SourceSpan::Page, §9.2
## Out of scope
- OCR for scanned PDFs (separate future task; reuses p6-2 OCR adapter).
- Layout reconstruction (multi-column reading order, tables).
- Math rendering / formula detection.
- Form-field extraction.
- Bookmark / outline ingestion (could become heading_path later — note for P+).
## Risks / notes
- `pdf-extract` text quality varies wildly. For broken-glyph PDFs, the text may be unicode noise; downstream embedding still works but quality is poor. Mark such pages with a confidence-style warning when feasible.
- Some PDFs have layered text (selectable text + scanned image overlay). v1 captures the selectable text only.
- For very large PDFs (> 1k pages), memory usage may spike. Document a soft limit (`config.pdf.max_pages` default 5000) and refuse beyond it.

114
tasks/p7/p7-2-pdf-page-chunker.md Normal file
View File

@@ -0,0 +1,114 @@
---
phase: P7
component: kb-chunk (pdf-page-v1)
task_id: p7-2
title: "PDF page-aware chunker (pdf-page-v1)"
status: planned
depends_on: [p7-1]
unblocks: []
contract_source: ../../docs/superpowers/specs/2026-04-27-kb-final-form-design.md
contract_sections: [§3.5 Chunk, §4.2 chunk_id recipe, §0 Q3 citation, §9 versioning]
---
# p7-2 — PDF page chunker
## Goal
Implement `Chunker` with `chunker_version = "pdf-page-v1"`. Honors page boundaries (no chunk crosses a page) and subdivides long pages by paragraph budget. Produces the same `Chunk` shape as `md-heading-v1` so retrieval is uniform.
## Why now / why this size
Per-medium chunkers must stay tiny and obvious. Page-aware logic is small but its `chunker_version` label is load-bearing for downstream embedding records.
## Allowed dependencies
- `kb-core`
- `kb-config`
- `serde`, `serde_json`
- `blake3` (policy_hash)
- `serde-json-canonicalizer`
- `thiserror`
## Forbidden dependencies
- `kb-source-fs`, `kb-parse-md`, `kb-parse-pdf` (consumes `CanonicalDocument` via `kb-core` only), `kb-normalize`, `kb-store-*`, `kb-embed*`, `kb-search`, `kb-llm*`, `kb-rag`, `kb-tui`, `kb-desktop`
## Inputs
| input | type | source |
|-------|------|--------|
| `CanonicalDocument` (produced by `pdf-text-v1`) | `kb_core::CanonicalDocument` | p7-1 |
| `ChunkPolicy` | `kb_core::ChunkPolicy` | `kb-app` |
## Outputs
| output | type | downstream |
|--------|------|------------|
| `Vec<Chunk>` | `kb_core::Chunk` | `kb-store-sqlite`, `kb-embed*` |
## Public surface (signatures only — no new types)
```rust
pub struct PdfPageV1Chunker;
impl kb_core::Chunker for PdfPageV1Chunker {
fn chunker_version(&self) -> kb_core::ChunkerVersion { kb_core::ChunkerVersion("pdf-page-v1".into()) }
fn policy_hash(&self, policy: &kb_core::ChunkPolicy) -> String;
fn chunk(&self, doc: &kb_core::CanonicalDocument, policy: &kb_core::ChunkPolicy) -> anyhow::Result<Vec<kb_core::Chunk>>;
}
```
`policy_hash` = `blake3(canonical_json(policy))` truncated to 16 hex chars.
## Behavior contract
- Only operates on documents whose blocks all carry `SourceSpan::Page` (i.e., from `kb-parse-pdf`). Other documents → return `anyhow::Error("PdfPageV1Chunker only handles PDF docs")`.
- For each page block (1 block per page after p7-1):
- If `text.len()` (byte estimate) ≤ `policy.target_tokens * 4` (proxy for tokens) → emit one chunk for the entire page.
- Else → split by paragraphs (split text on `\n\n` or sentence-ending punctuation followed by whitespace) and group adjacent paragraphs until the running byte total approaches `policy.target_tokens * 4`. Apply `policy.overlap_tokens * 4` bytes of trailing overlap into the next chunk's prefix.
- A chunk NEVER crosses a page boundary.
- Each chunk's `source_spans` contains exactly one `SourceSpan::Page { page: i, char_start: Some(start), char_end: Some(end) }` with `start`/`end` in characters within the page.
- `heading_path = []` (PDFs have no heading tree at v1).
- `block_ids = [page_block.block_id]` (one block per chunk).
- `text` = the chunk's slice of page text. If overlap is applied, the slice includes the overlap prefix from the previous chunk.
- `token_estimate = byte_len / 4` (matches `md-heading-v1` proxy).
- `chunk_id` per design §4.2 with `(doc_id, "pdf-page-v1", block_ids, policy_hash)`.
- Determinism: identical inputs + identical policy → identical chunk IDs and text slices.
## Storage / wire effects
- None.
## Test plan
| kind | description | fixture / data |
|------|-------------|----------------|
| unit | 3-page PDF where each page < target_tokens → 3 chunks, 1 per page | seeded `CanonicalDocument` |
| unit | 1-page PDF whose text >> target_tokens → multiple chunks all on page 1 with overlap honored | seeded |
| unit | chunk crossing page boundary never produced | property test (10 random docs) |
| unit | empty page block → 0 chunks for that page (skipped) | inline |
| unit | non-PDF doc returns error | inline (Markdown-style doc) |
| determinism | same input → same chunk_ids twice | inline |
| snapshot | `Vec<Chunk>` JSON for fixture stable | `fixtures/pdf/three-page-en.pdf` (chunked) |
All tests under `cargo test -p kb-chunk pdf`.
## Definition of Done
- [ ] `cargo check -p kb-chunk` passes (existing `md-heading-v1` continues to pass)
- [ ] `cargo test -p kb-chunk pdf` passes
- [ ] Snapshot stable across two runs
- [ ] No imports outside Allowed dependencies
- [ ] PR links design §3.5, §0 Q3, §9
## Out of scope
- Token-accurate splitting (real tokenizer integration is P+).
- Cross-page sentence merging (kept off; page citation simplicity wins).
- Section/heading inference from font metadata (P+).
## Risks / notes
- Byte-based proxy can over- or under-estimate. The chunker is intentionally crude; a proper tokenizer slot lives in P3+ and replaces this proxy across all chunkers in one PR.
- Sentence-splitting uses simple regex; languages without clear sentence punctuation (e.g., Japanese) may produce uneven chunks. Document this and accept for v1.
- Bumping `chunker_version` to `pdf-page-v2` invalidates downstream embedding records for all PDFs; treat as a versioning event per §9.