kebab/tasks/p7/p7-2-pdf-page-chunker.md

phase, component, task_id, title, status, depends_on, unblocks, contract_source, contract_sections

phase	component	task_id	title	status	depends_on	unblocks	contract_source	contract_sections
P7	kebab-chunk (pdf-page-v1)	p7-2	PDF page-aware chunker (pdf-page-v1)	completed	p7-1		../../docs/superpowers/specs/2026-04-27-kebab-final-form-design.md	§3.5 Chunk §4.2 chunk_id recipe §0 Q3 citation §9 versioning

p7-2 — PDF page chunker

Goal

Implement Chunker with chunker_version = "pdf-page-v1". Honors page boundaries (no chunk crosses a page) and subdivides long pages by paragraph budget. Produces the same Chunk shape as md-heading-v1 so retrieval is uniform.

Why now / why this size

Per-medium chunkers must stay tiny and obvious. Page-aware logic is small but its chunker_version label is load-bearing for downstream embedding records.

Allowed dependencies

• kebab-core
• kebab-config
• serde, serde_json
• blake3 (policy_hash)
• serde-json-canonicalizer
• thiserror

Forbidden dependencies

• kebab-source-fs, kebab-parse-md, kebab-parse-pdf (consumes CanonicalDocument via kebab-core only), kebab-normalize, kebab-store-*, kebab-embed*, kebab-search, kebab-llm*, kebab-rag, kebab-tui, kebab-desktop

Inputs

input	type	source
CanonicalDocument (produced by pdf-text-v1)	kebab_core::CanonicalDocument	p7-1
ChunkPolicy	kebab_core::ChunkPolicy	kebab-app

Outputs

output	type	downstream
Vec<Chunk>	kebab_core::Chunk	kebab-store-sqlite, kebab-embed*

Public surface (signatures only — no new types)

pub struct PdfPageV1Chunker;

impl kebab_core::Chunker for PdfPageV1Chunker {
    fn chunker_version(&self) -> kebab_core::ChunkerVersion { kebab_core::ChunkerVersion("pdf-page-v1".into()) }
    fn policy_hash(&self, policy: &kebab_core::ChunkPolicy) -> String;
    fn chunk(&self, doc: &kebab_core::CanonicalDocument, policy: &kebab_core::ChunkPolicy) -> anyhow::Result<Vec<kebab_core::Chunk>>;
}

policy_hash = blake3(canonical_json(policy)) truncated to 16 hex chars.

Behavior contract

• Only operates on documents whose blocks all carry SourceSpan::Page (i.e., from kebab-parse-pdf). Other documents → return anyhow::Error("PdfPageV1Chunker only handles PDF docs").
• For each page block (1 block per page after p7-1):
  • If text.len() (byte estimate) ≤ policy.target_tokens * 4 (proxy for tokens) → emit one chunk for the entire page.
  • Else → split by paragraphs (split text on \n\n or sentence-ending punctuation followed by whitespace) and group adjacent paragraphs until the running byte total approaches policy.target_tokens * 4. Apply policy.overlap_tokens * 4 bytes of trailing overlap into the next chunk's prefix.
• A chunk NEVER crosses a page boundary.
• Each chunk's source_spans contains exactly one SourceSpan::Page { page: i, char_start: Some(start), char_end: Some(end) } with start/end in characters within the page.
• heading_path = [] (PDFs have no heading tree at v1).
• block_ids = [page_block.block_id] (one block per chunk).
• text = the chunk's slice of page text. If overlap is applied, the slice includes the overlap prefix from the previous chunk.
• token_estimate = byte_len / 4 (matches md-heading-v1 proxy).
• chunk_id per design §4.2 with (doc_id, "pdf-page-v1", block_ids, policy_hash).
• Determinism: identical inputs + identical policy → identical chunk IDs and text slices.

Storage / wire effects

• None.

Test plan

kind	description	fixture / data
unit	3-page PDF where each page < target_tokens → 3 chunks, 1 per page	seeded CanonicalDocument
unit	1-page PDF whose text >> target_tokens → multiple chunks all on page 1 with overlap honored	seeded
unit	chunk crossing page boundary never produced	property test (10 random docs)
unit	empty page block → 0 chunks for that page (skipped)	inline
unit	non-PDF doc returns error	inline (Markdown-style doc)
determinism	same input → same chunk_ids twice	inline
snapshot	Vec<Chunk> JSON for fixture stable	fixtures/pdf/three-page-en.pdf (chunked)

All tests under cargo test -p kebab-chunk pdf.

Definition of Done

• cargo check -p kebab-chunk passes (existing md-heading-v1 continues to pass)
• cargo test -p kebab-chunk pdf passes
• Snapshot stable across two runs
• No imports outside Allowed dependencies
• PR links design §3.5, §0 Q3, §9

Out of scope

• Token-accurate splitting (real tokenizer integration is P+).
• Cross-page sentence merging (kept off; page citation simplicity wins).
• Section/heading inference from font metadata (P+).

Risks / notes

• Byte-based proxy can over- or under-estimate. The chunker is intentionally crude; a proper tokenizer slot lives in P3+ and replaces this proxy across all chunkers in one PR.
• Sentence-splitting uses simple regex; languages without clear sentence punctuation (e.g., Japanese) may produce uneven chunks. Document this and accept for v1.
• Bumping chunker_version to pdf-page-v2 invalidates downstream embedding records for all PDFs; treat as a versioning event per §9.