tasks: add p1-5 chunk component spec
This commit is contained in:
kb
113
tasks/p1/p1-5-chunk.md
Normal file
113
tasks/p1/p1-5-chunk.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
---
|
||||||
|
phase: P1
|
||||||
|
component: kb-chunk
|
||||||
|
task_id: p1-5
|
||||||
|
title: "Markdown heading-aware chunker (md-heading-v1)"
|
||||||
|
status: planned
|
||||||
|
depends_on: [p1-4]
|
||||||
|
unblocks: [p1-6, p2-2, p3-2]
|
||||||
|
contract_source: ../../docs/superpowers/specs/2026-04-27-kb-final-form-design.md
|
||||||
|
contract_sections: [§3.5 Chunk, §4.2 chunk_id recipe, §7.2 Chunker, §0 Q3 citation]
|
||||||
|
---
|
||||||
|
|
||||||
|
# p1-5 — Markdown heading-aware chunker
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Implement `Chunker` trait emitting `chunker_version = "md-heading-v1"`. Block-aware: respect heading boundaries, never split code/table, propagate `heading_path` and merged `source_spans`.
|
||||||
|
|
||||||
|
## Why now / why this size
|
||||||
|
|
||||||
|
The first concrete `Chunker`. Establishes how subsequent chunkers (PDF page chunker, audio segment chunker) are scoped: per-medium chunker version label. Independent of any store/embed.
|
||||||
|
|
||||||
|
## Allowed dependencies
|
||||||
|
|
||||||
|
- `kb-core`
|
||||||
|
- `kb-config`
|
||||||
|
- `serde`
|
||||||
|
- `blake3` (policy_hash)
|
||||||
|
- `serde-json-canonicalizer`
|
||||||
|
- `thiserror`
|
||||||
|
|
||||||
|
## Forbidden dependencies
|
||||||
|
|
||||||
|
- `kb-source-fs`, `kb-parse-md`, `kb-normalize` (consumes `CanonicalDocument` only via `kb-core`), `kb-store-*`, `kb-embed*`, `kb-search`, `kb-llm*`, `kb-rag`, `kb-tui`, `kb-desktop`
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
| input | type | source |
|
||||||
|
|-------|------|--------|
|
||||||
|
| `CanonicalDocument` | `kb_core::CanonicalDocument` | p1-4 |
|
||||||
|
| `ChunkPolicy` | `kb_core::ChunkPolicy` | `kb-app` from config |
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
| output | type | downstream |
|
||||||
|
|--------|------|------------|
|
||||||
|
| `Vec<Chunk>` | `kb_core::Chunk` | `kb-store-sqlite` (p1-6), `kb-embed*` (P3) |
|
||||||
|
|
||||||
|
## Public surface (signatures only — no new types)
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct MdHeadingV1Chunker;
|
||||||
|
|
||||||
|
impl kb_core::Chunker for MdHeadingV1Chunker {
|
||||||
|
fn chunker_version(&self) -> kb_core::ChunkerVersion;
|
||||||
|
fn policy_hash(&self, policy: &kb_core::ChunkPolicy) -> String;
|
||||||
|
fn chunk(&self, doc: &kb_core::CanonicalDocument, policy: &kb_core::ChunkPolicy) -> anyhow::Result<Vec<kb_core::Chunk>>;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`policy_hash` = `blake3(canonical_json(policy))` hex truncated to 16 chars.
|
||||||
|
|
||||||
|
## Behavior contract
|
||||||
|
|
||||||
|
- Priority order (per design §0 / report §14):
|
||||||
|
1. heading boundary first
|
||||||
|
2. never split a code block
|
||||||
|
3. table stays in a single chunk if possible
|
||||||
|
4. long sections split by paragraph
|
||||||
|
5. propagate `heading_path` from blocks
|
||||||
|
6. carry merged `source_spans` (each chunk lists every contributing block's span)
|
||||||
|
7. record `chunker_version = "md-heading-v1"` and `policy_hash`
|
||||||
|
- `target_tokens` and `overlap_tokens` from `ChunkPolicy`. Token estimate is byte-based proxy until a real tokenizer is introduced (note in `Chunk.token_estimate`).
|
||||||
|
- `chunk_id` per design §4.2: tagged tuple of `(doc_id, chunker_version, block_ids, policy_hash)`.
|
||||||
|
- `block_ids` listed in document order (significant — affects ID).
|
||||||
|
- ImageRef / AudioRef blocks are emitted as their own chunks (text portion = alt + caption preview if present, else empty string with `token_estimate=0`). They still receive `chunk_id` so future image/audio search can locate them.
|
||||||
|
|
||||||
|
## Storage / wire effects
|
||||||
|
|
||||||
|
- None directly. Outputs feed p1-6.
|
||||||
|
|
||||||
|
## Test plan
|
||||||
|
|
||||||
|
| kind | description | fixture / data |
|
||||||
|
|------|-------------|----------------|
|
||||||
|
| unit | heading boundary respected (no chunk crosses H2 → H2) | inline |
|
||||||
|
| unit | code block of 800 tokens stays in one chunk even when target=500 | inline |
|
||||||
|
| unit | table block stays single chunk if size < 2× target | inline |
|
||||||
|
| unit | long paragraph split with overlap_tokens applied | inline |
|
||||||
|
| unit | ImageRefBlock produces a chunk with token_estimate=0 | inline |
|
||||||
|
| determinism | identical input + identical policy → identical chunk_ids | inline |
|
||||||
|
| snapshot | `fixtures/markdown/long-section.md` → Vec<Chunk> JSON stable | fixture |
|
||||||
|
|
||||||
|
All tests under `cargo test -p kb-chunk`.
|
||||||
|
|
||||||
|
## Definition of Done
|
||||||
|
|
||||||
|
- [ ] `cargo check -p kb-chunk` passes
|
||||||
|
- [ ] `cargo test -p kb-chunk` passes
|
||||||
|
- [ ] Snapshot stable across two runs
|
||||||
|
- [ ] No imports outside Allowed dependencies
|
||||||
|
- [ ] PR links design §3.5, §4.2
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- DB persistence (p1-6).
|
||||||
|
- Embedding (P3).
|
||||||
|
- Reranking / hybrid (P3).
|
||||||
|
|
||||||
|
## Risks / notes
|
||||||
|
|
||||||
|
- Token estimate proxy: a real tokenizer (e.g., sentencepiece for the embedding model) replaces this in P3. The proxy must err toward overestimation so chunks fit in real tokenizer budget.
|
||||||
|
- Changing `chunker_version` invalidates all downstream embedding records. Bump only with PR documenting the migration plan (design §9).
|
||||||
Reference in New Issue
Block a user