altair823-org/kebab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 40 Wiki Activity

feat(p1-5): kb-chunk md-heading-v1 chunker #10

Merged
altair823 merged 6 commits from feat/p1-5-chunk into main 2026-04-30 16:58:35 +00:00
Conversation 1 Commits 6 Files Changed 8 +1353
altair823 commented 2026-04-30 16:55:54 +00:00
Owner
Copy Link

요약

P1-5 task spec을 구현했습니다. 신규 crate kb-chunk — md-heading-v1 라벨의 첫 번째 구체 Chunker 구현체. kb_core::CanonicalDocument를 받아 우선순위 기반(heading 경계 우선 / code 및 table atomic / 단락 + overlap 분할)으로 Vec<kb_core::Chunk>를 산출합니다. chunk_id는 design §4.2 recipe (doc_id, chunker_version, block_ids, policy_hash)에 따라 결정적이며, policy_hash는 blake3(canonical_json(ChunkPolicy))의 hex 16자입니다.

기준 문서: tasks/p1/p1-5-chunk.md, docs/superpowers/specs/2026-04-27-kb-final-form-design.md (§3.5 Chunk / §4.2 chunk_id / §7.2 Chunker / §0 Q3 / §14).

구현

crates/kb-chunk:

  • lib.rs — MdHeadingV1Chunker만 노출.
  • md_heading_v1.rs — Chunker trait 구현. 우선순위 logic, 토큰 추정 proxy(bytes / 3, 한국어 안전한 over-estimation), policy_hash, overlap seeding 포함.
  • tests/long_section_snapshot.rs — long-section.md 픽스처 → Vec<Chunk> JSON 결정적 round-trip + 5-iter 결정성.
  • fixtures/markdown/long-section.md (~3KB, 3 H1 / 1 H2 / 코드블록 / 표 / ImageRef 혼합) + long-section.chunks.snapshot.json baseline.

동작 contract 핵심

  • 우선순위: heading boundary 우선 → code/table atomic → ImageRef·AudioRef는 자체 chunk(token_estimate=0) → 긴 단락은 overlap_tokens만큼 prior tail을 seed로 carry → chunker_version/policy_hash 기록.
  • heading_path 전파 시 첫 블록이 Block::Heading이면 parent_path + [heading.text]로 합성하여 heading-only chunk가 인용 컨텍스트를 잃지 않도록 보장(I2 수정).
  • Overlap seed budget을 min(overlap_tokens, target_tokens / 2)로 clamp하여 병리적 정책(overlap >= target)에서 1-block-per-chunk 패턴을 차단(I3 수정).
  • policy_hash ID에만 fold됨 — design §3.5의 Chunk struct에는 policy_hash 필드가 없는 게 정합. P1-6 SQLite chunks.policy_hash 컬럼은 caller가 Chunker::policy_hash(policy)로 재계산하여 채울 책임.
  • MdHeadingV1Chunker는 zero-sized 무상태 — trait method 호출 간 캐싱 없음.

검증

  • cargo check / build (RUSTFLAGS=\"-D warnings\") / clippy --all-targets -D warnings 모두 clean
  • cargo test -p kb-chunk → 11 unit + 2 integration 통과
  • cargo test --workspace → 워크스페이스 전체 그린
  • cargo tree -p kb-chunk --depth 1 → 허용 deps만 (kb-core, anyhow, serde_json_canonicalizer, blake3, tracing). kb-parse-md / kb-normalize는 [dev-dependencies]에만 등장(§8 boundary 준수).
  • 결정성: deterministic_chunk_ids_1000 1000회 byte-identical chunk_id, long_section_chunks_are_deterministic 5-iter 통합 결정성, snapshot baseline 안정.

Review trail

  • spec compliance: PASS — must-fix 0건. cluster 6 (Chunk struct에 policy_hash 필드 부재)는 design §3.5가 권위적이라 구현자 해석 정확.
  • code quality round 1: APPROVED-WITH-MINOR — Important 3건 (I1 respect_markdown_headings 무시 / I2 heading-only chunk path 손실 / I3 overlap seed double-count) + Minor 2건 (M3 policy_hash panic 문서화 / M4 would_exceed 회계 주석).
  • 반영 3 commit (f780c71, a81460f, ceeac9f) — heading_path 합성 로직 + overlap clamp + 문서화. Snapshot baseline 무변화.
  • code quality round 2 (re-review): APPROVED — must-fix-now 0건, 새 회귀 없음.

후속 자연 해소 (비차단)

  • P1-6 SQLite put_chunks(...) 시그니처에 policy_hash 또는 policy: &ChunkPolicy 인자 추가 필요(§5.5 컬럼이 NOT NULL). P1-6 책임.
  • pedantic clippy lint(doc_markdown 백틱, missing_errors_doc 등)는 별도 hardening 패스에서.

다음

머지 후 feat/p1-6-store-sqlite 분기 → P1-6 (kb-store-sqlite) 진행. P1 마지막 task.

## 요약 P1-5 task spec을 구현했습니다. 신규 crate `kb-chunk` — `md-heading-v1` 라벨의 첫 번째 구체 `Chunker` 구현체. `kb_core::CanonicalDocument`를 받아 우선순위 기반(heading 경계 우선 / code 및 table atomic / 단락 + overlap 분할)으로 `Vec<kb_core::Chunk>`를 산출합니다. `chunk_id`는 design §4.2 recipe (`doc_id, chunker_version, block_ids, policy_hash`)에 따라 결정적이며, `policy_hash`는 `blake3(canonical_json(ChunkPolicy))`의 hex 16자입니다. 기준 문서: tasks/p1/p1-5-chunk.md, docs/superpowers/specs/2026-04-27-kb-final-form-design.md (§3.5 Chunk / §4.2 chunk_id / §7.2 Chunker / §0 Q3 / §14). ## 구현 `crates/kb-chunk`: - `lib.rs` — `MdHeadingV1Chunker`만 노출. - `md_heading_v1.rs` — `Chunker` trait 구현. 우선순위 logic, 토큰 추정 proxy(`bytes / 3`, 한국어 안전한 over-estimation), policy_hash, overlap seeding 포함. - `tests/long_section_snapshot.rs` — `long-section.md` 픽스처 → `Vec<Chunk>` JSON 결정적 round-trip + 5-iter 결정성. - `fixtures/markdown/long-section.md` (~3KB, 3 H1 / 1 H2 / 코드블록 / 표 / ImageRef 혼합) + `long-section.chunks.snapshot.json` baseline. ### 동작 contract 핵심 - 우선순위: heading boundary 우선 → code/table atomic → ImageRef·AudioRef는 자체 chunk(`token_estimate=0`) → 긴 단락은 `overlap_tokens`만큼 prior tail을 seed로 carry → `chunker_version`/`policy_hash` 기록. - `heading_path` 전파 시 첫 블록이 `Block::Heading`이면 `parent_path + [heading.text]`로 합성하여 heading-only chunk가 인용 컨텍스트를 잃지 않도록 보장(I2 수정). - Overlap seed budget을 `min(overlap_tokens, target_tokens / 2)`로 clamp하여 병리적 정책(`overlap >= target`)에서 1-block-per-chunk 패턴을 차단(I3 수정). - `policy_hash` ID에만 fold됨 — design §3.5의 `Chunk` struct에는 `policy_hash` 필드가 없는 게 정합. P1-6 SQLite `chunks.policy_hash` 컬럼은 caller가 `Chunker::policy_hash(policy)`로 재계산하여 채울 책임. - `MdHeadingV1Chunker`는 zero-sized 무상태 — trait method 호출 간 캐싱 없음. ## 검증 - `cargo check / build (RUSTFLAGS=\"-D warnings\") / clippy --all-targets -D warnings` 모두 clean - `cargo test -p kb-chunk` → 11 unit + 2 integration 통과 - `cargo test --workspace` → 워크스페이스 전체 그린 - `cargo tree -p kb-chunk --depth 1` → 허용 deps만 (kb-core, anyhow, serde_json_canonicalizer, blake3, tracing). `kb-parse-md` / `kb-normalize`는 `[dev-dependencies]`에만 등장(§8 boundary 준수). - 결정성: `deterministic_chunk_ids_1000` 1000회 byte-identical chunk_id, `long_section_chunks_are_deterministic` 5-iter 통합 결정성, snapshot baseline 안정. ## Review trail - spec compliance: PASS — must-fix 0건. cluster 6 (`Chunk` struct에 `policy_hash` 필드 부재)는 design §3.5가 권위적이라 구현자 해석 정확. - code quality round 1: APPROVED-WITH-MINOR — Important 3건 (I1 `respect_markdown_headings` 무시 / I2 heading-only chunk path 손실 / I3 overlap seed double-count) + Minor 2건 (M3 `policy_hash` panic 문서화 / M4 `would_exceed` 회계 주석). - 반영 3 commit (`f780c71`, `a81460f`, `ceeac9f`) — heading_path 합성 로직 + overlap clamp + 문서화. Snapshot baseline 무변화. - code quality round 2 (re-review): **APPROVED** — must-fix-now 0건, 새 회귀 없음. ## 후속 자연 해소 (비차단) - P1-6 SQLite `put_chunks(...)` 시그니처에 `policy_hash` 또는 `policy: &ChunkPolicy` 인자 추가 필요(§5.5 컬럼이 `NOT NULL`). P1-6 책임. - pedantic clippy lint(`doc_markdown` 백틱, `missing_errors_doc` 등)는 별도 hardening 패스에서. ## 다음 머지 후 `feat/p1-6-store-sqlite` 분기 → P1-6 (kb-store-sqlite) 진행. P1 마지막 task.
altair823 added 6 commits 2026-04-30 16:55:55 +00:00
p1-5: scaffold kb-chunk crate with MdHeadingV1Chunker skeleton 8142449eb7 
Adds the new workspace member with the bare Chunker impl shape:
chunker_version() returns "md-heading-v1"; policy_hash() blake3-hashes
canonical JSON of ChunkPolicy and truncates to 16 hex chars; chunk()
is an empty stub the next commits fill in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
p1-5: implement md-heading-v1 chunking rules 0237022d0e 
Fills in MdHeadingV1Chunker::chunk with the priority-ordered ruleset
from the design (§0 / §14):

  1. Heading is a hard boundary; the heading itself starts and is
     included in its chunk so heading text is retrievable.
  2. Code blocks never split, regardless of size.
  3. Tables stay single-chunk (row-split deferred per task spec).
  4. Long sections split at target_tokens with paragraph-level
     overlap_tokens worth of seeded tail blocks.
  5. ImageRef / AudioRef each become their own chunk with
     token_estimate = 0.
  6. heading_path lifts from the first contributing non-Heading
     block; source_spans concatenate in document order.
  7. chunk_id derives from id_for_chunk(doc_id, chunker_version,
     block_ids, policy_hash) per §4.2.

Covers the unit + determinism rows of the P1-5 test plan: heading
boundary respected, 800-token code block stays single, small table
stays single, long paragraph chain splits with overlap, ImageRef
chunk has token_estimate=0, and 1000-iter chunk_id determinism.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
p1-5: add long-section fixture + Vec<Chunk> snapshot test 58f7b8573d 
Bakes the chunker output for fixtures/markdown/long-section.md (3 H1s,
nested H2 under Alpha, a 50-line code block, a 3-col x 4-row table,
and a multi-paragraph Gamma section) into the JSON snapshot baseline.
Confirms the priority rules end-to-end:

  - Heading boundaries hold across H1 → H2 → H1 transitions
  - The code block emits one chunk at 427 tokens > target=200
  - The table stays single-chunk
  - Gamma's paragraph stream splits with one block of overlap seed

A second test runs the full parse → normalize → chunk pipeline 5
times and asserts identical chunk_ids each pass.

Drops the unused `kb-config` and `serde` from regular dependencies —
they were declared but no source path imports them; `serde` flows in
transitively via `kb-core` as a public API requirement, and
`ChunkingCfg` lives in `kb-config` but the chunker takes
`ChunkPolicy` directly. Production deps are now exactly the allowed
set actually used: anyhow, blake3, kb-core, serde_json_canonicalizer,
tracing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
p1-5: heading-only chunk carries self in heading_path f780c71ce0 
When a Heading block opens a chunk and is followed by another Heading
or an atomic block (Code, Table, ImageRef, AudioRef) with no
intervening prose, the prior fallback used `common.heading_path` from
the heading itself — which per kb-normalize convention does NOT
include the heading's own text. Result: heading-only and heading-led
chunks for `# Alpha\n## Beta\n...` patterns landed with
`heading_path = []`, losing citation context.

Synthesize the leading heading into the chunk's heading_path when
blocks[0] is a Heading: parent path + heading.text. The first
non-Heading branch (existing logic for normal mid-section chunks) is
unchanged.

`chunk_id` recipe is `(doc_id, chunker_version, block_ids,
policy_hash)` — `heading_path` is not in the recipe, so this fix does
NOT shift chunk_ids. Snapshot baseline `long-section.chunks.snapshot.json`
also unchanged because every heading in that fixture is followed by a
paragraph (the bug only triggers on direct heading→heading or
heading→atomic adjacency).

Adds `heading_with_parents` test helper and a regression test pinning
the `# Alpha\n## Beta\n[code]` pattern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
p1-5: clamp overlap seed budget to target_tokens / 2 a81460f9d0 
A pathological `ChunkPolicy { overlap_tokens >= target_tokens }`
caused `md-heading-v1` to degenerate into 1-block-per-chunk: the
seeded `acc.text_tokens` already exceeded `target_tokens` before any
fresh content landed, so the next paragraph immediately tripped the
`would_exceed` flush.

Clamp the seed budget in `collect_overlap_seed` to
`min(overlap_tokens, target_tokens / 2)`. This guarantees that after
seeding, the chunk has at least `target/2` worth of room for new
content before flushing, restoring the intended paragraph-overlap
behavior on every reasonable and unreasonable policy.

Adds a regression test pinning a 50/200 (overlap = 4× target) policy
and asserting every emitted chunk holds ≥2 blocks.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
p1-5: doc rationale for respect_markdown_headings, policy_hash panic, overlap accounting ceeac9f974 
Doc-only follow-ups for review minors I1, M3, M4. No behavior change.

* I1: rustdoc on `MdHeadingV1Chunker` now records that
  `policy.respect_markdown_headings` flows into `policy_hash` only;
  the `md-heading-v1` variant unconditionally treats headings as
  boundaries by name. To disable heading awareness, ship a different
  `chunker_version` (none in P1-5).

* M3: `# Panics` rustdoc on `policy_hash` documents the
  unreachable-in-practice failure mode of
  `serde_json_canonicalizer::to_vec` and explains why the `expect` is
  retained as future-proofing.

* M4: Inline comment at the `would_exceed` decision noting that
  `acc.text_tokens` already includes the prior chunk's overlap seed,
  and that the I3 clamp guarantees a flush here never produces a
  chunk smaller than the seed budget.

* Heading-path bullet in the behavior contract updated to reflect the
  I2 fix wording.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-reviewer-01 reviewed 2026-04-30 16:56:18 +00:00
claude-reviewer-01 left a comment
Member
Copy Link

리뷰: 모든 항목 해소 — APPROVE 권고 (gate 정책상 COMMENT 등록)

내부 review loop를 3회 완료했습니다. 마지막 리뷰어 verdict: APPROVED, must-fix-now 0건.

Review trail

Round Verdict 반영
Spec compliance PASS — must-fix 0 —
Quality round 1 APPROVED-WITH-MINOR (Important 3건 + Minor 2건) f780c71, a81460f, ceeac9f
Quality round 2 APPROVED — 새 회귀 0 —

핵심 검증

항목 상태
Chunker trait chunker_version="md-heading-v1" + policy_hash 16hex + chunk 결정성 ✅
우선순위 1-7 (heading boundary / code-table atomic / paragraph split + overlap / heading_path / merged spans / ImageRef·AudioRef 자체 chunk + token_estimate=0) ✅ 모두 테스트로 핀
chunk_id recipe per §4.2 (block_ids는 document order, 순서 유의성 검증) ✅
Heading-only chunk가 self-text를 path에 포함 (I2) ✅ # A\n## B\n[code] → [A], [A,B], [A,B]
Overlap seed budget clamp min(overlap_tokens, target_tokens / 2) (I3) ✅ 병리적 정책에서 1-block-per-chunk 패턴 차단
결정성 1000 iter < 1초 ✅ ~104ms
Chunk struct에 policy_hash 필드 부재 → ID에만 fold (design §3.5 정합) ✅

Gate 통과

  • cargo check / build (RUSTFLAGS=-D warnings) / clippy --all-targets -D warnings clean
  • cargo test -p kb-chunk → 11 unit + 2 integration pass (was 9+2, +2 신규: heading_only_chunk_carries_self_in_path, overlap_clamped_when_overlap_exceeds_target)
  • cargo test --workspace 전체 그린
  • Snapshot baseline long-section.chunks.snapshot.json 무변화 — I2 fix는 chunk_id 영향 없음(heading_path는 recipe에 미포함), I3 clamp는 정상 정책에서 무영향(min(40, 100)=40).

후속 자연 해소 (비차단)

  • P1-6에서 §5.5 SQLite chunks.policy_hash NOT NULL 컬럼을 채우려면 put_chunks 시그니처 보정 필요. caller가 Chunker::policy_hash(policy)로 재계산하여 전달하는 방식이 자연스럽습니다.
  • pedantic clippy hardening (M1, M2, M5, M6, M7)은 추후 별도 패스.

결론

papered over 없이 root-cause fix 완료. self-review gate 정책상 본 코멘트는 COMMENT — author 측 수동 APPROVE + merge 부탁드립니다.

## 리뷰: 모든 항목 해소 — APPROVE 권고 (gate 정책상 COMMENT 등록) 내부 review loop를 3회 완료했습니다. 마지막 리뷰어 verdict: **APPROVED**, must-fix-now 0건. ### Review trail | Round | Verdict | 반영 | |-------|---------|------| | Spec compliance | PASS — must-fix 0 | — | | Quality round 1 | APPROVED-WITH-MINOR (Important 3건 + Minor 2건) | `f780c71`, `a81460f`, `ceeac9f` | | Quality round 2 | **APPROVED** — 새 회귀 0 | — | ### 핵심 검증 | 항목 | 상태 | |------|------| | Chunker trait `chunker_version="md-heading-v1"` + `policy_hash` 16hex + `chunk` 결정성 | ✅ | | 우선순위 1-7 (heading boundary / code-table atomic / paragraph split + overlap / heading_path / merged spans / ImageRef·AudioRef 자체 chunk + token_estimate=0) | ✅ 모두 테스트로 핀 | | `chunk_id` recipe per §4.2 (block_ids는 document order, 순서 유의성 검증) | ✅ | | Heading-only chunk가 self-text를 path에 포함 (I2) | ✅ `# A\n## B\n[code]` → `[A]`, `[A,B]`, `[A,B]` | | Overlap seed budget clamp `min(overlap_tokens, target_tokens / 2)` (I3) | ✅ 병리적 정책에서 1-block-per-chunk 패턴 차단 | | 결정성 1000 iter < 1초 | ✅ ~104ms | | `Chunk` struct에 `policy_hash` 필드 부재 → ID에만 fold (design §3.5 정합) | ✅ | ### Gate 통과 - `cargo check / build (RUSTFLAGS=-D warnings) / clippy --all-targets -D warnings` clean - `cargo test -p kb-chunk` → 11 unit + 2 integration pass (was 9+2, +2 신규: `heading_only_chunk_carries_self_in_path`, `overlap_clamped_when_overlap_exceeds_target`) - `cargo test --workspace` 전체 그린 - Snapshot baseline `long-section.chunks.snapshot.json` 무변화 — I2 fix는 `chunk_id` 영향 없음(heading_path는 recipe에 미포함), I3 clamp는 정상 정책에서 무영향(`min(40, 100)=40`). ### 후속 자연 해소 (비차단) - P1-6에서 §5.5 SQLite `chunks.policy_hash NOT NULL` 컬럼을 채우려면 `put_chunks` 시그니처 보정 필요. caller가 `Chunker::policy_hash(policy)`로 재계산하여 전달하는 방식이 자연스럽습니다. - pedantic clippy hardening (M1, M2, M5, M6, M7)은 추후 별도 패스. ### 결론 papered over 없이 root-cause fix 완료. self-review gate 정책상 본 코멘트는 COMMENT — author 측 수동 APPROVE + merge 부탁드립니다.
altair823 merged commit b46e69b9c0 into main 2026-04-30 16:58:35 +00:00
altair823 deleted branch feat/p1-5-chunk 2026-04-30 16:58:36 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
claude-reviewer-01
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
altair823 claude-reviewer-01
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: altair823-org/kebab#10
Delete Branch "feat/p1-5-chunk"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?