feat(rag): fb-41 PR-3b-ii — ScriptedLm + 5 multi-hop tests + refusal hop trace + carry-over
PR-3b 의 분할 두 번째 PR — PR-3b-i 의 dynamic decide loop 위에서:
1. **ScriptedLm + ScriptedRetriever helper** (kebab-rag tests/common/mod.rs)
per-call 다른 response 반환. decompose / decide×N / synthesize 의 각
LLM call 을 구분하는 다단계 multi-hop 시나리오를 mock-only 로 exercise
가능. `Vec<&str>` / `Vec<Vec<SearchHit>>` 받아 call sequence 순서대로
emit. Send + Sync.
2. **5 multi-hop integration tests** (kebab-rag tests/multi_hop.rs 신규)
- decide_stop_triggers_synthesize: decide [] → 즉시 synthesize
- decide_continue_adds_more_chunks: decide ["q2"] → iter 2 retrieve + pool 확장
- max_depth_force_stops: depth cap → forced_stop + decide LLM call skip
- pool_chunks_dedup_by_chunk_id: 같은 chunk_id 두 sub-query 에서 1 회
- decide_parse_failure_falls_through_to_synthesize: parse fail = graceful
synthesize (refusal 아님, spec §9)
3. **refuse_* helper hops trace 보존** (회차 1 carry-over)
refuse_no_chunks / refuse_score_gate 시그니처에 `hops:
Option<Vec<HopRecord>>` 인자 추가. ask_multi_hop 의 score-gate /
no-chunks refusal 시 누적된 hops 그대로 Answer.hops 에 보존.
single-pass ask 는 None 전달 — wire 변동 없음 (skip_serializing_if).
4. **HopRecord doc 보강** (회차 1 carry-over)
sub_queries 의 per-kind 의미 명시 (Decompose=initial / Decide=next-iter
or empty=stop / Synthesize=always empty). llm_call_ms=0 의 ambiguity
(no call vs 0ms call) doc 명시.
5. **MULTI_HOP_MAX_SUB_QUERIES_DEFAULT → _HARD_CAP rename** (회차 1 carry-over)
const 의 의도 명확화 — config knob `multi_hop_max_sub_queries_per_iter`
(5, prompt-side soft hint) 와 const (10, parse-side hard ceiling)
분리. 두 layer 의 책임 doc 동기화. test 도 rename.
6. **decide guard 단순화 + preview budget doc** (회차 1 carry-over)
parse_decompose_response 의 post-condition (Some=non-empty 보장)
doc 명시. defensive `Some(qs) if !qs.is_empty()` →
`decide_result.unwrap_or_default()` 단순화. decide preview 의
snippet-only path (full chunk text 안 fetch) 의도 doc.
검증
- `cargo test -p kebab-rag -j 1` — 31 unit + 19 pipeline + 5 multi_hop
+ 3 prompt_template + 3 streaming 모두 통과.
- `cargo clippy -p kebab-rag --all-targets -j 1 -- -D warnings` clean.
Spec / plan
- design: docs/superpowers/specs/2026-05-25-p9-fb-41-multi-hop-rag-design.md
- plan: docs/superpowers/plans/2026-05-25-p9-fb-41-multi-hop-rag.md (PR-3b 단락)
다음 단계 = PR-4 (CLI --multi-hop + wire schema + error_wire).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -79,8 +79,18 @@ pub struct HopRecord {
|
||||
/// the final iter is the synthesize call.
|
||||
pub iter: u32,
|
||||
pub kind: HopKind,
|
||||
/// Sub-queries the LLM emitted at this iter. For the synthesize
|
||||
/// hop this is empty (no sub-queries — just the final answer).
|
||||
/// Sub-queries associated with this hop. The meaning depends on
|
||||
/// `kind`:
|
||||
///
|
||||
/// - [`HopKind::Decompose`]: the initial sub-queries the LLM
|
||||
/// broke the original user query into. These drive the
|
||||
/// `iter=1` retrieval round.
|
||||
/// - [`HopKind::Decide`]: the *new* sub-queries the LLM
|
||||
/// emitted to drive the next retrieval round. Empty when the
|
||||
/// LLM signalled stop OR when `forced_stop = true` (cap hit
|
||||
/// or parse-degraded).
|
||||
/// - [`HopKind::Synthesize`]: always empty — the final hop
|
||||
/// produces the user-visible answer, not more sub-queries.
|
||||
#[serde(default)]
|
||||
pub sub_queries: Vec<String>,
|
||||
/// Number of *new* chunks the retrieval round contributed to the
|
||||
@@ -98,6 +108,13 @@ pub struct HopRecord {
|
||||
/// Wall-clock latency of the LLM call for this hop, in
|
||||
/// milliseconds. Useful for cost / latency analysis when a
|
||||
/// `kebab eval` run records `Answer.hops`.
|
||||
///
|
||||
/// `0` is overloaded: it means "no LLM call happened at this
|
||||
/// hop" when (a) the hop was a Decide skipped due to
|
||||
/// `forced_stop` (depth-cap or pool-cap fired before the LLM
|
||||
/// was asked) or (b) the pool was empty before any decide
|
||||
/// could run. Treat `0` as "absent or instantaneous" rather
|
||||
/// than as a genuine measurement.
|
||||
pub llm_call_ms: u32,
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user