altair823-org/kebab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 40 Wiki Activity

feat(nli): fb-41 PR-9a — kebab-nli crate skeleton + workspace deps #176

Merged
altair823 merged 3 commits from feat/fb-41-pr-9a-kebab-nli-crate into main 2026-05-25 21:34:51 +00:00
Conversation 2 Commits 3 Files Changed 7 +1560
altair823 commented 2026-05-25 21:23:56 +00:00
Owner
Copy Link

요약

fb-41 multi-hop RAG 의 dogfood S7 hallucination root cause (LLM-self-judge ceiling) 대응 — NLI-based post-synthesis verification (mDeBERTa-v3 XNLI) 도입. PR-9a 는 첫 sub-PR: trait surface + scaffolding + workspace dep chain. 실제 ONNX inference 는 PR-9b.

설계: docs/superpowers/specs/2026-05-25-p9-fb-41-finalize-spec.md (review_round=5, 4 OMC reviewer APPROVE)
계획: docs/superpowers/plans/2026-05-25-p9-fb-41-finalize-plan.md (plan_review_round=3, 4 OMC reviewer APPROVE)

변경 사항

신규 crate kebab-nli

  • crates/kebab-nli/src/lib.rs:
    • NliScores { entailment, neutral, contradiction } — XNLI 3-channel (id2label 표준).
    • NliScores::faithfulness() → entailment 채널 (production accept signal).
    • NliScores::from_xnli_logits(logits: [f32; 3]) — softmax3 적용.
    • pub trait NliVerifier: Send + Sync { fn score(&self, &str, &str) -> Result<NliScores>; }.
    • private softmax3 (log-sum-exp 안전).
  • crates/kebab-nli/src/onnx.rs:
    • OnnxNliVerifier placeholder (PR-9b 가 model_id, cache_dir, session: OnceLock<ort::Session>, tokenizer: OnceLock 추가).
    • new(&Config) -> Result<Self> placeholder (default config 통과).
    • NliVerifier::score → bail!("PR-9a stub — ONNX inference lands in PR-9b").

Workspace deps

Cargo.toml 의 [workspace.dependencies] 신규 (fastembed transitive 와 정확히 일치):

  • ort = "=2.0.0-rc.9" (default-features=false, ndarray)
  • tokenizers = "0.21" (default-features=false, onig — pre-flight lock)
  • hf-hub = "0.4" (default-features=false, ureq+rustls-tls)
  • ndarray = "0.16"

PR-9b 가 crates/kebab-nli/Cargo.toml 에 추가 (현재는 workspace 만 선언).

Spec + plan 동봉

PR-9 design contract 가 본 PR-9a 부터 다음 sub-PR (PR-9b/9c-1/9c-2/9d/cut PR) 모두 참조. 별 PR 분리 시 contract 의 main reference 가 지연 → 본 PR 안에 docs commit 으로 분리하여 함께 머지.

Cargo features 결정 trace

Pre-flight 1 — HF reachability (spec §2.1):

curl -I https://huggingface.co/Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7/resolve/main/onnx/model.onnx → HTTP/2 302 (HF S3 routing, file 존재)
curl -I https://huggingface.co/Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7/resolve/main/tokenizer.json → HTTP/2 302

Pre-flight 2 — tokenizers features lock (spec §3 PR-9a):

cargo new --bin /tmp/nli-tok-probe
cargo add tokenizers --no-default-features -F onig
wget tokenizer.json (16 MB)
cargo run --release
→ ok | n_tokens(ko) = 9
→ ok | n_tokens(en) = 11

Lock: tokenizers = { version = "0.21", default-features = false, features = ["onig"] }. SentencePiece mDeBERTa tokenizer.json 로드 + KR/EN encode 정상.

검증

  • cargo test -p kebab-nli -j 1 — 6/6 통과:
    • softmax3_normalises_to_unit, softmax3_is_invariant_to_constant_shift,
    • nli_scores_from_xnli_logits_orders_correctly, faithfulness_returns_entailment_channel,
    • new_succeeds_on_default_config, score_returns_err_in_skeleton.
  • cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings clean.
  • cargo test --workspace --no-fail-fast -j 1 — pre-existing kebab-mcp::tools_call_ask_multi_hop::ask_tool_routes_multi_hop_true_to_decompose_first 1 fail. main baseline 동일 fail 확인 — PR-9a 무관 (ingest fixture / Ollama 의존 flaky, HOTFIXES candidate).
  • 워크스페이스 빌드 클린 (kebab-nli 의 dep chain 도입이 18 integration test binary link 단계에 회귀 없음).

비범위

  • ONNX inference (PR-9b 의 OnnxNliVerifier::score 실 구현).
  • crates/kebab-nli/Cargo.toml 에 ort/tokenizers/hf-hub/ndarray 의존 추가 (PR-9b).
  • Pipeline integration (PR-9c-2 의 ask_multi_hop step 8.5 hook).
  • Wire schema 변경 (PR-9c-1 의 answer.v1.verification + RefusalReason::NliVerificationFailed 등).

시험 항목 (Test Plan)

  • cargo test -p kebab-nli -j 1 6/6 통과.
  • cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings clean.
  • cargo test --workspace -j 1 회귀 0 (pre-existing 1 fail 은 main baseline 동일 — PR-9a 무관).
  • Pre-flight 1 + 2 결과 첨부 (## Cargo features 결정 trace).

Assisted-by: Claude Code

## 요약 fb-41 multi-hop RAG 의 dogfood S7 hallucination root cause (LLM-self-judge ceiling) 대응 — NLI-based post-synthesis verification (mDeBERTa-v3 XNLI) 도입. **PR-9a 는 첫 sub-PR**: trait surface + scaffolding + workspace dep chain. 실제 ONNX inference 는 PR-9b. 설계: docs/superpowers/specs/2026-05-25-p9-fb-41-finalize-spec.md (review_round=5, 4 OMC reviewer APPROVE) 계획: docs/superpowers/plans/2026-05-25-p9-fb-41-finalize-plan.md (plan_review_round=3, 4 OMC reviewer APPROVE) ## 변경 사항 ### 신규 crate kebab-nli - `crates/kebab-nli/src/lib.rs`: - `NliScores { entailment, neutral, contradiction }` — XNLI 3-channel (id2label 표준). - `NliScores::faithfulness()` → entailment 채널 (production accept signal). - `NliScores::from_xnli_logits(logits: [f32; 3])` — softmax3 적용. - `pub trait NliVerifier: Send + Sync { fn score(&self, &str, &str) -> Result<NliScores>; }`. - private `softmax3` (log-sum-exp 안전). - `crates/kebab-nli/src/onnx.rs`: - `OnnxNliVerifier` placeholder (PR-9b 가 `model_id`, `cache_dir`, `session: OnceLock<ort::Session>`, `tokenizer: OnceLock` 추가). - `new(&Config) -> Result<Self>` placeholder (default config 통과). - `NliVerifier::score → bail!("PR-9a stub — ONNX inference lands in PR-9b")`. ### Workspace deps `Cargo.toml` 의 `[workspace.dependencies]` 신규 (fastembed transitive 와 정확히 일치): - `ort = "=2.0.0-rc.9"` (default-features=false, ndarray) - `tokenizers = "0.21"` (default-features=false, onig — pre-flight lock) - `hf-hub = "0.4"` (default-features=false, ureq+rustls-tls) - `ndarray = "0.16"` PR-9b 가 `crates/kebab-nli/Cargo.toml` 에 추가 (현재는 workspace 만 선언). ### Spec + plan 동봉 PR-9 design contract 가 본 PR-9a 부터 다음 sub-PR (PR-9b/9c-1/9c-2/9d/cut PR) 모두 참조. 별 PR 분리 시 contract 의 main reference 가 지연 → 본 PR 안에 docs commit 으로 분리하여 함께 머지. ## Cargo features 결정 trace **Pre-flight 1 — HF reachability** (spec §2.1): ``` curl -I https://huggingface.co/Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7/resolve/main/onnx/model.onnx → HTTP/2 302 (HF S3 routing, file 존재) curl -I https://huggingface.co/Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7/resolve/main/tokenizer.json → HTTP/2 302 ``` **Pre-flight 2 — tokenizers features lock** (spec §3 PR-9a): ``` cargo new --bin /tmp/nli-tok-probe cargo add tokenizers --no-default-features -F onig wget tokenizer.json (16 MB) cargo run --release → ok | n_tokens(ko) = 9 → ok | n_tokens(en) = 11 ``` **Lock**: `tokenizers = { version = "0.21", default-features = false, features = ["onig"] }`. SentencePiece mDeBERTa tokenizer.json 로드 + KR/EN encode 정상. ## 검증 - `cargo test -p kebab-nli -j 1` — 6/6 통과: - softmax3_normalises_to_unit, softmax3_is_invariant_to_constant_shift, - nli_scores_from_xnli_logits_orders_correctly, faithfulness_returns_entailment_channel, - new_succeeds_on_default_config, score_returns_err_in_skeleton. - `cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings` clean. - `cargo test --workspace --no-fail-fast -j 1` — pre-existing `kebab-mcp::tools_call_ask_multi_hop::ask_tool_routes_multi_hop_true_to_decompose_first` 1 fail. **main baseline 동일 fail 확인 — PR-9a 무관** (ingest fixture / Ollama 의존 flaky, HOTFIXES candidate). - 워크스페이스 빌드 클린 (kebab-nli 의 dep chain 도입이 18 integration test binary link 단계에 회귀 없음). ## 비범위 - ONNX inference (PR-9b 의 `OnnxNliVerifier::score` 실 구현). - `crates/kebab-nli/Cargo.toml` 에 ort/tokenizers/hf-hub/ndarray 의존 추가 (PR-9b). - Pipeline integration (PR-9c-2 의 `ask_multi_hop` step 8.5 hook). - Wire schema 변경 (PR-9c-1 의 `answer.v1.verification` + `RefusalReason::NliVerificationFailed` 등). ## 시험 항목 (Test Plan) - [x] cargo test -p kebab-nli -j 1 6/6 통과. - [x] cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings clean. - [x] cargo test --workspace -j 1 회귀 0 (pre-existing 1 fail 은 main baseline 동일 — PR-9a 무관). - [x] Pre-flight 1 + 2 결과 첨부 (## Cargo features 결정 trace). Assisted-by: Claude Code
altair823 added 2 commits 2026-05-25 21:23:56 +00:00
docs(rag): fb-41 PR-9 spec + plan — NLI verification + v0.18.0 cut 44fbffff26 
fb-41 multi-hop RAG 의 dogfood S7 hallucination root cause = LLM-self-judge ceiling.
대응 = NLI-based post-synthesis verification (mDeBERTa-v3 XNLI, 280 MB ONNX).

산출물:
- docs/superpowers/specs/2026-05-25-p9-fb-41-finalize-spec.md (review_round=5,
  4 OMC reviewer APPROVE: 1 CRITICAL + 9 MAJOR + 3 MINOR → 1 NIT carry-forward).
- docs/superpowers/plans/2026-05-25-p9-fb-41-finalize-plan.md (plan_review_round=3,
  4 OMC reviewer APPROVE: 15 issues → 0 actionable).

5 sub-PR (PR-9a~9d) + cut PR. 작업 21-31h / wall time 28-44h.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat(nli): fb-41 PR-9a — kebab-nli crate skeleton + workspace deps 1eb0bbecb3 
- 신규 crate kebab-nli (trait + impl 동일 crate, v0.18 scope = ONNX adapter 1개).
- NliVerifier trait + NliScores struct (XNLI 3-channel: entailment/neutral/contradiction).
- private softmax3 (log-sum-exp 안전).
- OnnxNliVerifier placeholder (PR-9b 가 ONNX inference + model download 추가).
- workspace.dependencies 추가: ort 2.0-rc.9, tokenizers 0.21 (default-features=false, onig), hf-hub 0.4, ndarray 0.16.

Pre-flight (PR-9 design contract 의 gate):
- HF Xenova/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 model.onnx + tokenizer.json → HTTP/2 302 (HF S3 routing, file 존재).
- tokenizers --no-default-features -F onig 의 standalone repro: SentencePiece mDeBERTa tokenizer.json 로드 OK (KR 9 tokens / EN 11 tokens 정상 encode).
- Cargo features 결정 trace: tokenizers = { default-features = false, features = ["onig"] } lock.

Tests: 6 unit (softmax3 정규화 + 불변성 + XNLI logits 변환 + faithfulness + new + score stub) — 통과.
Verification: cargo test -p kebab-nli -j 1 (6/6) + cargo clippy -p kebab-nli --all-targets -j 1 -- -D warnings clean.
Workspace: cargo test --workspace -j 1 — pre-existing kebab-mcp::tools_call_ask_multi_hop 1 fail (main baseline 동일 fail, PR-9a 무관 — ingest fixture/Ollama 의존 flaky).

Wire 영향: 없음 (crate 도입만).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-reviewer-01 requested changes 2026-05-25 21:25:14 +00:00
Dismissed
claude-reviewer-01 left a comment
Member
Copy Link

회차 1 — PR-9a kebab-nli skeleton 의 trait surface + scaffolding 검토.

전반적으로 spec §2.2 와 정합 (trait surface, NliScores 3-channel, softmax3 log-sum-exp 안전성, OnnxNliVerifier placeholder 의 명시적 stub err). 6 unit test 모두 의도대로 동작 + clippy clean. workspace.dependencies 추가도 fastembed transitive 와 정확히 일치 (pre-flight 결과 trace inline).

작은 nit 1건 inline. 그 외는 의도된 placeholder 패턴 — PR-9b 가 자연스럽게 채울 자리.

칭찬: pre-flight 2단계 결과 (HF reachability + tokenizers features lock) 의 trace 가 PR body 의 ## Cargo features 결정 trace 절에 reproducible 명시 — 후속 PR-9b 작업자가 동일 셋업 즉시 가능. 또한 lib.rs::softmax3 의 log-sum-exp 안전성 test (softmax3_is_invariant_to_constant_shift 의 +100 shift) 가 numerical stability 명시 검증 — 좋은 회귀 핀.

회차 1 — PR-9a kebab-nli skeleton 의 trait surface + scaffolding 검토. 전반적으로 spec §2.2 와 정합 (trait surface, NliScores 3-channel, softmax3 log-sum-exp 안전성, OnnxNliVerifier placeholder 의 명시적 stub err). 6 unit test 모두 의도대로 동작 + clippy clean. workspace.dependencies 추가도 fastembed transitive 와 정확히 일치 (pre-flight 결과 trace inline). 작은 nit 1건 inline. 그 외는 의도된 placeholder 패턴 — PR-9b 가 자연스럽게 채울 자리. 칭찬: pre-flight 2단계 결과 (HF reachability + tokenizers features lock) 의 trace 가 PR body 의 ## Cargo features 결정 trace 절에 reproducible 명시 — 후속 PR-9b 작업자가 동일 셋업 즉시 가능. 또한 lib.rs::softmax3 의 log-sum-exp 안전성 test (`softmax3_is_invariant_to_constant_shift` 의 +100 shift) 가 numerical stability 명시 검증 — 좋은 회귀 핀.
crates/kebab-nli/src/lib.rs Outdated
@@ -0,0 +25,4 @@
impl NliScores {
/// Faithfulness score = entailment channel. The rag crate compares this
/// against `rag.nli_faithfulness_min` to decide whether to refuse.
claude-reviewer-01 commented 2026-05-25 21:25:14 +00:00
Member
Copy Link

comment 의 rag.nli_faithfulness_min 가 spec §2.5/§2.6 의 실 config knob 이름 rag.nli_threshold 와 mismatch. 본 PR-9a 는 trait surface 만 도입 (config 자체는 PR-9c-1 이 추가) 이라 caller 가 없어 회귀 없음 — 그러나 main 머지 시점 spec authority 와 산문상 inconsistency 가 한 PR 동안 잔존. comment 를 rag.nli_threshold 로 정정 권장 (1 단어 교체).

comment 의 `rag.nli_faithfulness_min` 가 spec §2.5/§2.6 의 실 config knob 이름 `rag.nli_threshold` 와 mismatch. 본 PR-9a 는 trait surface 만 도입 (config 자체는 PR-9c-1 이 추가) 이라 caller 가 없어 회귀 없음 — 그러나 main 머지 시점 spec authority 와 산문상 inconsistency 가 한 PR 동안 잔존. comment 를 `rag.nli_threshold` 로 정정 권장 (1 단어 교체).
altair823 added 1 commit 2026-05-25 21:25:50 +00:00
chore(nli): PR #176 회차 1 리뷰 반영 1d88dccf8a 
- lib.rs::NliScores::faithfulness doc 의 `rag.nli_faithfulness_min` → `rag.nli_threshold` (spec §2.5/§2.6 의 실 config knob 이름 정합).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-reviewer-01 approved these changes 2026-05-25 21:26:04 +00:00
claude-reviewer-01 left a comment
Member
Copy Link

회차 2 — 회차 1 N1 (lib.rs::faithfulness doc 의 rag.nli_faithfulness_min → rag.nli_threshold) 반영 완료. spec §2.5/§2.6 의 실 config knob 이름 정합 확인.

추가 actionable 없음. PR-9a scope (trait surface + scaffolding + workspace dep chain) 완성. PR-9b 가 OnnxNliVerifier 의 ONNX inference + model download 채울 자연스러운 baseline.

머지 OK.

회차 2 — 회차 1 N1 (lib.rs::faithfulness doc 의 `rag.nli_faithfulness_min` → `rag.nli_threshold`) 반영 완료. spec §2.5/§2.6 의 실 config knob 이름 정합 확인. 추가 actionable 없음. PR-9a scope (trait surface + scaffolding + workspace dep chain) 완성. PR-9b 가 OnnxNliVerifier 의 ONNX inference + model download 채울 자연스러운 baseline. 머지 OK.
altair823 merged commit 11ce7847a1 into main 2026-05-25 21:34:51 +00:00
altair823 deleted branch feat/fb-41-pr-9a-kebab-nli-crate 2026-05-25 21:34:52 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
claude-reviewer-01
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
altair823 claude-reviewer-01
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: altair823-org/kebab#176
Delete Branch "feat/fb-41-pr-9a-kebab-nli-crate"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?