altair823
58a11cc2b8
- new kb-eval crate: load_golden_set (YAML) + run_eval (per-query search/ask + persistence) - new kb-store-sqlite::eval module: record_eval_run_with_results (transactional), document_exists / chunk_exists probes - fixtures/golden_queries.yaml: 5-entry KO+EN template - tests: 13 pass (loader: parse, dup-id, missing chunk_id; runner: elapsed, snapshot, error capture, JSONL, determinism, persistence, config_snapshot) - per_query.jsonl mirror written to runs_dir/<run_id>/ - temperature=0 + fixed seed → byte-identical per_query.jsonl (lexical) deviations from spec (documented in code): - run_id uses uuid::Uuid::now_v7().simple() (timestamp-ordered hex) instead of ULID — uuid already in workspace deps - load_golden_set_validated kept #[cfg(test)] pub(crate) — production inlines validate_against_db - snapshot fixture uses normalized projection (id/query/mode/first_hit) — full byte-determinism covered by separate test - index_version in config_snapshot left null (composed per call by kb-app, not config-level) deferred to follow-up: - App reuse across queries (currently rebuilds App per query) - expand_path hoist to kb-config (3 crate clones now) - --max-queries flag (deferred to P5-2 per updated spec) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
46 lines
1.3 KiB
YAML
46 lines
1.3 KiB
YAML
# Golden query suite for `kb eval run` (P5-1 / P5-2).
|
|
#
|
|
# Top-level: list of queries. Required fields: `id`, `query`. All
|
|
# others are optional and default to empty / null.
|
|
#
|
|
# Curators: `expected_doc_ids` and `expected_chunk_ids` MUST refer to
|
|
# real rows in the active workspace's SQLite store at run time. Stale
|
|
# references make the runner bail at start. The shipped template
|
|
# leaves them empty so the file is loadable on any fresh workspace —
|
|
# fill them in after a `kb ingest` to enable hit@k / MRR metrics
|
|
# (P5-2).
|
|
#
|
|
# `must_contain` / `forbidden` drive the rule-based groundedness
|
|
# metric (P5-2).
|
|
|
|
- id: g001
|
|
query: "Cargo workspace 멤버 추가하는 법"
|
|
lang: ko
|
|
must_contain: ["[workspace]", "members"]
|
|
difficulty: easy
|
|
|
|
- id: g002
|
|
query: "What is Rust ownership?"
|
|
lang: en
|
|
must_contain: ["borrow", "lifetime"]
|
|
difficulty: easy
|
|
|
|
- id: g003
|
|
query: "Markdown chunking 규칙은?"
|
|
lang: ko
|
|
must_contain: ["heading"]
|
|
forbidden: ["embedding"]
|
|
difficulty: medium
|
|
|
|
- id: g004
|
|
query: "How does FTS5 tokenization work for Korean text?"
|
|
lang: en
|
|
must_contain: ["unicode61", "tokenizer"]
|
|
difficulty: medium
|
|
|
|
- id: g005
|
|
query: "RAG citation 검증은 어떻게 동작?"
|
|
lang: ko
|
|
must_contain: ["citation", "marker"]
|
|
difficulty: hard
|