altair823
58a11cc2b8
- new kb-eval crate: load_golden_set (YAML) + run_eval (per-query search/ask + persistence) - new kb-store-sqlite::eval module: record_eval_run_with_results (transactional), document_exists / chunk_exists probes - fixtures/golden_queries.yaml: 5-entry KO+EN template - tests: 13 pass (loader: parse, dup-id, missing chunk_id; runner: elapsed, snapshot, error capture, JSONL, determinism, persistence, config_snapshot) - per_query.jsonl mirror written to runs_dir/<run_id>/ - temperature=0 + fixed seed → byte-identical per_query.jsonl (lexical) deviations from spec (documented in code): - run_id uses uuid::Uuid::now_v7().simple() (timestamp-ordered hex) instead of ULID — uuid already in workspace deps - load_golden_set_validated kept #[cfg(test)] pub(crate) — production inlines validate_against_db - snapshot fixture uses normalized projection (id/query/mode/first_hit) — full byte-determinism covered by separate test - index_version in config_snapshot left null (composed per call by kb-app, not config-level) deferred to follow-up: - App reuse across queries (currently rebuilds App per query) - expand_path hoist to kb-config (3 crate clones now) - --max-queries flag (deferred to P5-2 per updated spec) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
30 lines
1.0 KiB
Rust
30 lines
1.0 KiB
Rust
//! `kb-eval` — golden-fixture eval runner (P5-1).
|
|
//!
|
|
//! Loads `fixtures/golden_queries.yaml`, runs each entry through the
|
|
//! [`kb_app`] facade (lexical / vector / hybrid + optional RAG), and
|
|
//! persists results into `eval_runs` / `eval_query_results` plus
|
|
//! `runs_dir/<run_id>/per_query.jsonl` (design §5.7, §6.3).
|
|
//!
|
|
//! Metric computation lives in P5-2 (`kb-eval::metrics`); this crate is
|
|
//! the **data collector** only.
|
|
//!
|
|
//! ## Allowed deps (per task spec)
|
|
//!
|
|
//! `kb-core`, `kb-config`, `kb-app`, `kb-store-sqlite`, plus `serde`,
|
|
//! `serde_yaml`, `serde_json`, `time`, `tracing`,
|
|
//! `anyhow`, `uuid`. Retrieval / embedding / LLM crates are NOT
|
|
//! reachable here — every retrieval and `ask` call must go through
|
|
//! `kb-app`.
|
|
//!
|
|
//! ## `run_id` recipe
|
|
//!
|
|
//! `run_id` uses UUIDv7 simple — timestamp-ordered, lowercase hex.
|
|
|
|
mod loader;
|
|
mod runner;
|
|
mod types;
|
|
|
|
pub use loader::load_golden_set;
|
|
pub use runner::{run_eval, run_eval_with_config};
|
|
pub use types::{EvalRun, EvalRunOpts, GoldenQuery, QueryResult};
|