altair823-org/kebab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 40 Wiki Activity
282 Commits 7 Branches 40 Tags
c97e8e00efec81603195fd7e90e4193c75ea77e7
Commit Graph

6 Commits

Author SHA1 Message Date
altair823
c97e8e00ef feat(kebab-core + kebab-store-sqlite): p9-fb-17 chat session storage (V005) 
도그푸딩 item 13/14 (multi-turn 영속화) — TUI Ask 의 "이전 대화
이어가기" + 향후 CLI `--session foo` (p9-fb-18) backing store. session
header + per-turn 두 테이블, ON DELETE CASCADE 로 reset --data-only 가
한꺼번에 wipe.

## 핵심 변경

- **SQLite V005 migration** `chat_sessions` (session_id PK + created_at
  + updated_at + title + config_snapshot_json) + `chat_turns` (turn_id
  PK + session_id FK ON DELETE CASCADE + turn_index + question +
  answer + citations_json + created_at + UNIQUE(session_id, turn_index))
  + `idx_chat_turns_session(session_id, turn_index)`. 모두 `STRICT`.
- **`kebab_core::ChatSessionRepo`** trait (6 method): create_session /
  get_session / list_sessions(limit, ORDER BY updated_at DESC) /
  delete_session / append_turn / list_turns(ORDER BY turn_index ASC)
- **`kebab_core::{ChatSessionRow, ChatTurnRow}`** structs — Serialize
  + Deserialize 둘 다 (CLI / wire 출력 호환)
- **`kebab-store-sqlite::SqliteStore`** impl 신규 모듈 `chat_sessions.rs`.
  `append_turn` 이 insert + parent updated_at bump 같은 connection
  에서 처리.
- **frozen design §5** 에 §5.7a chat_sessions / chat_turns 절 신설
  (full schema + trait 메서드 6 개 명시).

## HOTFIXES (V004 → V005)

spec p9-fb-17 의 `V004__chat_sessions.sql` 가 p9-fb-19 의
`V004__kv.sql` (이미 머지) 와 refinery migration number 충돌. 무중단
정정: `V005__chat_sessions.sql` 로 시프트. schema / 동작 동일, 파일명
만 이동. HOTFIXES entry 추가.

## 테스트

- 9 신규 integration unit (create/get roundtrip, missing→None, PK
  collision error, append+list ordered, dup turn_index error,
  append bumps updated_at, delete CASCADE turns, list_sessions
  ORDER BY updated_at DESC, list_sessions LIMIT)
- workspace 전체 `cargo test --workspace --no-fail-fast -j 1` exit 0
- `cargo clippy --workspace --all-targets -- -D warnings` clean

## 문서

- frozen design §5.7a 신설
- HANDOFF: 2026-05-03 entry
- HOTFIXES: V004 → V005 rename rationale
- spec status planned → in_progress

## Out of scope

- session 검색 / 필터 UI (p9-fb-18 의 `kebab ask --session list`
  같은 admin command 가 후속)
- 다른 store backend (postgres 등) — trait 만 정의, impl 은 SQLite

unblocks p9-fb-18 (CLI session/repl).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 05:37:53 +00:00
altair823
0e408fb1b5 feat(kebab-app + kebab-store-sqlite): p9-fb-19 search LRU cache + corpus_revision 
도그푸딩 item 15 — TUI / 같은 process 안에서 동일 query 반복 시 SQLite
FTS + Lance + RRF 재계산이 매번 발생하던 비용 해소. in-process LRU
캐시 + 모노토닉 corpus_revision 카운터로 ingest commit 발생 시 모든
entry 자동 stale.

## 핵심 변경

- **SQLite V004 migration**: `kv (key TEXT PRIMARY KEY, value TEXT)
  STRICT` + `corpus_revision = '0'` seed. 미래의 다른 scalar 도 같은
  테이블에 들어갈 수 있는 generic shape.
- **`SqliteStore::corpus_revision()` / `bump_corpus_revision()`** —
  `UPDATE ... CAST AS INTEGER + 1` atomic. INSERT-OR-IGNORE 도 함께
  실행 (V004 seed 가 무슨 이유로 누락된 케이스 paranoid).
- **`kebab-app::ingest_with_config_cancellable`** — `new + updated > 0`
  시 bump, no-op (skipped-only) reingest 는 cache 보존.
- **`App.search_cache: Option<Mutex<LruCache<SearchCacheKey, Vec<
  SearchHit>>>>`** — `config.search.cache_capacity` (default 256, 0
  비활성). `lru = "0.12"` workspace dep 추가.
- **`SearchCacheKey`** = `query_norm` (NFKC + trim + lowercase) +
  `mode` + `k` + `snippet_chars` + `embedding_version` (vector/hybrid
  만, lexical 은 빈 문자열) + `chunker_version` + `corpus_revision`
  snapshot.
- **`App::search`** rewrite — cache 활성 시 lookup → miss 면 기존
  `search_uncached` 호출 후 put. cache 비활성이거나 lock 실패면
  straight-line.
- **`App::search_uncached`** (rename of pre-fb-19 `search` body) +
  `search_uncached_with_config` facade — CLI `kebab search --no-cache`
  로 진입.
- **`Config.search.cache_capacity: usize`** field, `#[serde(default)]`
  로 기존 config 호환.
- **CLI `--no-cache`** flag — 디버깅용 (CLI 는 매 호출이 새 process
  라 사실상 no-op 이지만 spec 명시 + 향후 long-lived process 호환).
- **frozen design §9 versioning** 표에 `corpus_revision` row 추가
  (기존 `index_version` 라벨과 다른 차원: 라벨은 retrieval 형상,
  corpus_revision 은 ingest commit ack).

## 테스트

- `kebab-store-sqlite` 신규 3 unit (fresh=0, monotonic bump, persist
  across reopen)
- `kebab-app` 신규 4 integration (cached repeat 같은 hits, NFKC 정규화
  로 case/whitespace collapse, --no-cache parity, first ingest bumps
  corpus_revision)
- 워크스페이스 전체 `cargo test --workspace --no-fail-fast -j 1` exit 0
- `cargo clippy --workspace --all-targets -- -D warnings` clean

## 문서

- README `kebab search` 행: 캐시 동작 + `--no-cache` 안내 + corpus_
  revision 무효화 메커니즘
- docs/SMOKE.md `[search]` 절에 `cache_capacity` 라인 추가
- HANDOFF: 2026-05-03 entry
- spec status planned → in_progress

## Out of scope

- patch-and-merge incremental (RRF 정규화 전체 hit set 기준이라 어려움)
- SQLite 영속 cache (P+)
- 다른 process 간 cache 공유 (in-process 만 — corpus_revision 이
  cross-process 무효화는 O(1))

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 05:01:31 +00:00
altair823
3cd5117a7e feat(p3-3): kb-store-vector — LanceDB VectorStore + V003 embedding status 
First VectorStore implementation. Per-model Lance tables under
config.storage.vector_dir, two-phase upsert (SQLite-pending → Lance
MergeInsert → SQLite-committed) with crash-safe retry, search via
cosine distance with the spec's score-shift (preserves negative
similarity ranking signal that clamping would crush).

V003 migration:
- Adds status (CHECK constraint pending|committed|tombstone, default
  pending) and vector_committed columns to embedding_records.
- BEFORE DELETE trigger on chunks flips dependent rows to tombstone.
  Currently overshadowed by V001's ON DELETE CASCADE FK; trigger UPDATE
  runs first then row vanishes via CASCADE. Spec-faithful tombstone
  preservation requires recreating embedding_records to drop the
  CASCADE — deferred to a P+ migration since no production rows exist
  yet (P3-3 is the first writer). V003 SQL comment explains.

LanceVectorStore:
- ensure_table is idempotent: opens existing or creates with the
  Arrow schema (chunk_id, doc_id, embedding FixedSizeList<Float32,
  dim>, model_id, embedding_version, text, heading_path, created_at).
- IndexId computed via id_for_index with collection="chunk_embeddings",
  index_kind="flat", params_hash = blake3(descriptor JSON). Schema
  bumps automatically rotate the IndexId.
- upsert: phase-1 INSERT OR REPLACE INTO embedding_records (status=
  'pending') in a single SQLite tx; phase-2 Lance MergeInsert keyed
  on chunk_id (idempotent re-run); phase-3 UPDATE status='committed',
  vector_committed=1. If phase-2 fails the rows stay 'pending' and
  the next upsert call retries idempotently.
- search joins embedding_records WHERE status='committed' so partial-
  write rows never surface. Cosine distance from Lance ∈ [0, 2] →
  similarity = 1 - distance ∈ [-1, 1] → score = (similarity + 1)/2 ∈
  [0, 1]. NaN coerced to 0 with tracing::warn. Filter by SearchFilters
  via SqliteStore::filter_chunks (added in this commit).
- Sync trait + async LanceDB bridged by an embedded current-thread
  tokio runtime. Doc-comment on the struct flags the "do NOT call
  from inside another tokio runtime" panic (block_on cannot nest).
  kb-app's job scheduler is sync today.

kb-store-sqlite additions:
- pub fn put_embedding_records_pending(&[EmbeddingRecordRow]) — phase-1
  INSERT OR REPLACE (status='pending', vector_committed=0).
- pub fn mark_embedding_records_committed(&[EmbeddingId]) — phase-3
  single UPDATE … WHERE embedding_id IN (?, ?, …) via
  params_from_iter, guarded by WHERE status='pending' so tombstones
  don't get clobbered.
- pub fn filter_chunks(&[ChunkId], &SearchFilters) → Vec<ChunkId>
  consolidates the JOIN against documents/document_tags/
  embedding_records + path_glob via globset. Lets kb-store-vector
  honor SearchFilters without depending on rusqlite or globset
  directly. (kb-search's filter logic is structurally different —
  interleaved with the FTS5 SELECT — so it stays as-is for now;
  consolidation is a P+ refactor.)
- 4 new unit tests cover the phase-1 round-trip, empty batch,
  replay reset of pending rows, and the WHERE-status-pending guard.

Tests:
- 9 lib unit tests in kb-store-vector covering paths/sanitization,
  arrow_batch dim validation + descriptor hash, bm25-style cosine
  score shift math.
- 4 new kb-store-sqlite unit tests on filter_chunks (committed-only,
  tags/lang/trust/path_glob, order preservation, empty input).
- 4 new kb-store-sqlite unit tests on the embedding_records helpers.
- 8 integration tests in upsert_search.rs and 1 snapshot test marked
  #[ignore = "requires AVX-capable hardware (LanceDB)"]. They invoke
  require_avx_or_panic() at the top of each body so a missing-AVX
  --ignored run fails loudly instead of silently passing. This dev
  host (qemu64 model) lacks AVX so these were NOT exercised end-to-
  end here — first CI lane on AVX hardware will validate them.
- Snapshot fixture tests/fixtures/vector/run-1.json is a placeholder
  with an _comment marker. Snapshot test panics until the placeholder
  is replaced via KB_UPDATE_SNAPSHOTS=1 on AVX hardware.
- Workspace 241 passed, 19 ignored, 0 failed; cargo clippy --workspace
  --all-targets -- -D warnings clean.

Allowed deps respected (kb-core, kb-config, kb-store-sqlite, lancedb,
arrow + arrow-array + arrow-schema, serde, serde_json, tracing,
thiserror) plus forced waivers — anyhow (trait return type), tokio
+ futures (LanceDB async-only API), blake3 (params_hash). rusqlite
and globset are NOT direct deps of kb-store-vector — confirmed via
cargo metadata --no-deps. rusqlite stays in [dev-dependencies] for
the test fixture seeder only.

Out of scope: IVF/PQ index tuning (P+), image vectors (P6), kb-app
embed_index orchestration (P3-4 facade).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 10:01:31 +00:00
altair823
94bfc50efd feat(p2-1): chunks_fts virtual table + sync triggers (V002 migration) 
Adds FTS5 lexical index for chunks per design §5.5: chunks_fts virtual
table (unicode61 remove_diacritics 2 tokenizer, contentless w/ UNINDEXED
chunk_id+doc_id) plus chunks_ai/chunks_ad/chunks_au triggers that mirror
every chunks mutation into chunks_fts inside the host transaction.

V002 ships the verbatim §5.5 SQL block plus a one-shot backfill INSERT
so existing P1 databases gain searchability without re-ingest. Refinery
bookkeeping makes double-apply naturally idempotent.

Adds rebuild_chunks_fts(&Connection) escape hatch for kb index
--rebuild-fts (CLI wiring deferred to a later task). Uses SAVEPOINT
instead of Transaction so callers can invoke from inside an outer
transaction; WAL serializes writers so no DELETE/INSERT race vs.
concurrent chunks mutators is possible.

Tests (10): V001-only → V002 cold-upgrade backfill (literal path),
chunks_ai/ad/au trigger sync, MATCH-token verification, rebuild
idempotency, drift recovery, double-run no-op, V002 ↔ design §5.5
verbatim diff guard (anchored extraction from both files), WAL/SHM
release on store drop. All 185 workspace tests pass.

Allowed deps respected (kb-core, kb-config, rusqlite, refinery — no
new deps). FTS query implementation deferred to p2-2 (lexical-retriever).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 04:42:15 +00:00
altair823
a3390d5171 p1-6: scaffold kb-store-sqlite crate + V001 full §5 DDL 
New workspace member crate `kb-store-sqlite` (allowed deps only:
kb-core, kb-config, rusqlite[bundled], refinery, serde, serde_json,
time, blake3, tracing, anyhow, thiserror; dev-deps add kb-parse-md /
kb-normalize / kb-chunk for the contract round-trip test).

Migration V001 replaces the P0-1 stub with the full §5 DDL (assets,
documents, document_tags, blocks, chunks with policy_hash,
embedding_records, jobs, ingest_runs, answers, eval_runs,
eval_query_results) plus the §5 indexes. FTS5 virtual table + triggers
remain deferred to V002 (P2-1).

Public surface per task spec:
  SqliteStore::open / run_migrations / put_asset_with_bytes
  impl DocumentStore for SqliteStore (7 trait methods)
  impl JobRepo for SqliteStore (4 trait methods)
  StoreError { Sqlx, Migration, Conflict }

Behavior:
- Pragmas at open: foreign_keys=ON, journal_mode=WAL,
  synchronous=NORMAL, temp_store=MEMORY.
- Asset writer: byte_len ≤ copy_threshold_mb * 1MiB → copy to
  data_dir/assets/<aa>/<asset_id> (mode 0o644 on Unix), else
  reference. blake3(bytes) verified against asset.checksum; mismatch →
  Conflict.
- Idempotency: put_document UPSERTs and bumps doc_version + 1 on
  conflict; put_blocks / put_chunks DELETE-then-INSERT; document_tags
  re-derived per put_document.
- get_document rehydrates blocks via payload_json ordered by stream
  ordinal.
- list_documents builds dynamic WHERE from DocFilter (lang / trust_min
  / path_glob via GLOB / tags_any via document_tags subquery).
- JobRepo: jobs.kind/status are stored as lowercase enum tags; create
  mints a 32-hex JobId via blake3(kind || payload || nanos).

Tests follow in subsequent commits.
 2026-04-30 17:08:36 +00:00
altair823
a166b7051c p0-1: wire-schema stubs, doc/spec stubs, V001 migration, fixtures 
- docs/wire-schema/v1/ ships 7 schema stubs (citation, search_hit,
  answer, ingest_report, doc_summary, chunk_inspection, doctor) that
  pin schema_version + required fields per design §2. Full property
  validation lands in later phases.
- docs/spec/ ships 7 markdown stubs each linking to the canonical
  frozen design (domain-model, ids, canonical-document, chunk-policy,
  citation-policy, module-boundaries, ai-generation-guidelines).
- migrations/V001__init.sql contains only schema_meta + migrations
  tables per design §5.1; data tables ship in P1-6/P2-1/P3-3.
- fixtures/ has the 11 subdirectories every downstream task references
  (markdown, source-fs, search/{lexical,hybrid}, embed, vector, rag,
  eval, image, pdf, audio). Empty subdirs use .gitkeep so they track.
  fixtures/markdown/ ships the 3 phase-0 fixtures: simple-note.md,
  nested-headings.md, code-and-table.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 05:17:32 +00:00