altair823
a48b055358
asset(문서) 단위뿐이던 ingest 진행 이벤트에 문서 내부 phase 가시성을 추가.
큰 문서가 expansion(별칭 LLM, 청크당 순차)으로 수십 분 걸려도 진행바가
1/N 에 멈춘 듯 보이던 문제 해결.
wire ingest_progress.v1 additive (backward-compat):
- asset_chunked {idx,total,chunks} — 청킹 직후, markdown/image/pdf 전 경로
- expansion_progress {idx,total,done,chunks} — expansion 루프 스로틀
(25청크 또는 1s, 종료 시 done==chunks). 캐시 히트도 done 에 포함
- asset_timings {idx,total,parse_ms,chunk_ms,expansion_ms,embed_ms,store_ms}
— markdown 경로 phase별 wall-clock
설계: timing 은 kebab_core::IngestItem(wire-stable) 변경을 피해 신규
AssetTimings 이벤트로 ingest_one_asset 가 직접 emit (AssetFinished 무변경).
CLI(progress.rs): 진행바 sub-message(→ N chunks / 별칭 확장 done/chunks) +
asset 종료 시 phase timing 한 줄(fmt_ms). TUI reducer no-op arm.
검증: clippy -D warnings exit 0; cargo test -p kebab-app -p kebab-cli
312 passed/0 failed. ordering-invariant 테스트 재작성 + 신규 직렬화 테스트.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
71 lines
5.7 KiB
JSON
71 lines
5.7 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://kb.local/wire/v1/ingest_progress.schema.json",
|
|
"title": "IngestProgressEvent v1",
|
|
"description": "Streaming progress event emitted by `kebab ingest --json`. One event per line (line-delimited JSON). Discriminated by `kind`. The terminal events are `completed` and `aborted` — every ingest run ends with exactly one of them. The final stdout line of a `--json` ingest is still the existing `ingest_report.v1` for backwards compatibility; progress events stream above it.",
|
|
"type": "object",
|
|
"required": ["schema_version", "kind", "ts"],
|
|
"properties": {
|
|
"schema_version": { "const": "ingest_progress.v1" },
|
|
"kind": {
|
|
"type": "string",
|
|
"enum": [
|
|
"scan_started",
|
|
"scan_completed",
|
|
"asset_started",
|
|
"asset_finished",
|
|
"asset_chunked",
|
|
"expansion_progress",
|
|
"asset_timings",
|
|
"embed_batch_started",
|
|
"embed_batch_finished",
|
|
"pdf_ocr_started",
|
|
"pdf_ocr_finished",
|
|
"completed",
|
|
"aborted"
|
|
]
|
|
},
|
|
"ts": { "type": "string", "format": "date-time", "description": "RFC 3339 timestamp at the moment the event was emitted." },
|
|
"root": { "type": "string", "description": "scan_started: workspace root being walked." },
|
|
"total": { "type": "integer", "minimum": 0, "description": "scan_completed / asset_started / asset_finished: total assets discovered." },
|
|
"idx": { "type": "integer", "minimum": 1, "description": "asset_started / asset_finished: 1-based index of the current asset within the scan." },
|
|
"path": { "type": "string", "description": "asset_started: workspace-relative path of the asset being processed." },
|
|
"media": { "type": "string", "description": "asset_started: media kind label (e.g. `markdown`, `pdf`, `image`)." },
|
|
"result": {
|
|
"type": "string",
|
|
"enum": ["new", "updated", "skipped", "error"],
|
|
"description": "asset_finished: per-asset outcome (mirrors `ingest_report.v1.items[].kind`)."
|
|
},
|
|
"chunks": { "type": "integer", "minimum": 0, "description": "asset_finished / asset_chunked / expansion_progress (v0.24.0): chunk count produced for this asset." },
|
|
"done": { "type": "integer", "minimum": 0, "description": "expansion_progress (v0.24.0, additive): chunks processed so far in the per-chunk alias-expansion loop (cache hits included). Throttled: emitted at most every 25 chunks or once per second, plus a final frame where done == chunks." },
|
|
"parse_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): parse phase wall-clock (ms). Markdown path only." },
|
|
"chunk_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): chunk phase wall-clock (ms). Markdown path only." },
|
|
"expansion_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): alias-expansion phase wall-clock (ms). Markdown path only; 0 when expansion is disabled." },
|
|
"embed_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): embed + vector-upsert phase wall-clock (ms). Markdown path only." },
|
|
"store_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): SQLite persist phase wall-clock (ms). Markdown path only." },
|
|
"n_chunks": { "type": "integer", "minimum": 0, "description": "embed_batch_started / embed_batch_finished: chunks in this embedding batch." },
|
|
"ms": { "type": "integer", "minimum": 0, "description": "embed_batch_finished / pdf_ocr_finished: wall-clock duration (ms). pdf_ocr_finished skip path 의 의미는 mixed (DCTDecode 부재 시 0, engine 실패 시 latency-before-bail)." },
|
|
"chars": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished: char count of OCR result. Skip 시 0." },
|
|
"page": { "type": "integer", "minimum": 1, "description": "pdf_ocr_started / pdf_ocr_finished: 1-based PDF page number under OCR." },
|
|
"ocr_engine": { "type": "string", "description": "pdf_ocr_finished: engine_name (e.g. 'ollama-vision')." },
|
|
"skipped": { "type": "boolean", "description": "pdf_ocr_finished: true 일 시 OCR 미수행 (DCTDecode 부재 또는 engine 실패). chars=0 만으로는 skip 과 0-char result 구분 불가." },
|
|
"image_byte_size": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished (optional, v0.20.x): raster image byte size." },
|
|
"image_width": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished (optional, v0.20.x): raster image width px." },
|
|
"image_height": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished (optional, v0.20.x): raster image height px." },
|
|
"failure_reason": { "type": "string", "description": "pdf_ocr_finished (optional, v0.20.x): OCR failure reason. Present iff skipped=true due to engine error. Values: timeout | ocr_error | network_error | other." },
|
|
"counts": {
|
|
"type": "object",
|
|
"description": "completed / aborted: aggregate counters at the moment the run ended (mirrors fields on `ingest_report.v1`).",
|
|
"properties": {
|
|
"scanned": { "type": "integer", "minimum": 0 },
|
|
"new": { "type": "integer", "minimum": 0 },
|
|
"updated": { "type": "integer", "minimum": 0 },
|
|
"skipped": { "type": "integer", "minimum": 0 },
|
|
"errors": { "type": "integer", "minimum": 0 },
|
|
"chunks_indexed": { "type": "integer", "minimum": 0 },
|
|
"embeddings_indexed": { "type": "integer", "minimum": 0 }
|
|
}
|
|
}
|
|
}
|
|
}
|