feat(app): #228 삭제 sweep 을 진행바·ndjson 로그에 노출

`sweep_deleted_files` 는 walker 가 끝난 직후 asset 루프 전에 도는데, 이
구간이 관측 가능한 신호를 하나도 내지 않았다. 진행바는 walker 총계
(`0/12115`)를 표시한 채 멈춰 보이고 ndjson 로그는 0바이트로 남는다.
`tracing::info!` 은 나가지만 wire 이벤트가 아니라 사용자가 볼 산출물이
없다. 프로세스는 CPU 100% 로 정상 동작 중인데 밖에서는 hang 과 구별할 수
없고, 실제 도그푸딩에서 세 번 연속 Ctrl-C 로 죽였다.

`ingest_progress.v1` 에 세 이벤트를 추가한다 (additive — 기존 소비자는
모르는 kind 를 무시한다).

  sweep_started   { total }
  sweep_progress  { idx, total, path, removed }
  sweep_completed { checked, purged, ms }

`total` 은 `all_workspace_paths()` 에서 이번 스캔이 덮은 경로를 뺀 값이라
루프 진입 전에 확정 분모가 나온다 (이슈 제안 1). `removed` 는 "정말 없어서
문서를 지웠다" 와 "아직 디스크에 있어 그대로 뒀다" 를 가른다 — 수천 건을
훑고 하나도 안 지우는 sweep 이 있으므로, 지울 때만 움직이는 진행바는 바로
그 경우에 다시 멈춰 보인다.

`purged` 를 두 이벤트에서 다른 타입으로 쓰지 않으려고 sweep_progress 쪽은
`removed`(bool), sweep_completed 쪽은 `purged`(정수) 로 이름을 나눴다.
같은 wire 키가 두 타입을 갖는 건 소비자 입장에서 함정이다.

ndjson 로그에는 `purge { ts, doc_path }` 와 `sweep_summary { ts, checked,
purged, ms }` 를 추가한다 (이슈 제안 2 가 요청한 형태). 로그가 유일한 사후
기록이다 — tracing 은 stderr 로 흘러가고 남지 않는다.

CLI 는 sweep 을 asset 진행바와 별 phase 로 그린다. 후보 12k 를 훑는 일과
asset 12k 를 색인하는 일은 분모가 다른 별개의 작업이라 카운터를 공유하면
두 번째 구간이 처음부터 다시 시작하는 것처럼 보인다. 비-TTY 는 실제로
지운 것만 줄로 찍는다 — 검사한 후보마다 한 줄이면 그대로 둔 경로들이
run 의 진짜 출력을 덮는다.

실측 (문서 30건 색인 → 21건 삭제 → 재색인):

  ingest: sweeping 21 deleted-file candidates…
    purged doc11.md
    … (21줄)
  ingest: sweep complete (checked=21 purged=21 in 134ms)

`--json` 은 세 이벤트를 ingest_progress.v1 로 내보내고, ndjson 로그에는
purge 21줄 + sweep_summary 1줄이 남는다.

이슈가 참고로 적은 `reset --orphans-only` 는 그대로 뒀다. reset 에는 진행
채널 자체가 없어 sweep 하나를 위해 배선을 새로 깔아야 하는데, #229#230 이 머지된 지금 이 경로의 문서당 비용이 약 800배 떨어져 "몇 시간
무표시" 상황이 애초에 안 나온다.

곁다리 — clippy 게이트가 붉었다:

`cargo clippy --workspace --all-targets -- -D warnings` 가 main 에서
실패하고 있었다. 툴체인이 올라가면서 새 lint 둘(`question_mark`,
`manual_assert_eq`)이 기존 코드에 걸린 것이고 각각 한 줄이다. 방치하면 안
되는 이유를 이번에 겪었다 — `kebab-parse-code` 가 먼저 실패해 뒤 크레이트가
아예 컴파일되지 않았고, 그 그늘에 PR #235 에서 내가 넣은
`unnested_or_patterns` 위반이 숨어 있었다. 셋 다 여기서 고친다.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017c9JwQq8ZkGvYjpKXMiDhF
This commit is contained in:
2026-08-16 20:57:34 +09:00
parent 2cbac4d4e8
commit 1626a40a28
11 changed files with 666 additions and 55 deletions

View File

@@ -2,11 +2,17 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://kb.local/wire/v1/ingest_progress.schema.json",
"title": "IngestProgressEvent v1",
"description": "Streaming progress event emitted by `kebab ingest --json`. One event per line (line-delimited JSON). Discriminated by `kind`. The terminal events are `completed` and `aborted` — every ingest run ends with exactly one of them. The final stdout line of a `--json` ingest is still the existing `ingest_report.v1` for backwards compatibility; progress events stream above it.",
"description": "Streaming progress event emitted by `kebab ingest --json`. One event per line (line-delimited JSON). Discriminated by `kind`. The terminal events are `completed` and `aborted` — every ingest run ends with exactly one of them. The final stdout line of a `--json` ingest is still the existing `ingest_report.v1` for backwards compatibility; progress events stream above it. `sweep_*` events (v0.33.0) cover the deleted-file sweep that runs between the scan and the asset loop; before them that phase emitted nothing and a long sweep was indistinguishable from a hang (issue #228).",
"type": "object",
"required": ["schema_version", "kind", "ts"],
"required": [
"schema_version",
"kind",
"ts"
],
"properties": {
"schema_version": { "const": "ingest_progress.v1" },
"schema_version": {
"const": "ingest_progress.v1"
},
"kind": {
"type": "string",
"enum": [
@@ -21,53 +27,200 @@
"embed_batch_finished",
"pdf_ocr_started",
"pdf_ocr_finished",
"sweep_started",
"sweep_progress",
"sweep_completed",
"completed",
"aborted"
]
},
"ts": { "type": "string", "format": "date-time", "description": "RFC 3339 timestamp at the moment the event was emitted." },
"root": { "type": "string", "description": "scan_started: workspace root being walked." },
"total": { "type": "integer", "minimum": 0, "description": "scan_completed / asset_started / asset_finished: total assets discovered." },
"idx": { "type": "integer", "minimum": 1, "description": "asset_started / asset_finished: 1-based index of the current asset within the scan." },
"path": { "type": "string", "description": "asset_started: workspace-relative path of the asset being processed." },
"media": { "type": "string", "description": "asset_started: media kind label (e.g. `markdown`, `pdf`, `image`)." },
"result": {
"ts": {
"type": "string",
"enum": ["new", "updated", "skipped", "error"],
"format": "date-time",
"description": "RFC 3339 timestamp at the moment the event was emitted."
},
"root": {
"type": "string",
"description": "scan_started: workspace root being walked."
},
"total": {
"type": "integer",
"minimum": 0,
"description": "scan_completed / asset_started / asset_finished: total assets discovered. sweep_started / sweep_progress: total stored paths the deleted-file sweep will examine (the store's paths minus the ones this scan covered)."
},
"idx": {
"type": "integer",
"minimum": 1,
"description": "asset_started / asset_finished: 1-based index of the current asset within the scan. sweep_progress: 1-based index of the current sweep candidate."
},
"path": {
"type": "string",
"description": "asset_started: workspace-relative path of the asset being processed. sweep_progress: workspace path of the candidate examined."
},
"media": {
"type": "string",
"description": "asset_started: media kind label (e.g. `markdown`, `pdf`, `image`)."
},
"result": {
"type": "string",
"enum": [
"new",
"updated",
"skipped",
"error"
],
"description": "asset_finished: per-asset outcome (mirrors `ingest_report.v1.items[].kind`)."
},
"chunks": { "type": "integer", "minimum": 0, "description": "asset_finished / asset_chunked (v0.24.0): chunk count produced for this asset." },
"phase": { "type": "string", "enum": ["ocr", "caption", "embed"], "description": "asset_phase (v0.26.1): the slow internal phase the asset just entered. Short phases (parse/chunk/store) are not emitted." },
"model": { "type": ["string", "null"], "description": "asset_phase (v0.26.1): model performing the phase — vision LLM id for ocr/caption, embedder model_id for embed. null when the phase runs without a configured model." },
"parse_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): parse phase wall-clock (ms). Emitted by markdown / image / PDF paths." },
"chunk_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): chunk phase wall-clock (ms). Emitted by markdown / image / PDF paths." },
"expansion_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): retained for wire compatibility but always 0 — doc-side expansion was removed (HOTFIXES 2026-06-03)." },
"embed_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): embed + vector phase wall-clock (ms) — embedding, vector upsert, and stale-vector purge." },
"store_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.24.0, additive): SQLite persist phase wall-clock (ms) — put_asset/document/blocks/chunks only." },
"ocr_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.26.1, additive, default 0): image/PDF OCR phase wall-clock (ms). 0 on the markdown path (no OCR)." },
"caption_ms": { "type": "integer", "minimum": 0, "description": "asset_timings (v0.26.1, additive, default 0): image caption phase wall-clock (ms). 0 on markdown / PDF paths." },
"n_chunks": { "type": "integer", "minimum": 0, "description": "embed_batch_started / embed_batch_finished: chunks in this embedding batch." },
"ms": { "type": "integer", "minimum": 0, "description": "embed_batch_finished / pdf_ocr_finished: wall-clock duration (ms). pdf_ocr_finished skip path 의 의미는 mixed (DCTDecode 부재 시 0, engine 실패 시 latency-before-bail)." },
"chars": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished: char count of OCR result. Skip 시 0." },
"page": { "type": "integer", "minimum": 1, "description": "pdf_ocr_started / pdf_ocr_finished: 1-based PDF page number under OCR." },
"ocr_engine": { "type": "string", "description": "pdf_ocr_finished: engine_name (e.g. 'ollama-vision')." },
"skipped": { "type": "boolean", "description": "pdf_ocr_finished: true 일 시 OCR 미수행 (DCTDecode 부재 또는 engine 실패). chars=0 만으로는 skip 과 0-char result 구분 불가." },
"image_byte_size": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished (optional, v0.20.x): raster image byte size." },
"image_width": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished (optional, v0.20.x): raster image width px." },
"image_height": { "type": "integer", "minimum": 0, "description": "pdf_ocr_finished (optional, v0.20.x): raster image height px." },
"failure_reason": { "type": "string", "description": "pdf_ocr_finished (optional, v0.20.x): OCR failure reason. Present iff skipped=true due to engine error. Values: timeout | ocr_error | network_error | other." },
"counts": {
"chunks": {
"type": "integer",
"minimum": 0,
"description": "asset_finished / asset_chunked (v0.24.0): chunk count produced for this asset."
},
"phase": {
"type": "string",
"enum": [
"ocr",
"caption",
"embed"
],
"description": "asset_phase (v0.26.1): the slow internal phase the asset just entered. Short phases (parse/chunk/store) are not emitted."
},
"model": {
"type": [
"string",
"null"
],
"description": "asset_phase (v0.26.1): model performing the phase — vision LLM id for ocr/caption, embedder model_id for embed. null when the phase runs without a configured model."
},
"parse_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.24.0, additive): parse phase wall-clock (ms). Emitted by markdown / image / PDF paths."
},
"chunk_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.24.0, additive): chunk phase wall-clock (ms). Emitted by markdown / image / PDF paths."
},
"expansion_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.24.0, additive): retained for wire compatibility but always 0 — doc-side expansion was removed (HOTFIXES 2026-06-03)."
},
"embed_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.24.0, additive): embed + vector phase wall-clock (ms) — embedding, vector upsert, and stale-vector purge."
},
"store_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.24.0, additive): SQLite persist phase wall-clock (ms) — put_asset/document/blocks/chunks only."
},
"ocr_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.26.1, additive, default 0): image/PDF OCR phase wall-clock (ms). 0 on the markdown path (no OCR)."
},
"caption_ms": {
"type": "integer",
"minimum": 0,
"description": "asset_timings (v0.26.1, additive, default 0): image caption phase wall-clock (ms). 0 on markdown / PDF paths."
},
"n_chunks": {
"type": "integer",
"minimum": 0,
"description": "embed_batch_started / embed_batch_finished: chunks in this embedding batch."
},
"ms": {
"type": "integer",
"minimum": 0,
"description": "embed_batch_finished / pdf_ocr_finished: wall-clock duration (ms). pdf_ocr_finished skip path 의 의미는 mixed (DCTDecode 부재 시 0, engine 실패 시 latency-before-bail). sweep_completed: wall-clock of the whole sweep phase."
},
"chars": {
"type": "integer",
"minimum": 0,
"description": "pdf_ocr_finished: char count of OCR result. Skip 시 0."
},
"page": {
"type": "integer",
"minimum": 1,
"description": "pdf_ocr_started / pdf_ocr_finished: 1-based PDF page number under OCR."
},
"ocr_engine": {
"type": "string",
"description": "pdf_ocr_finished: engine_name (e.g. 'ollama-vision')."
},
"skipped": {
"type": "boolean",
"description": "pdf_ocr_finished: true 일 시 OCR 미수행 (DCTDecode 부재 또는 engine 실패). chars=0 만으로는 skip 과 0-char result 구분 불가."
},
"image_byte_size": {
"type": "integer",
"minimum": 0,
"description": "pdf_ocr_finished (optional, v0.20.x): raster image byte size."
},
"image_width": {
"type": "integer",
"minimum": 0,
"description": "pdf_ocr_finished (optional, v0.20.x): raster image width px."
},
"image_height": {
"type": "integer",
"minimum": 0,
"description": "pdf_ocr_finished (optional, v0.20.x): raster image height px."
},
"failure_reason": {
"type": "string",
"description": "pdf_ocr_finished (optional, v0.20.x): OCR failure reason. Present iff skipped=true due to engine error. Values: timeout | ocr_error | network_error | other."
},
"counts": {
"type": "object",
"description": "completed / aborted: aggregate counters at the moment the run ended (mirrors fields on `ingest_report.v1`).",
"properties": {
"scanned": { "type": "integer", "minimum": 0 },
"new": { "type": "integer", "minimum": 0 },
"updated": { "type": "integer", "minimum": 0 },
"skipped": { "type": "integer", "minimum": 0 },
"errors": { "type": "integer", "minimum": 0 },
"chunks_indexed": { "type": "integer", "minimum": 0 },
"embeddings_indexed": { "type": "integer", "minimum": 0 }
"scanned": {
"type": "integer",
"minimum": 0
},
"new": {
"type": "integer",
"minimum": 0
},
"updated": {
"type": "integer",
"minimum": 0
},
"skipped": {
"type": "integer",
"minimum": 0
},
"errors": {
"type": "integer",
"minimum": 0
},
"chunks_indexed": {
"type": "integer",
"minimum": 0
},
"embeddings_indexed": {
"type": "integer",
"minimum": 0
}
}
},
"removed": {
"type": "boolean",
"description": "sweep_progress: true when the file was truly absent and its document was purged; false when it is still on disk (left untouched) or the purge failed."
},
"checked": {
"type": "integer",
"minimum": 0,
"description": "sweep_completed: sweep candidates examined."
},
"purged": {
"type": "integer",
"minimum": 0,
"description": "sweep_completed: documents removed because their source file is gone."
}
}
}