Merge pull request 'feat(tag-vocab): #3 태그 vocab — prompt + telemetry (v0.2.3 6/7)' (#18) from feat/v023-tag-vocab into main
Reviewed-on: #18
This commit was merged in pull request #18.
This commit is contained in:
1091
docs/superpowers/plans/2026-05-02-v023-tag-vocab.md
Normal file
1091
docs/superpowers/plans/2026-05-02-v023-tag-vocab.md
Normal file
File diff suppressed because it is too large
Load Diff
@@ -154,7 +154,7 @@ v0.2.2 ────────[ dogfood 동결, 병렬 진행 ]─────
|
||||
|
||||
**Out:** per-note retry 버튼 (NoteCard), failed reason 별 차등 정책, retry progress UI, retry rate-limit
|
||||
|
||||
### #3 태그 vocab (6번)
|
||||
### #3 태그 vocab (6번) ✓ 완료
|
||||
|
||||
**In:**
|
||||
- `NoteRepository.getTopUsedTags(N=20)`:
|
||||
|
||||
255
docs/superpowers/specs/2026-05-02-v023-tag-vocab-design.md
Normal file
255
docs/superpowers/specs/2026-05-02-v023-tag-vocab-design.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# v0.2.3 #3 태그 vocab — Design Spec
|
||||
|
||||
> 작성: 2026-05-02 · v0.2.3 dogfood feedback roadmap §3 #3 (6번째 cut)
|
||||
|
||||
## 1. Goal
|
||||
|
||||
기존 `tags` 테이블의 자주 쓰인 태그들을 AI prompt 에 vocabulary 로 주입해, AI 가 의미 일치 시 새 태그 생성 대신 기존 태그를 재사용하도록 유도. 효과는 `tag_vocab_hit` / `tag_vocab_miss` telemetry 로 측정.
|
||||
|
||||
## 2. Decisions (mini-brainstorm 합의)
|
||||
|
||||
| # | 질문 | 선택 | 이유 |
|
||||
|---|---|---|---|
|
||||
| Q1 | vocab pool 범위 | **C** AI+user 통합 + kebab-case 필터 | 사용자가 형식 맞춰 단 태그도 재사용 가치 있음, 단 형식 안 맞는 한글/공백 태그는 prompt 오염 |
|
||||
| Q2 | telemetry emit 단위 | **A** 태그별 (per-tag hit/miss) | roadmap §3 #3 합의 시그니처 + 기존 누적 카운터 통계 모델과 정합 |
|
||||
| Q3 | prompt 강제력 강도 | **B** "Prefer" (우선) | "MUST" 는 semantic mismatch 시 false hit, "For reference" 는 효과 미미; "Prefer" 는 우선순위 신호 + escape hatch 보장 |
|
||||
| Q4 | 기존 노트 재처리 | **A** 자연 진화 (X) | invariant (user-edited 결과 보호) 와 합치, 새 노트만으로 hit/miss 충분 수집, B 는 사용자 결과 변경 |
|
||||
|
||||
## 3. Architecture & data flow
|
||||
|
||||
```
|
||||
AiWorker.processJob()
|
||||
├─ const vocab = repo.getTopUsedTags(20) ← SQL fetch (kebab-case 필터)
|
||||
├─ provider.generate({ ..., vocab }) ← 새 input 필드
|
||||
│ └─ LocalOllamaProvider.generate()
|
||||
│ └─ buildPrompt(rawText, todayKst, candidates, vocab)
|
||||
│ └─ vocab.length > 0 시 prompt 라인 추가
|
||||
├─ AI response (tags: ['design', 'meeting', ...])
|
||||
├─ repo.updateAiResult(...) ← 기존 흐름, tag insert
|
||||
└─ for tag of res.tags: ← per-tag hit/miss 분류
|
||||
if vocabSet.has(tag):
|
||||
tagId = repo.getTagIdByName(tag) ← insert 후 보장
|
||||
emit tag_vocab_hit { tagId, vocabSize }
|
||||
else:
|
||||
emit tag_vocab_miss { vocabSize }
|
||||
```
|
||||
|
||||
### 3.1 Invariants
|
||||
|
||||
1. **매 generate 마다 SQL fetch** — vocab 캐싱/invalidation 안 함 (out of scope)
|
||||
2. **vocab 빈 케이스 (N=0)** → prompt 라인 자체 생략, AI 자유롭게 새 태그 생성
|
||||
3. **tagId** 는 hit 시 db tag id (`getTagIdByName` lookup, `updateAiResult` 후 호출이라 insert 보장)
|
||||
4. **PROMPT_VERSION 3 → 4** (marker only, retry 트리거 X)
|
||||
5. **vocab snapshot 동결** — 같은 generate call 의 `vocab` 배열로 hit/miss 판정. 처리 중 다른 노트가 새 태그 추가해도 이번 노트 분류엔 영향 X
|
||||
6. **emit 순서** — `updateAiResult` 후 emit (tagId 확보 보장)
|
||||
|
||||
## 4. Components
|
||||
|
||||
### 4.1 `NoteRepository`
|
||||
|
||||
#### `getTopUsedTags(limit = 20): string[]`
|
||||
|
||||
```sql
|
||||
SELECT t.name, COUNT(*) c
|
||||
FROM tags t
|
||||
JOIN note_tags nt ON nt.tag_id = t.id
|
||||
JOIN notes n ON n.id = nt.note_id
|
||||
WHERE n.deleted_at IS NULL
|
||||
GROUP BY t.id
|
||||
ORDER BY c DESC, t.id ASC
|
||||
LIMIT ?
|
||||
```
|
||||
|
||||
JS-side 후처리:
|
||||
```typescript
|
||||
return rows
|
||||
.map((r) => r.name)
|
||||
.filter((n) => /^[a-z0-9-]+$/.test(n));
|
||||
```
|
||||
|
||||
- `source` 무시 (AI+user 통합 — Q1=C)
|
||||
- `t.id ASC` tiebreaker (deterministic)
|
||||
- regex 필터로 한글/공백/대문자 태그 제외
|
||||
|
||||
#### `getTagIdByName(name: string): number | null`
|
||||
|
||||
```sql
|
||||
SELECT id FROM tags WHERE name = ? COLLATE NOCASE LIMIT 1
|
||||
```
|
||||
|
||||
대소문자 무시 (tag table `name COLLATE NOCASE` 와 정합).
|
||||
|
||||
### 4.2 `prompt.ts`
|
||||
|
||||
```typescript
|
||||
export const PROMPT_VERSION = 4; // bump from 3
|
||||
|
||||
export function buildPrompt(
|
||||
rawText: string,
|
||||
todayKst: string,
|
||||
candidates: ParseResult[] = [],
|
||||
vocab: string[] = []
|
||||
): string {
|
||||
const candidateBlock = ...; // 기존 로직 유지
|
||||
const vocabBlock = vocab.length > 0
|
||||
? `\nExisting vocabulary tags (most-used first): ${vocab.join(', ')}\nPrefer reusing a vocabulary tag when the meaning matches; create new tags only when the meaning is genuinely new.\n`
|
||||
: '';
|
||||
return `... ${candidateBlock} ${vocabBlock} ...`;
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 `InferenceProvider` + `LocalOllamaProvider`
|
||||
|
||||
```typescript
|
||||
export interface GenerateInput {
|
||||
text: string;
|
||||
todayKst: string;
|
||||
dueDateCandidates: ParseResult[];
|
||||
vocab?: string[]; // optional, 미전달 시 buildPrompt 가 빈 배열 처리
|
||||
}
|
||||
```
|
||||
|
||||
`LocalOllamaProvider.generate()` 가 `buildPrompt(text, todayKst, candidates, input.vocab ?? [])` 호출.
|
||||
|
||||
### 4.4 `AiWorker.processJob`
|
||||
|
||||
generate 호출 직전:
|
||||
```typescript
|
||||
const vocab = this.repo.getTopUsedTags(20);
|
||||
const res = await this.provider.generate({
|
||||
text: note.rawText,
|
||||
todayKst: todayIso,
|
||||
dueDateCandidates: candidates,
|
||||
vocab
|
||||
});
|
||||
```
|
||||
|
||||
`updateAiResult` 후 emit 루프:
|
||||
```typescript
|
||||
const vocabSet = new Set(vocab);
|
||||
for (const tagName of res.tags) {
|
||||
if (vocabSet.has(tagName)) {
|
||||
const tagId = this.repo.getTagIdByName(tagName);
|
||||
if (tagId !== null && this.telemetry) {
|
||||
await this.telemetry.emit({
|
||||
kind: 'tag_vocab_hit',
|
||||
payload: { tagId, vocabSize: vocab.length }
|
||||
}).catch(() => {});
|
||||
}
|
||||
} else if (this.telemetry) {
|
||||
await this.telemetry.emit({
|
||||
kind: 'tag_vocab_miss',
|
||||
payload: { vocabSize: vocab.length }
|
||||
}).catch(() => {});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.5 `telemetryEvents.ts` — zod schema
|
||||
|
||||
```typescript
|
||||
const TagVocabHitPayload = z.object({
|
||||
tagId: z.number().int().positive(),
|
||||
vocabSize: z.number().int().nonnegative()
|
||||
}).strict();
|
||||
|
||||
const TagVocabMissPayload = z.object({
|
||||
vocabSize: z.number().int().nonnegative()
|
||||
}).strict();
|
||||
```
|
||||
|
||||
`TelemetryEventSchema` discriminatedUnion 13 → **15** entries.
|
||||
|
||||
### 4.6 `telemetryStats.ts` — 누적
|
||||
|
||||
- `DailyRow` 에 `tag_vocab_hit: number`, `tag_vocab_miss: number` 추가
|
||||
- accumulator 분기 2개
|
||||
- table 컬럼 2개 추가
|
||||
- summary 라인:
|
||||
```
|
||||
- 태그 vocab: hit/miss = {N}/{M} (적중률 {X}%)
|
||||
```
|
||||
N+M=0 시 `(데이터 없음)` 표기
|
||||
|
||||
### 4.7 `TelemetryService.EmitInput` union 확장 (15 entries)
|
||||
|
||||
### 4.8 `AiWorker.AiTelemetryEmitter` interface 확장
|
||||
|
||||
```typescript
|
||||
export interface AiTelemetryEmitter {
|
||||
emit(input:
|
||||
| { kind: 'ai_succeeded'; payload: ... }
|
||||
| { kind: 'ai_failed'; payload: ... }
|
||||
| { kind: 'tag_vocab_hit'; payload: { tagId: number; vocabSize: number } }
|
||||
| { kind: 'tag_vocab_miss'; payload: { vocabSize: number } }
|
||||
): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Privacy invariant
|
||||
|
||||
- `tag_vocab_hit.payload.tagId` — 숫자 id 만, 태그 이름 X
|
||||
- `tag_vocab_miss.payload` — `vocabSize` 만 (tagId 없음)
|
||||
- prompt 본문에 vocab 이름 들어가지만 **prompt 는 telemetry 가 아님** (모델 컨텍스트, local Ollama 머신 내부에서만 처리)
|
||||
- `.strict()` zod 가드 + extra field 거부 테스트로 invariant 보호
|
||||
|
||||
## 6. Tests (≥19개)
|
||||
|
||||
### NoteRepository.test.ts (7)
|
||||
1. 빈 db → `[]`
|
||||
2. 정렬 (count desc, id asc tiebreaker)
|
||||
3. kebab-case 필터 — 한글/공백/대문자 태그 제외
|
||||
4. AI+user source 통합 카운트
|
||||
5. `deleted_at IS NULL` 필터
|
||||
6. LIMIT 적용 (>20 시 잘림)
|
||||
7. `getTagIdByName` — 존재 시 id, 없으면 null
|
||||
|
||||
### prompt.test.ts (4)
|
||||
8. `PROMPT_VERSION === 4`
|
||||
9. vocab=[] → 라인 자체 생략
|
||||
10. vocab 1+ → "Prefer reusing..." 문구 + comma-separated 리스트
|
||||
11. vocab 라인 위치 (candidate block 뒤, JSON rules 앞)
|
||||
|
||||
### AiWorker.test.ts (4)
|
||||
12. vocab fetch + provider.generate 에 vocab 전달 + hit emit
|
||||
13. miss emit (vocab 밖의 tag), vocabSize 정확
|
||||
14. vocab=[] 시 모든 응답 태그 miss
|
||||
15. 응답 태그 3개 → 3개 emit (per-tag 검증)
|
||||
|
||||
### telemetryEvents.test.ts (3)
|
||||
16. `tag_vocab_hit` valid parse
|
||||
17. `tag_vocab_hit` extra field 거부 (privacy)
|
||||
18. `tag_vocab_miss` valid parse, tagId 필드 없음
|
||||
|
||||
### telemetryStats.test.ts (1)
|
||||
19. hit 5 + miss 3 → daily row + summary "적중률 62.5%"
|
||||
|
||||
기존 단위 363 + **19** = **382** 예상. Q3 phrasing 변경으로 LocalOllamaProvider 기존 테스트 일부 string assertion 수정 가능 (±5).
|
||||
|
||||
## 7. Out of scope
|
||||
|
||||
(roadmap §3 #3 + 본 cut 결정)
|
||||
|
||||
- 임베딩 유사도 dedup ("회의" ↔ "meeting" semantic 매핑)
|
||||
- 사용자 controlled vocabulary 화이트리스트
|
||||
- 자동 normalize ("회의" ↔ "미팅")
|
||||
- top-N 튜닝 (N=20 hardcoded)
|
||||
- vocab cache invalidation 정책 (매번 SQL fetch)
|
||||
- vocab 시간 범위 필터 (최근 N일 → 전체 사용)
|
||||
- 기존 `ai_status='done'` 노트 일괄 재처리 (Q4=A 자연 진화)
|
||||
- 명시적 "AI 결과 재처리" trigger UI (v0.2.4 backlog)
|
||||
- `promptVersion` 을 telemetry payload 에 포함 (v0.2.4 검토 — 단일 버전 cut 에선 무의미)
|
||||
- `idx_note_tags_tag_id` 인덱스 추가 (현재 dogfood 규모에선 불필요, v0.2.4 검토)
|
||||
|
||||
## 8. Gates (roadmap §3.1 공통)
|
||||
|
||||
- typecheck 0
|
||||
- 단위 363 → 382 (+19), 모두 통과
|
||||
- e2e 1/1
|
||||
- 새 SQL: `getTopUsedTags` (3-table JOIN) + `getTagIdByName` (single-table) — 인덱스 영향 dogfood 규모에서 무시
|
||||
|
||||
## 9. Roadmap relation
|
||||
|
||||
- v0.2.3 dogfood feedback #3 (6번째 cut)
|
||||
- 다음 cut: #6 리마인드 1 spike (7번째, 마지막)
|
||||
- v0.2.4 후속: top-N 튜닝, controlled vocabulary, normalize, embeddings dedup
|
||||
@@ -32,6 +32,8 @@ export interface AiTelemetryEmitter {
|
||||
emit(input:
|
||||
| { kind: 'ai_succeeded'; payload: { noteId: string; durationMs: number; attempts: number } }
|
||||
| { kind: 'ai_failed'; payload: { noteId: string; reason: 'unreachable' | 'schema' | 'timeout' | 'other'; attempts: number } }
|
||||
| { kind: 'tag_vocab_hit'; payload: { tagId: number; vocabSize: number } }
|
||||
| { kind: 'tag_vocab_miss'; payload: { vocabSize: number } }
|
||||
): Promise<void>;
|
||||
}
|
||||
|
||||
@@ -132,10 +134,12 @@ export class AiWorker {
|
||||
const todayDate = todayKstAsDate(nowDate);
|
||||
const todayIso = todayKstAsIso(nowDate);
|
||||
const candidates = parseAllCandidates(note.rawText, todayDate);
|
||||
const vocab = this.repo.getTopUsedTags(20);
|
||||
const res = await this.provider.generate({
|
||||
text: note.rawText,
|
||||
todayKst: todayIso,
|
||||
dueDateCandidates: candidates
|
||||
dueDateCandidates: candidates,
|
||||
vocab
|
||||
});
|
||||
// AI primary: AI's dueDate is final (no rule merge)
|
||||
this.repo.updateAiResult(job.noteId, {
|
||||
@@ -161,6 +165,25 @@ export class AiWorker {
|
||||
attempts: attempt + 1
|
||||
}
|
||||
}).catch(() => {});
|
||||
// v0.2.3 #3 — per-tag vocab hit/miss 분류 (updateAiResult 후 → tagId 보장)
|
||||
// dedup: AI 응답에 같은 태그 중복 가능 — INSERT OR IGNORE 와 정합한 1-emit/태그 보장
|
||||
const vocabSet = new Set(vocab);
|
||||
for (const tagName of new Set(res.tags)) {
|
||||
if (vocabSet.has(tagName)) {
|
||||
const tagId = this.repo.getTagIdByName(tagName);
|
||||
if (tagId !== null) {
|
||||
await this.telemetry.emit({
|
||||
kind: 'tag_vocab_hit',
|
||||
payload: { tagId, vocabSize: vocab.length }
|
||||
}).catch(() => {});
|
||||
}
|
||||
} else {
|
||||
await this.telemetry.emit({
|
||||
kind: 'tag_vocab_miss',
|
||||
payload: { vocabSize: vocab.length }
|
||||
}).catch(() => {});
|
||||
}
|
||||
}
|
||||
}
|
||||
this.emit(job.noteId);
|
||||
return;
|
||||
|
||||
@@ -5,6 +5,7 @@ export interface GenerateInput {
|
||||
text: string;
|
||||
todayKst: string; // ISO YYYY-MM-DD in KST
|
||||
dueDateCandidates: ParseResult[];
|
||||
vocab?: string[]; // v0.2.3 #3 — top-N kebab-case 태그. 미전달 시 빈 배열로 처리.
|
||||
}
|
||||
|
||||
export interface HealthResult { ok: boolean; model?: string; reason?: string; }
|
||||
|
||||
@@ -37,7 +37,7 @@ export class LocalOllamaProvider implements InferenceProvider {
|
||||
headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: this.model,
|
||||
prompt: buildPrompt(input.text, input.todayKst, input.dueDateCandidates),
|
||||
prompt: buildPrompt(input.text, input.todayKst, input.dueDateCandidates, input.vocab ?? []),
|
||||
format: 'json',
|
||||
stream: false,
|
||||
options: { temperature: this.temperature, num_predict: this.numPredict }
|
||||
|
||||
@@ -1,21 +1,27 @@
|
||||
import type { ParseResult } from '../services/dueDateParser.js';
|
||||
|
||||
export const PROMPT_VERSION = 3;
|
||||
export const PROMPT_VERSION = 4;
|
||||
|
||||
export function buildPrompt(
|
||||
rawText: string,
|
||||
todayKst: string,
|
||||
candidates: ParseResult[] = []
|
||||
candidates: ParseResult[] = [],
|
||||
vocab: string[] = []
|
||||
): string {
|
||||
const candidateBlock = candidates.length > 0
|
||||
? `\nDate candidates extracted by a Korean rule parser (these are HINTS — you decide which is correct, or pick null):
|
||||
${candidates.map((c, i) => ` ${i + 1}. ${c.iso ?? '(ambiguous)'} — matched token: "${c.matchedToken ?? '?'}" (confidence: ${c.confidence ?? 'low'})`).join('\n')}\n`
|
||||
: '';
|
||||
|
||||
const vocabBlock = vocab.length > 0
|
||||
? `\nExisting vocabulary tags (most-used first): ${vocab.join(', ')}\nPrefer reusing a vocabulary tag when the meaning matches; create new tags only when the meaning is genuinely new.\n`
|
||||
: '';
|
||||
|
||||
// candidateBlock & vocabBlock are self-delimited with leading/trailing \n
|
||||
return `You organize raw personal notes into structured metadata.
|
||||
|
||||
Today's date in Korea Standard Time (KST): ${todayKst}
|
||||
${candidateBlock}
|
||||
${candidateBlock}${vocabBlock}
|
||||
Input note (raw text, may be fragmented, any language):
|
||||
---
|
||||
${rawText}
|
||||
|
||||
@@ -40,6 +40,8 @@ export interface ImportNoteResult {
|
||||
status: ImportNoteStatus;
|
||||
}
|
||||
|
||||
const KEBAB_CASE_RE = /^[a-z0-9-]+$/;
|
||||
|
||||
export class NoteRepository {
|
||||
constructor(private db: Database.Database) {}
|
||||
|
||||
@@ -215,6 +217,44 @@ export class NoteRepository {
|
||||
.run(nextRunAt, lastError.slice(0, 500), noteId);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.2.3 #3 — AI prompt 의 vocabulary 후보. 사용 빈도 높은 태그 top-N.
|
||||
* source 무시 (AI+user 통합), kebab-case 통과한 것만 (한글/공백/대문자 제외).
|
||||
* deleted_at IS NULL 만 (휴지통 노트 태그 제외).
|
||||
*
|
||||
* Note: LIMIT 가 SQL 단계에서 먼저 적용된 후 regex 필터링이 후처리 됨.
|
||||
* 따라서 반환 배열 length 가 limit 보다 작을 수 있음 (top-N 안에 비-kebab-case
|
||||
* 태그가 섞여 있을 때). v0.2.3 dogfood 규모에서는 실용적 영향 없음.
|
||||
*/
|
||||
getTopUsedTags(limit = 20): string[] {
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT t.name, COUNT(*) AS c
|
||||
FROM tags t
|
||||
JOIN note_tags nt ON nt.tag_id = t.id
|
||||
JOIN notes n ON n.id = nt.note_id
|
||||
WHERE n.deleted_at IS NULL
|
||||
GROUP BY t.id
|
||||
ORDER BY c DESC, t.id ASC
|
||||
LIMIT ?`
|
||||
)
|
||||
.all(limit) as Array<{ name: string; c: number }>;
|
||||
return rows
|
||||
.map((r) => r.name)
|
||||
.filter((n) => KEBAB_CASE_RE.test(n));
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.2.3 #3 — vocab hit telemetry 의 tagId 확보용. updateAiResult 후 호출 보장.
|
||||
* tags.name COLLATE NOCASE 라 case-insensitive lookup.
|
||||
*/
|
||||
getTagIdByName(name: string): number | null {
|
||||
const row = this.db
|
||||
.prepare(`SELECT id FROM tags WHERE name = ? COLLATE NOCASE LIMIT 1`)
|
||||
.get(name) as { id: number } | undefined;
|
||||
return row ? row.id : null;
|
||||
}
|
||||
|
||||
updateUserAiFields(
|
||||
id: string,
|
||||
fields: { title?: string; summary?: string; tags?: string[] }
|
||||
|
||||
@@ -28,7 +28,9 @@ export type EmitInput =
|
||||
| { kind: 'ollama_unreachable'; payload: { reason: string } }
|
||||
| { kind: 'ollama_recovered'; payload: { downtimeMs: number } }
|
||||
| { kind: 'ollama_recheck_manual'; payload: Record<string, never> }
|
||||
| { kind: 'ai_retry_manual'; payload: { failedCount: number } };
|
||||
| { kind: 'ai_retry_manual'; payload: { failedCount: number } }
|
||||
| { kind: 'tag_vocab_hit'; payload: { tagId: number; vocabSize: number } }
|
||||
| { kind: 'tag_vocab_miss'; payload: { vocabSize: number } };
|
||||
|
||||
export class TelemetryService {
|
||||
constructor(
|
||||
|
||||
@@ -50,6 +50,15 @@ const AiRetryManualPayload = z.object({
|
||||
failedCount: z.number().int().positive()
|
||||
}).strict();
|
||||
|
||||
const TagVocabHitPayload = z.object({
|
||||
tagId: z.number().int().positive(),
|
||||
vocabSize: z.number().int().nonnegative()
|
||||
}).strict();
|
||||
|
||||
const TagVocabMissPayload = z.object({
|
||||
vocabSize: z.number().int().nonnegative()
|
||||
}).strict();
|
||||
|
||||
export const TelemetryEventSchema = z.discriminatedUnion('kind', [
|
||||
z.object({ ts: z.string(), kind: z.literal('capture'), payload: CapturePayload }).strict(),
|
||||
z.object({ ts: z.string(), kind: z.literal('ai_succeeded'), payload: AiSucceededPayload }).strict(),
|
||||
@@ -63,7 +72,9 @@ export const TelemetryEventSchema = z.discriminatedUnion('kind', [
|
||||
z.object({ ts: z.string(), kind: z.literal('ollama_unreachable'), payload: OllamaUnreachablePayload }).strict(),
|
||||
z.object({ ts: z.string(), kind: z.literal('ollama_recovered'), payload: OllamaRecoveredPayload }).strict(),
|
||||
z.object({ ts: z.string(), kind: z.literal('ollama_recheck_manual'), payload: EmptyPayload }).strict(),
|
||||
z.object({ ts: z.string(), kind: z.literal('ai_retry_manual'), payload: AiRetryManualPayload }).strict()
|
||||
z.object({ ts: z.string(), kind: z.literal('ai_retry_manual'), payload: AiRetryManualPayload }).strict(),
|
||||
z.object({ ts: z.string(), kind: z.literal('tag_vocab_hit'), payload: TagVocabHitPayload }).strict(),
|
||||
z.object({ ts: z.string(), kind: z.literal('tag_vocab_miss'), payload: TagVocabMissPayload }).strict()
|
||||
]);
|
||||
|
||||
export type TelemetryEvent = z.infer<typeof TelemetryEventSchema>;
|
||||
|
||||
@@ -24,6 +24,8 @@ interface DailyRow {
|
||||
ollama_recovered: number;
|
||||
ollama_recheck_manual: number;
|
||||
ai_retry_manual: number;
|
||||
tag_vocab_hit: number;
|
||||
tag_vocab_miss: number;
|
||||
}
|
||||
|
||||
export interface StatsResult {
|
||||
@@ -47,6 +49,8 @@ export function aggregateStats(events: TelemetryEvent[], generatedAt: Date): Sta
|
||||
let ollamaRecheckManualCount = 0;
|
||||
let aiRetryManualCount = 0;
|
||||
let aiRetryManualFailedSum = 0;
|
||||
let tagVocabHitCount = 0;
|
||||
let tagVocabMissCount = 0;
|
||||
for (const ev of events) {
|
||||
const day = kstDate(ev.ts);
|
||||
let row = byDay.get(day);
|
||||
@@ -57,7 +61,8 @@ export function aggregateStats(events: TelemetryEvent[], generatedAt: Date): Sta
|
||||
trash: 0, restore: 0, permanent_delete: 0, empty_trash: 0,
|
||||
expired_banner_shown: 0, expired_batch_trash: 0,
|
||||
ollama_unreachable: 0, ollama_recovered: 0, ollama_recheck_manual: 0,
|
||||
ai_retry_manual: 0
|
||||
ai_retry_manual: 0,
|
||||
tag_vocab_hit: 0, tag_vocab_miss: 0
|
||||
};
|
||||
byDay.set(day, row);
|
||||
}
|
||||
@@ -99,6 +104,12 @@ export function aggregateStats(events: TelemetryEvent[], generatedAt: Date): Sta
|
||||
row.ai_retry_manual += 1;
|
||||
aiRetryManualCount += 1;
|
||||
aiRetryManualFailedSum += ev.payload.failedCount;
|
||||
} else if (ev.kind === 'tag_vocab_hit') {
|
||||
row.tag_vocab_hit += 1;
|
||||
tagVocabHitCount += 1;
|
||||
} else if (ev.kind === 'tag_vocab_miss') {
|
||||
row.tag_vocab_miss += 1;
|
||||
tagVocabMissCount += 1;
|
||||
}
|
||||
}
|
||||
const days = Array.from(byDay.values()).sort((a, b) => a.date.localeCompare(b.date));
|
||||
@@ -115,6 +126,10 @@ export function aggregateStats(events: TelemetryEvent[], generatedAt: Date): Sta
|
||||
? 'N/A'
|
||||
: `${Math.round(ollamaDowntimeSum / ollamaRecoveredCount)}`;
|
||||
const totalUnreachable = days.reduce((s, r) => s + r.ollama_unreachable, 0);
|
||||
const tagVocabTotal = tagVocabHitCount + tagVocabMissCount;
|
||||
const tagVocabSummary = tagVocabTotal === 0
|
||||
? '(데이터 없음)'
|
||||
: `hit/miss = ${tagVocabHitCount}/${tagVocabMissCount} (적중률 ${(tagVocabHitCount / tagVocabTotal * 100).toFixed(1)}%)`;
|
||||
const lines: string[] = [];
|
||||
lines.push('# Inkling Telemetry Stats');
|
||||
lines.push('');
|
||||
@@ -123,10 +138,10 @@ export function aggregateStats(events: TelemetryEvent[], generatedAt: Date): Sta
|
||||
lines.push('');
|
||||
lines.push('## 일자별 카운트');
|
||||
lines.push('');
|
||||
lines.push('| 일자 | capture | ai_succeeded | ai_failed | trash | restore | permanent_delete | empty_trash | expired_banner_shown | expired_batch_trash | ollama_unreachable | ollama_recovered | ollama_recheck_manual | ai_retry_manual |');
|
||||
lines.push('|------|---------|--------------|-----------|-------|---------|------------------|-------------|----------------------|---------------------|--------------------|------------------|----------------------|-----------------|');
|
||||
lines.push('| 일자 | capture | ai_succeeded | ai_failed | trash | restore | permanent_delete | empty_trash | expired_banner_shown | expired_batch_trash | ollama_unreachable | ollama_recovered | ollama_recheck_manual | ai_retry_manual | tag_vocab_hit | tag_vocab_miss |');
|
||||
lines.push('|------|---------|--------------|-----------|-------|---------|------------------|-------------|----------------------|---------------------|--------------------|------------------|----------------------|-----------------|---------------|----------------|');
|
||||
for (const row of days) {
|
||||
lines.push(`| ${row.date} | ${row.capture} | ${row.ai_succeeded} | ${row.ai_failed} | ${row.trash} | ${row.restore} | ${row.permanent_delete} | ${row.empty_trash} | ${row.expired_banner_shown} | ${row.expired_batch_trash} | ${row.ollama_unreachable} | ${row.ollama_recovered} | ${row.ollama_recheck_manual} | ${row.ai_retry_manual} |`);
|
||||
lines.push(`| ${row.date} | ${row.capture} | ${row.ai_succeeded} | ${row.ai_failed} | ${row.trash} | ${row.restore} | ${row.permanent_delete} | ${row.empty_trash} | ${row.expired_banner_shown} | ${row.expired_batch_trash} | ${row.ollama_unreachable} | ${row.ollama_recovered} | ${row.ollama_recheck_manual} | ${row.ai_retry_manual} | ${row.tag_vocab_hit} | ${row.tag_vocab_miss} |`);
|
||||
}
|
||||
lines.push('');
|
||||
lines.push('## 핵심 ratio');
|
||||
@@ -139,6 +154,7 @@ export function aggregateStats(events: TelemetryEvent[], generatedAt: Date): Sta
|
||||
lines.push(`- 평균 downtimeMs (recovered): ${avgDowntime}`);
|
||||
lines.push(`- 수동 recheck 사용량: ${ollamaRecheckManualCount}건`);
|
||||
lines.push(`- AI 수동 재시도: ${aiRetryManualCount}회 / 누적 ${aiRetryManualFailedSum}건`);
|
||||
lines.push(`- 태그 vocab: ${tagVocabSummary}`);
|
||||
lines.push('');
|
||||
return { md: lines.join('\n'), eventCount };
|
||||
}
|
||||
|
||||
@@ -3,9 +3,12 @@ import Database from 'better-sqlite3';
|
||||
import { runMigrations } from '@main/db/migrations/index.js';
|
||||
import { NoteRepository } from '@main/repository/NoteRepository.js';
|
||||
import { AiWorker } from '@main/ai/AiWorker.js';
|
||||
import type { AiTelemetryEmitter } from '@main/ai/AiWorker.js';
|
||||
import type { InferenceProvider } from '@main/ai/InferenceProvider.js';
|
||||
import type { AiResponse } from '@main/ai/schema.js';
|
||||
|
||||
type EmittedEvent = { kind: string; payload: unknown };
|
||||
|
||||
function makeProvider(overrides: Partial<InferenceProvider> = {}): InferenceProvider {
|
||||
return {
|
||||
name: 'mock',
|
||||
@@ -197,10 +200,10 @@ describe('AiWorker', () => {
|
||||
describe('AiWorker telemetry emit', () => {
|
||||
let db: Database.Database;
|
||||
let repo: NoteRepository;
|
||||
let events: Array<{ kind: string; payload: { noteId: string; durationMs?: number; reason?: string; attempts: number } }>;
|
||||
const collectingTelemetry = {
|
||||
emit: async (ev: { kind: string; payload: { noteId: string; durationMs?: number; reason?: string; attempts: number } }) => {
|
||||
events.push(ev);
|
||||
let events: Array<{ kind: string; payload: { noteId?: string; durationMs?: number; reason?: string; attempts?: number; tagId?: number; vocabSize?: number } }>;
|
||||
const collectingTelemetry: AiTelemetryEmitter = {
|
||||
emit: async (ev) => {
|
||||
events.push(ev as typeof events[number]);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -420,3 +423,138 @@ describe('AiWorker — unreachable/timeout infinite retry (v0.2.3 #2)', () => {
|
||||
expect((w as any).unreachableBackoffStep).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('AiWorker — vocab fetch + per-tag hit/miss (v0.2.3 #3 T7)', () => {
|
||||
let db: Database.Database;
|
||||
let repo: NoteRepository;
|
||||
|
||||
beforeEach(() => {
|
||||
db = new Database(':memory:');
|
||||
runMigrations(db);
|
||||
repo = new NoteRepository(db);
|
||||
});
|
||||
|
||||
it('fetches vocab and passes to provider.generate', async () => {
|
||||
// Pre-seed 1 note with tag 'design' so vocab non-empty
|
||||
const seed = repo.create({ rawText: 'seed' }).id;
|
||||
repo.updateAiResult(seed, { title: 't', summary: 'a\nb\nc', tags: ['design'], provider: 'p' });
|
||||
|
||||
const { id } = repo.create({ rawText: 'x' });
|
||||
const generateMock = vi.fn(async () => ({
|
||||
title: '제목', summary: 'a\nb\nc', tags: ['design'], dueDate: null
|
||||
}));
|
||||
const w = new AiWorker(repo, makeProvider({ generate: generateMock }), {
|
||||
backoffsMs: [0, 0, 0]
|
||||
});
|
||||
await w.enqueue(id);
|
||||
await w.drain();
|
||||
expect(generateMock).toHaveBeenCalledWith(expect.objectContaining({
|
||||
vocab: expect.arrayContaining(['design'])
|
||||
}));
|
||||
});
|
||||
|
||||
it('emits tag_vocab_hit for vocab tags + tag_vocab_miss for new tags', async () => {
|
||||
// Pre-seed: 'design' in vocab
|
||||
const seed = repo.create({ rawText: 'seed' }).id;
|
||||
repo.updateAiResult(seed, { title: 't', summary: 'a\nb\nc', tags: ['design'], provider: 'p' });
|
||||
|
||||
const { id } = repo.create({ rawText: 'x' });
|
||||
const provider = makeProvider({
|
||||
generate: vi.fn(async () => ({
|
||||
title: 't', summary: 'a\nb\nc',
|
||||
tags: ['design', 'newtag'], // 1 hit + 1 miss
|
||||
dueDate: null
|
||||
}))
|
||||
});
|
||||
const emits: EmittedEvent[] = [];
|
||||
const w = new AiWorker(repo, provider, {
|
||||
backoffsMs: [0, 0, 0],
|
||||
telemetry: {
|
||||
emit: vi.fn(async (input) => { emits.push(input); })
|
||||
}
|
||||
});
|
||||
await w.enqueue(id);
|
||||
await w.drain();
|
||||
const hit = emits.filter((e) => e.kind === 'tag_vocab_hit');
|
||||
const miss = emits.filter((e) => e.kind === 'tag_vocab_miss');
|
||||
expect(hit).toHaveLength(1);
|
||||
expect(miss).toHaveLength(1);
|
||||
const hitPayload = hit[0]!.payload as { tagId: number; vocabSize: number };
|
||||
const missPayload = miss[0]!.payload as { vocabSize: number };
|
||||
expect(hitPayload.tagId).toBeGreaterThan(0);
|
||||
expect(hitPayload.vocabSize).toBe(1);
|
||||
expect(missPayload.vocabSize).toBe(1);
|
||||
});
|
||||
|
||||
it('all tags miss when vocab is empty', async () => {
|
||||
// No seed → vocab=[]
|
||||
const { id } = repo.create({ rawText: 'x' });
|
||||
const provider = makeProvider({
|
||||
generate: vi.fn(async () => ({
|
||||
title: 't', summary: 'a\nb\nc',
|
||||
tags: ['design', 'meeting', 'qa'],
|
||||
dueDate: null
|
||||
}))
|
||||
});
|
||||
const emits: EmittedEvent[] = [];
|
||||
const w = new AiWorker(repo, provider, {
|
||||
backoffsMs: [0, 0, 0],
|
||||
telemetry: { emit: vi.fn(async (input) => { emits.push(input); }) }
|
||||
});
|
||||
await w.enqueue(id);
|
||||
await w.drain();
|
||||
const miss = emits.filter((e) => e.kind === 'tag_vocab_miss');
|
||||
expect(miss).toHaveLength(3);
|
||||
expect(emits.filter((e) => e.kind === 'tag_vocab_hit')).toHaveLength(0);
|
||||
});
|
||||
|
||||
it('emits one event per tag (3 tags → 3 events)', async () => {
|
||||
// Pre-seed: all 3 in vocab
|
||||
const seed = repo.create({ rawText: 'seed' }).id;
|
||||
repo.updateAiResult(seed, { title: 't', summary: 'a\nb\nc', tags: ['design', 'meeting', 'qa'], provider: 'p' });
|
||||
|
||||
const { id } = repo.create({ rawText: 'x' });
|
||||
const provider = makeProvider({
|
||||
generate: vi.fn(async () => ({
|
||||
title: 't', summary: 'a\nb\nc',
|
||||
tags: ['design', 'meeting', 'qa'],
|
||||
dueDate: null
|
||||
}))
|
||||
});
|
||||
const emits: EmittedEvent[] = [];
|
||||
const w = new AiWorker(repo, provider, {
|
||||
backoffsMs: [0, 0, 0],
|
||||
telemetry: { emit: vi.fn(async (input) => { emits.push(input); }) }
|
||||
});
|
||||
await w.enqueue(id);
|
||||
await w.drain();
|
||||
const hits = emits.filter((e) => e.kind === 'tag_vocab_hit');
|
||||
expect(hits).toHaveLength(3);
|
||||
});
|
||||
|
||||
it('dedupes duplicate tags in AI response (one emit per unique tag)', async () => {
|
||||
// Pre-seed: 'design' in vocab
|
||||
const seed = repo.create({ rawText: 'seed' }).id;
|
||||
repo.updateAiResult(seed, { title: 't', summary: 'a\nb\nc', tags: ['design'], provider: 'p' });
|
||||
|
||||
const { id } = repo.create({ rawText: 'x' });
|
||||
const provider = makeProvider({
|
||||
generate: vi.fn(async () => ({
|
||||
title: 't', summary: 'a\nb\nc',
|
||||
tags: ['design', 'design', 'meeting'], // 중복 'design' 의도적
|
||||
dueDate: null
|
||||
}))
|
||||
});
|
||||
const emits: EmittedEvent[] = [];
|
||||
const w = new AiWorker(repo, provider, {
|
||||
backoffsMs: [0, 0, 0],
|
||||
telemetry: { emit: vi.fn(async (input) => { emits.push(input); }) }
|
||||
});
|
||||
await w.enqueue(id);
|
||||
await w.drain();
|
||||
const hit = emits.filter((e) => e.kind === 'tag_vocab_hit');
|
||||
const miss = emits.filter((e) => e.kind === 'tag_vocab_miss');
|
||||
expect(hit).toHaveLength(1); // 'design' 중복 → 1 hit (dedup)
|
||||
expect(miss).toHaveLength(1); // 'meeting' 1 miss
|
||||
});
|
||||
});
|
||||
|
||||
@@ -26,6 +26,25 @@ describe('LocalOllamaProvider', () => {
|
||||
expect(r.title).toBe('회의');
|
||||
});
|
||||
|
||||
it('generate passes vocab into prompt body', async () => {
|
||||
let capturedBody: string = '';
|
||||
mock.get('http://localhost:11434').intercept({
|
||||
path: '/api/generate', method: 'POST'
|
||||
}).reply((opts) => {
|
||||
capturedBody = opts.body as string;
|
||||
return { statusCode: 200, data: JSON.stringify({
|
||||
response: JSON.stringify({ title: '회의', summary: 'a\nb\nc', tags: ['design'] })
|
||||
}) };
|
||||
});
|
||||
await new LocalOllamaProvider().generate({
|
||||
text: 'x', todayKst: '2026-05-02', dueDateCandidates: [],
|
||||
vocab: ['design', 'meeting']
|
||||
});
|
||||
const parsed = JSON.parse(capturedBody) as { prompt: string };
|
||||
expect(parsed.prompt).toContain('design, meeting');
|
||||
expect(parsed.prompt).toContain('Prefer reusing');
|
||||
});
|
||||
|
||||
it('generate throws on non-JSON', async () => {
|
||||
mock.get('http://localhost:11434').intercept({ path: '/api/generate', method: 'POST' }).reply(200, {
|
||||
response: 'not json'
|
||||
|
||||
@@ -652,4 +652,94 @@ describe('NoteRepository — failed retry helpers', () => {
|
||||
expect(job.attempts).toBe(1); // 변화 없음
|
||||
expect(job.nextRunAt).toBe('2026-05-01T12:00:00.000Z');
|
||||
});
|
||||
|
||||
it('getTopUsedTags returns [] when no notes', () => {
|
||||
expect(repo.getTopUsedTags()).toEqual([]);
|
||||
});
|
||||
|
||||
it('getTopUsedTags orders by count desc, id asc tiebreaker', () => {
|
||||
const a = repo.create({ rawText: 'a' }).id;
|
||||
const b = repo.create({ rawText: 'b' }).id;
|
||||
const c = repo.create({ rawText: 'c' }).id;
|
||||
repo.updateAiResult(a, { title: 't', summary: 'a\nb\nc', tags: ['design', 'meeting'], provider: 'p' });
|
||||
repo.updateAiResult(b, { title: 't', summary: 'a\nb\nc', tags: ['design'], provider: 'p' });
|
||||
repo.updateAiResult(c, { title: 't', summary: 'a\nb\nc', tags: ['design', 'meeting', 'qa'], provider: 'p' });
|
||||
// counts: design=3, meeting=2, qa=1
|
||||
expect(repo.getTopUsedTags()).toEqual(['design', 'meeting', 'qa']);
|
||||
});
|
||||
|
||||
it('getTopUsedTags filters non-kebab-case (한글/대문자/공백)', () => {
|
||||
const a = repo.create({ rawText: 'a' }).id;
|
||||
// user route 가 한글/공백 태그 들어올 수 있음 → vocab 에서 제외 검증
|
||||
repo.updateUserAiFields(a, { tags: ['design', '회의', 'Meeting', 'two words', 'api-timeout'] });
|
||||
expect(repo.getTopUsedTags()).toEqual(expect.arrayContaining(['design', 'api-timeout']));
|
||||
expect(repo.getTopUsedTags()).not.toContain('회의');
|
||||
expect(repo.getTopUsedTags()).not.toContain('Meeting');
|
||||
expect(repo.getTopUsedTags()).not.toContain('two words');
|
||||
});
|
||||
|
||||
it('getTopUsedTags counts AI + user sources together', () => {
|
||||
const a = repo.create({ rawText: 'a' }).id;
|
||||
const b = repo.create({ rawText: 'b' }).id;
|
||||
const c = repo.create({ rawText: 'c' }).id;
|
||||
// design: 1 AI (a) + 1 user (b) = 2 total; meeting: 1 AI (c) = 1 total
|
||||
// → design must rank first (proves source merging, not AI-only count)
|
||||
// Note: updateUserAiFields REPLACES tags (DELETE+reinsert), so each note
|
||||
// gets exactly the tags passed in the call.
|
||||
repo.updateAiResult(a, { title: 't', summary: 'x\ny\nz', tags: ['design'], provider: 'p' });
|
||||
repo.updateUserAiFields(b, { tags: ['design'] });
|
||||
repo.updateAiResult(c, { title: 't', summary: 'x\ny\nz', tags: ['meeting'], provider: 'p' });
|
||||
const top = repo.getTopUsedTags();
|
||||
expect(top[0]).toBe('design'); // 2 (AI+user) > 1 (AI only)
|
||||
expect(top.indexOf('meeting')).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('getTopUsedTags excludes tags from deleted notes', () => {
|
||||
const a = repo.create({ rawText: 'a' }).id;
|
||||
repo.updateAiResult(a, { title: 't', summary: 'x\ny\nz', tags: ['lonely'], provider: 'p' });
|
||||
repo.trash(a, new Date().toISOString());
|
||||
expect(repo.getTopUsedTags()).not.toContain('lonely');
|
||||
});
|
||||
|
||||
it('getTopUsedTags respects LIMIT parameter', () => {
|
||||
const ids: string[] = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const id = repo.create({ rawText: `n${i}` }).id;
|
||||
ids.push(id);
|
||||
repo.updateAiResult(id, {
|
||||
title: 't', summary: 'a\nb\nc',
|
||||
tags: [`tag-${i}`],
|
||||
provider: 'p'
|
||||
});
|
||||
}
|
||||
expect(repo.getTopUsedTags(3)).toHaveLength(3);
|
||||
expect(repo.getTopUsedTags(10)).toHaveLength(5);
|
||||
});
|
||||
|
||||
it('getTopUsedTags result may be shorter than limit when top-N includes non-kebab tags', () => {
|
||||
// 비-kebab 1개 (한글) + kebab 2개 → top-3 으로 SQL 가져온 후 regex 필터로 한글 제외
|
||||
// 결과 length = 2 (limit=3 보다 작음)
|
||||
const a = repo.create({ rawText: 'a' }).id;
|
||||
const b = repo.create({ rawText: 'b' }).id;
|
||||
const c = repo.create({ rawText: 'c' }).id;
|
||||
repo.updateUserAiFields(a, { tags: ['회의'] }); // 한글 — SQL top 에 포함될 수 있지만 regex 통과 X
|
||||
repo.updateUserAiFields(b, { tags: ['design'] });
|
||||
repo.updateUserAiFields(c, { tags: ['meeting'] });
|
||||
const top = repo.getTopUsedTags(3);
|
||||
expect(top.length).toBeLessThan(3); // SQL 은 3개 가져왔지만 regex 가 1개 제거
|
||||
expect(top).not.toContain('회의');
|
||||
expect(top).toEqual(expect.arrayContaining(['design', 'meeting']));
|
||||
});
|
||||
|
||||
it('getTagIdByName returns id when present, null when absent', () => {
|
||||
const a = repo.create({ rawText: 'a' }).id;
|
||||
repo.updateAiResult(a, { title: 't', summary: 'a\nb\nc', tags: ['hello'], provider: 'p' });
|
||||
const id = repo.getTagIdByName('hello');
|
||||
expect(typeof id).toBe('number');
|
||||
expect(id).toBeGreaterThan(0);
|
||||
// case-insensitive
|
||||
expect(repo.getTagIdByName('HELLO')).toBe(id);
|
||||
// absent
|
||||
expect(repo.getTagIdByName('nothere')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -148,7 +148,7 @@ describe('TelemetryService.readAllRecent', () => {
|
||||
expect(events).toHaveLength(3);
|
||||
// discriminant narrowing — noteId 없는 kind(empty_trash/expired_banner_shown/expired_batch_trash) 가 섞이면 명시적으로 실패
|
||||
expect(events.map((e) =>
|
||||
(e.kind === 'empty_trash' || e.kind === 'expired_banner_shown' || e.kind === 'expired_batch_trash' || e.kind === 'ollama_unreachable' || e.kind === 'ollama_recovered' || e.kind === 'ollama_recheck_manual' || e.kind === 'ai_retry_manual')
|
||||
(e.kind === 'empty_trash' || e.kind === 'expired_banner_shown' || e.kind === 'expired_batch_trash' || e.kind === 'ollama_unreachable' || e.kind === 'ollama_recovered' || e.kind === 'ollama_recheck_manual' || e.kind === 'ai_retry_manual' || e.kind === 'tag_vocab_hit' || e.kind === 'tag_vocab_miss')
|
||||
? null
|
||||
: e.payload.noteId
|
||||
)).toEqual(['a', 'b', 'b']);
|
||||
@@ -164,7 +164,7 @@ describe('TelemetryService.readAllRecent', () => {
|
||||
expect(events).toHaveLength(1);
|
||||
const ev = events[0]!;
|
||||
expect(ev.kind).toBe('capture');
|
||||
if (ev.kind !== 'empty_trash' && ev.kind !== 'expired_banner_shown' && ev.kind !== 'expired_batch_trash' && ev.kind !== 'ollama_unreachable' && ev.kind !== 'ollama_recovered' && ev.kind !== 'ollama_recheck_manual' && ev.kind !== 'ai_retry_manual') expect(ev.payload.noteId).toBe('a');
|
||||
if (ev.kind !== 'empty_trash' && ev.kind !== 'expired_banner_shown' && ev.kind !== 'expired_batch_trash' && ev.kind !== 'ollama_unreachable' && ev.kind !== 'ollama_recovered' && ev.kind !== 'ollama_recheck_manual' && ev.kind !== 'ai_retry_manual' && ev.kind !== 'tag_vocab_hit' && ev.kind !== 'tag_vocab_miss') expect(ev.payload.noteId).toBe('a');
|
||||
});
|
||||
|
||||
it('returns [] when dir missing', async () => {
|
||||
|
||||
31
tests/unit/prompt.test.ts
Normal file
31
tests/unit/prompt.test.ts
Normal file
@@ -0,0 +1,31 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { buildPrompt, PROMPT_VERSION } from '@main/ai/prompt.js';
|
||||
|
||||
describe('prompt', () => {
|
||||
it('PROMPT_VERSION is 4', () => {
|
||||
expect(PROMPT_VERSION).toBe(4);
|
||||
});
|
||||
|
||||
it('buildPrompt with empty vocab omits vocabulary line entirely', () => {
|
||||
const out = buildPrompt('hello', '2026-05-02', [], []);
|
||||
expect(out).not.toContain('vocabulary');
|
||||
expect(out).not.toContain('Prefer reusing');
|
||||
});
|
||||
|
||||
it('buildPrompt with vocab includes Prefer instruction + comma-separated list', () => {
|
||||
const out = buildPrompt('hello', '2026-05-02', [], ['design', 'meeting', 'qa']);
|
||||
expect(out).toContain('Existing vocabulary tags');
|
||||
expect(out).toContain('design, meeting, qa');
|
||||
expect(out).toContain('Prefer reusing');
|
||||
});
|
||||
|
||||
it('vocab block appears after header and before JSON rules', () => {
|
||||
const out = buildPrompt('hello', '2026-05-02', [], ['design']);
|
||||
const headerIdx = out.indexOf("Today's date");
|
||||
const vocabIdx = out.indexOf('Existing vocabulary');
|
||||
const jsonRulesIdx = out.indexOf('Return a JSON object');
|
||||
expect(headerIdx).toBeGreaterThan(-1);
|
||||
expect(vocabIdx).toBeGreaterThan(headerIdx);
|
||||
expect(jsonRulesIdx).toBeGreaterThan(vocabIdx);
|
||||
});
|
||||
});
|
||||
@@ -278,3 +278,31 @@ describe('ai_retry_manual event', () => {
|
||||
})).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateEvent — tag vocab', () => {
|
||||
it('accepts tag_vocab_hit event', () => {
|
||||
const e = validateEvent({
|
||||
ts: '2026-05-02T00:00:00.000Z',
|
||||
kind: 'tag_vocab_hit',
|
||||
payload: { tagId: 42, vocabSize: 17 }
|
||||
});
|
||||
expect(e.kind).toBe('tag_vocab_hit');
|
||||
});
|
||||
|
||||
it('accepts tag_vocab_miss event without tagId', () => {
|
||||
const e = validateEvent({
|
||||
ts: '2026-05-02T00:00:00.000Z',
|
||||
kind: 'tag_vocab_miss',
|
||||
payload: { vocabSize: 17 }
|
||||
});
|
||||
expect(e.kind).toBe('tag_vocab_miss');
|
||||
});
|
||||
|
||||
it('rejects tag_vocab_hit with extra field (privacy invariant)', () => {
|
||||
expect(() => validateEvent({
|
||||
ts: '2026-05-02T00:00:00.000Z',
|
||||
kind: 'tag_vocab_hit',
|
||||
payload: { tagId: 42, vocabSize: 17, tagName: 'leak' }
|
||||
})).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -164,3 +164,27 @@ describe('aggregateStats — ai_retry_manual', () => {
|
||||
expect(r.md).toMatch(/AI 수동 재시도.*2회.*10건/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('aggregateStats — tag_vocab hit/miss', () => {
|
||||
it('aggregates tag_vocab hit/miss with success rate', () => {
|
||||
const events: TelemetryEvent[] = [
|
||||
e('2026-05-02T00:00:00Z', 'tag_vocab_hit', { tagId: 1, vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:01Z', 'tag_vocab_hit', { tagId: 2, vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:02Z', 'tag_vocab_hit', { tagId: 3, vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:03Z', 'tag_vocab_hit', { tagId: 4, vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:04Z', 'tag_vocab_hit', { tagId: 5, vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:05Z', 'tag_vocab_miss', { vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:06Z', 'tag_vocab_miss', { vocabSize: 10 }),
|
||||
e('2026-05-02T00:00:07Z', 'tag_vocab_miss', { vocabSize: 10 })
|
||||
];
|
||||
const r = aggregateStats(events, new Date('2026-05-03T00:00:00Z'));
|
||||
expect(r.md).toContain('태그 vocab: hit/miss = 5/3');
|
||||
expect(r.md).toContain('적중률 62.5%');
|
||||
});
|
||||
|
||||
it('태그 vocab summary shows 데이터 없음 when no events', () => {
|
||||
const r = aggregateStats([], new Date('2026-05-03T00:00:00Z'));
|
||||
expect(r.md).toContain('태그 vocab');
|
||||
expect(r.md).toContain('데이터 없음');
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user