test(v0.17.0/A5 follow-up): trigram tokenizer downstream test fixes

trigram tokenizer 가 snippet 단위 + 단어 경계 + BM25 raw score 분포를
모두 바꿔서 unicode61 assumption 기반의 3 test 가 regression.

- wire_search_response::search_json_truncates_with_max_tokens +
  search_plain_emits_truncated_hint_to_stderr: 단일 doc + 작은
  max_tokens 로는 snippet 이 짧아서 budget loop 가 trip 안 함.
  다중 doc fixture (5 doc) + budget 30 token 으로 hit-pop 경로
  통해 truncated=true 보장.
- fetch_integration::fetch_chunk_with_context_returns_neighbors:
  fixture body 의 2-char tokens (A1/A3 등) 가 trigram 비호환으로
  0-hit. apples/banana/cherry/durian/elder 5-char unique words
  로 갱신, query 도 cherry 로 deterministic pin.
- eval/runner::runner_per_query_snapshot_matches_fixture: trigram
  token stream 으로 BM25 raw score 변동. UPDATE_SNAPSHOTS=1 로
  regenerate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-24 12:21:34 +00:00
parent 8a68289499
commit 0ee18149e7
3 changed files with 31 additions and 8 deletions

View File

@@ -5,7 +5,7 @@
"chunk_id": "chunk000000000000000000000000000000",
"doc_id": "doc00000000000000000000000000000000",
"heading_path": [],
"score": 0.3429983854293823
"score": 0.35202541947364807
},
"has_answer": false,
"hits_count": 1,
@@ -19,7 +19,7 @@
"chunk_id": "chunk000000000000000000000000000002",
"doc_id": "doc00000000000000000000000000000002",
"heading_path": [],
"score": 0.3585492968559265
"score": 0.3414848744869232
},
"has_answer": false,
"hits_count": 1,