feat(embed): candle 모델 레지스트리 + arctic-embed-l-v2.0 (CLS pooling)

e5 하드코딩(HF_MODEL/SUPPORTED_MODEL/mean/query:+passage:) → 모델 레지스트리
EmbedModelSpec{name,hf_repo,pooling,query_prefix,doc_prefix,dim,version_tag}.
e5(mean, query:/passage:) + arctic(CLS, query:/무접두어). pooling 모델별 분기
(mean=attention-mask-weighted / CLS=hidden[:,0,:]), tokenize/forward/L2 공유.
arctic pooling=CLS 는 HF 1_Pooling/config.json(pooling_mode_cls_token:true) 확인.
model_version 은 arctic 일 때 +arctic-cls 태그(embedding_version cascade 트리거);
e5 는 fastembed-e5 호환(NUMA 드롭인) 위해 plain config.version 유지.

correctness 게이트: tests/arctic_ollama_parity.rs (#[ignore], live Ollama) —
candle arctic vs Ollama snowflake-arctic-embed2 per-sentence 코사인>0.99.
수동 실측 cosine_min=0.999984 (recall@10 130 재현 보장).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-03 04:59:11 +00:00
parent 7505645008
commit cbcae69abf
3 changed files with 390 additions and 84 deletions

View File

@@ -38,6 +38,9 @@ metal = ["candle-core/metal", "candle-nn/metal", "candle-transformers/metal"]
# not the library's own (non-dev) dependencies — so rayon/kebab-config/kebab-core
# are repeated here for tests/parity.rs and tests/thread_cap.rs.
kebab-embed-local = { path = "../kebab-embed-local" }
# arctic↔Ollama parity test drives the real Ollama adapter for the reference
# vectors (tests/arctic_ollama_parity.rs, `#[ignore]` — live Ollama).
kebab-embed-ollama = { path = "../kebab-embed-ollama" }
kebab-config = { path = "../kebab-config" }
kebab-core = { path = "../kebab-core" }
rayon = "1"