Оценка качества embeddings на собеседовании Data Scientist

Intrinsic eval

Properties embedding space без specific task.

Similarity tasks. Annotated word / sentence pairs с similarity scores. Compute cosine между embeddings, correlate с human ratings.

Analogies. «king - man + woman ≈ queen». Word2vec early benchmark.

Clustering. Words / docs cluster по semantic categories.

Cons. Не reflect actual task performance.

Extrinsic eval

Use embeddings в downstream task.

Classification. Train classifier на embeddings → measure accuracy.

Retrieval. NDCG, MRR на retrieval task.

STS (Semantic Textual Similarity). Predict similarity scores.

Pros. Reflects real-world utility.

Cons. Specific к use case — different task may show different ranking embeddings.

MTEB

Massive Text Embedding Benchmark. Hugging Face leaderboard.

50+ tasks, 8 categories:

• Classification.
• Clustering.
• Pair classification.
• Reranking.
• Retrieval.
• STS.
• Summarization.
• Bitext mining.

Standard benchmark — newest models compete.

Domain-specific

Generic benchmarks miss domain quirks.

Build custom eval set. 100-1000 representative pairs (similar / not).

Measure cosine similarity distribution. Should разделять similar from not.

Retrieval metrics. На golden retrieval задаче — measure recall.

В medical / legal / финансы — domain-specific embeddings significantly better чем generic.

Связанные темы

• Embeddings на собесе DS
• BERT vs GPT для DS
• word2vec, GloVe, FastText для DS
• Vector databases для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на работах Muennighoff 2022 (MTEB).

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.