RAG evaluation на собеседовании Data Scientist

Зачем RAG eval

RAG имеет 2 components:

• Retrieval — поиск relevant docs.
• Generation — answer based on retrieved.

Каждый может fail. Eval каждой stage independently.

Retrieval metrics

Recall@K. Доля right docs в top-K.

Precision@K. Сколько в top-K really relevant.

MRR. Position первого relevant.

NDCG. Graded relevance.

Требуется labeled data: query → relevant doc IDs.

Generation metrics

Faithfulness. Answer based на retrieved context (не hallucinated).

Answer relevancy. Answer addresses question.

Coherence. Grammatically / logically right.

Часто evaluated через LLM-as-judge:

"Given context: ... Answer: ...
Rate 1-5: does answer use only information from context?"

End-to-end metrics

Correctness. Comparing к ground truth answer.

Helpfulness. User-perceived (через ratings, или LLM judge).

Latency. End-to-end time.

Cost. Tokens API / compute.

RAGAS

Framework для RAG eval.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

dataset = ...  # questions + answers + retrieved + ground_truth
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])

Standard для RAG benchmarking.

Golden set

Manually curated test set.

Size. 100-1000 quality examples enough для regression testing.

Coverage. Diverse domains, hard cases, edge cases.

Use:

• CI eval на каждое change.
• Track regression.
• Compare model variants.

Без golden set — eval purely subjective.

Связанные темы

• RAG на собесе DS
• Vector databases для DS
• Hallucinations и LLM evals для DS
• Vector search optimization для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на документации RAGAS.

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.