VLM evaluation на собеседовании Data Scientist

Зачем eval VLM

VLM widely используются — но capabilities differ. Eval — measure for use case.

VQA benchmarks

VQAv2. Visual Question Answering — open-ended вопросы.

OK-VQA. Outside-knowledge questions (require world knowledge помимо image).

ScienceQA. Multimodal science questions.

TextVQA. Questions about text в images.

Document understanding

DocVQA. Document QA (forms, receipts, reports).

ChartQA. Charts / graphs reading.

InfographicVQA. Infographics.

RuDocVQA. Russian docs (если specific to RU).

Specific capabilities

MMMU. Multimodal Multidiscipline Understanding.

MathVista. Visual math.

MMVet. Comprehensive — knowledge, reasoning, OCR, math.

RealWorldQA. Real-world spatial reasoning.

Important — каждый model has weaknesses. GPT-4V sometimes fails counting, Claude — fails specific OCR cases.

Custom evaluation

Для production — golden set domain-specific.

Pipeline:

1. Curate 100-1000 representative cases.
2. Annotate ground truth answers.
3. Run candidate models.
4. Score через LLM-judge или human.
5. Track regressions.

В domains (medical, legal, financial) — generic benchmarks insufficient. Custom mandatory.

Связанные темы

• Multimodal LLM для DS
• CLIP multimodal для DS
• Hallucinations и LLM evals для DS
• RAG eval для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на VLM benchmarks papers (MMMU, MathVista, etc).

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.