MLOps мониторинг моделей на собеседовании Data Scientist

Что мониторить

Input. Distribution features.

Output. Distribution predictions.

Performance. Accuracy / metrics (когда labels available).

System. Latency, throughput, error rates.

Business. Conversion / revenue impact.

Data drift

Distribution input features changes vs training.

training: P(X_train).
production: P(X_prod).

Detection methods:

• KS test, PSI.
• Multi-variate (MMD, classifier).

Setup alerts when drift exceeds threshold.

Prediction drift

Output distribution changes (could be due to drift или real change).

training: 30% predicted "positive".
production: 60% — что произошло?

Doesn't necessarily imply degradation. Investigate.

Performance monitoring

Accuracy/AUC/etc — requires labels.

Issues:

• Labels delayed (chargeback за weeks).
• Labels expensive (manual annotation).
• Labels biased (only confident predictions reviewed).

Workarounds:

• Proxy metrics (early signals).
• Sampling for human review.
• Approximate labels.

Outliers и anomalies

Individual prediction outliers — model uncertain или OOD.

Confidence calibration. Low confidence → flag для review.

OOD detection. «Energy score», «Mahalanobis distance» в feature space.

Tools

Evidently AI. Open source.

Whylogs / WhyLabs. Profiles-based.

Arize, Fiddler, Aporia. Commercial.

Custom. Python + Prometheus + Grafana.

Best dashboards include:

• Feature distribution per day.
• Performance metrics over time.
• Alert thresholds.
• Compare текущий период vs baseline.

Связанные темы

• Data drift для DS
• MLflow и DVC для DS
• Canary и shadow deployment ML для DS
• Hallucinations и LLM evals для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на ML monitoring industry practices.

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.