ML inference latency optimization на собеседовании Data Scientist

Где latency

Total request time:

• Network — client to server.
• Queue (если batching).
• Preprocessing.
• Model forward.
• Postprocessing.
• Network response.

Каждое — opportunity optimization.

Model size

Smaller model — faster.

Distillation. Train small student от large teacher.

Pruning. Remove low-impact weights.

Quantization. FP32 → INT8 / INT4.

Use smaller architecture. EfficientNet vs ResNet, DistilBERT vs BERT.

Trade-off vs accuracy.

Batching

GPU underutilized на single requests. Batch — high throughput.

Static batching. Wait для batch_size, then process.

Dynamic batching (Triton). Tunable wait.

Continuous batching (vLLM для LLM). Different sequence lengths.

10-100× throughput improvement.

Trade-off. Tail latency increases (waiting).

Caching

Pre-computed embeddings. Если same input — cache result.

KV-cache для LLM. Reuse computation.

Feature cache. Pre-compute features, lookup при inference.

Output cache. Identical inputs → cached output.

Hardware

CPU. OK для small models, low traffic. Cheap.

GPU. ML deep models. NVIDIA H100, A100.

TPU. Google's tensor processing.

Edge. Apple Silicon, mobile NPUs.

ASICs / FPGAs. Custom — high cost, niche.

Inference servers. Triton, TorchServe, vLLM, TGI — optimized infrastructure.

Связанные темы

• Inference optimization для DS
• Квантование LLM для DS
• Pruning и distillation для DS
• KV-cache для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на ML inference industry practices.

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.