Mixture of Experts на собеседовании Data Scientist

Идея MoE

Several «expert» sub-networks. Router decides what experts активируется per input.

Total params: 100B.
Per-token activation: only 10B (1-2 experts).

10× larger model для same compute.

Sparse activation

Каждый token activates малое число experts (typically 1-2 of 8).

Input → router → top-2 experts → weighted sum.

Other experts «sleep» для that token. Compute-efficient.

Router

Learnable gating function.

gate_scores = softmax(router(x))
top_k_experts = topk(gate_scores, k=2)
output = Σ gate_score[i] · expert[i](x)

Router trained вместе с experts.

Load balancing

Без regulation — router может collapse к few experts. Auxiliary loss penalize unbalanced routing.

L_balance = Σ (frac_to_expert_i · gate_to_expert_i)

Encourages even usage всех experts.

Mixtral, GPT-4 rumors

Mixtral 8x7B (Mistral AI 2023). 8 experts, 2 active per token. 47B total params, ~13B active. Quality close to 70B dense.

GPT-4 rumored to be MoE (1T total, 220B active per query). Unconfirmed.

DeepSeek-V3. Modern MoE с innovative routing.

Switch Transformer (Google 2021). Original at scale.

Trend — frontier LLMs MoE. Better tradeoff scale / cost.

Связанные темы

• BERT vs GPT для DS
• Transformer на собесе DS
• Pruning и distillation для DS
• Inference optimization для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на работах Shazeer 2017 (MoE), Fedus 2021 (Switch Transformer), Mistral docs.

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.