GPT architecture на собеседовании Data Scientist

Decoder-only

GPT — decoder-only Transformer. Generates next token autoregressively.

Tokens [t1, t2, t3] → Transformer → P(t4 | t1, t2, t3)

No encoder. Self-attention с causal mask.

Causal masking

Token at position i может attend только к positions 0..i.

Attention mask:
[1 0 0 0]
[1 1 0 0]
[1 1 1 0]
[1 1 1 1]

Prevents «future leak» — model trains predict next token based только на previous.

Layer normalization

Where placed matters.

Post-LN. Original Transformer. Layer norm после residual.

Pre-LN. Modern. Layer norm перед attention / FFN. Better gradient flow для deep networks.

RMSNorm. Variant — only scale, no centering. Used Llama, GPT-NeoX. Cheaper.

Positional encoding (RoPE)

Original sinusoidal positional embeddings.

RoPE (Rotary Position Embedding). Modern. Encode position через rotation в complex space.

Pros:

• Better length extrapolation.
• Relative position-aware.

Used Llama, GPT-NeoX, Mistral.

ALiBi. Alternative — slope penalties в attention. Even longer context.

SwiGLU

Activation function. Variant GLU (Gated Linear Unit) с SiLU activation.

SwiGLU(x) = SiLU(xW_1) ⊗ (xW_2) W_3

Better чем ReLU / GELU на large LMs. Used Llama.

Связанные темы

• BERT vs GPT для DS
• Transformer для DS
• Attention для DS
• Mixture of Experts для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на работах Radford 2019 (GPT-2), Su 2021 (RoPE), Llama papers.

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.