Long context LLM на собеседовании Data Scientist

Why long context

Original GPT — 2K context. GPT-4 turbo — 128K. Claude — 200K. Gemini — 1M+.

Use cases:

• Whole code repository в context.
• Long documents без RAG chunking.
• Multi-turn conversations long history.
• Books / research papers.

Quadratic complexity

Standard attention — O(N²) memory + compute.

N=1K: 1M ops.
N=100K: 10B ops. — 10000× more.

Doubling context → 4× cost. Becomes infeasible.

Position encoding tricks

RoPE. Better extrapolation чем sinusoidal.

ALiBi. Linear bias attention. Trained на 2K, works на 16K без retrain.

YARN. RoPE scaling — extends RoPE smoothly.

Position interpolation. Compress positions linearly.

Models trained 4K → can extrapolate к 16K-32K с tweaks.

Sparse attention

Only attend к subset positions.

Sliding window. Recent N tokens (Mistral). Plus global tokens (begin / important).

Strided. Attend каждый Nth token.

Block sparse. Attention в blocks (Longformer, BigBird).

FlashAttention. Не sparse, но IO-efficient — practical долгий context.

Recurrent / state-space

Mamba (2023). Selective state-space model. Linear complexity.

RWKV. RNN-Transformer hybrid.

Linear attention. Various approximations.

Pros: linear scaling. Cons: still catching up Transformers in quality.

Mamba-2 / hybrid models — active research 2024-2026.

Связанные темы

• Transformer для DS
• Attention для DS
• KV-cache для DS
• GPT architecture для DS
• Подготовка к собесу Data Scientist

FAQ

Это официальная информация?

Нет. Статья основана на работах Su 2021 (RoPE), Press 2022 (ALiBi), Gu 2023 (Mamba).

Тренируйте Data Science — откройте тренажёр с 1500+ вопросами для собесов.