Airflow для аналитика с нуля

Зачем это знать

Airflow — де-факто стандарт orchestration в data world. Запускает ETL pipelines, обновляет dashboards, рассчитывает метрики. Аналитик, не умеющий читать / писать DAGs — ограничен в автоматизации.

На собесах middle+ Airflow знание ожидается. В Yandex, Tinkoff, Ozon, Avito — обязательный стек.

Что такое Airflow

Workflow orchestration platform:

• Schedule tasks
• Define dependencies
• Monitor runs
• Retry на fail
• Parallel execution

Originally from Airbnb, now Apache.

Основные концепции

DAG

Directed Acyclic Graph — workflow.

Nodes = tasks. Edges = dependencies.

Task

Unit of work. Например, «run SQL», «execute Python».

Operator

Template of task:

• BashOperator
• PythonOperator
• PostgresOperator
• BigQueryOperator
• и multiple другие

Scheduler

Daemon, monitoring DAGs и triggering runs.

Executor

Actually runs tasks (Local, Celery, Kubernetes).

Первый DAG

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def calculate_daily_revenue():
    # Your Python code
    print("Calculating revenue...")

default_args = {
    'owner': 'analytics',
    'retries': 2,
    'retry_delay': timedelta(minutes=5)
}

with DAG(
    dag_id='daily_revenue',
    start_date=datetime(2026, 1, 1),
    schedule_interval='@daily',
    default_args=default_args,
    catchup=False
) as dag:

    calc_task = PythonOperator(
        task_id='calculate_revenue',
        python_callable=calculate_daily_revenue
    )

Saved в dags/ folder → Airflow picks up.

Dependencies

extract >> transform >> load
# Или
extract.set_downstream(transform)

Task load ждёт transform, который ждёт extract.

Parallel

extract >> [transform_a, transform_b] >> load

A и B в parallel после extract.

Schedule

• @daily — 00:00 каждый день
• @hourly — каждый час
• 0 2 * * * — cron (2 AM)
• None — trigger manually only

Backfill

catchup=False — не запускать за past dates. True — run все дни between start_date и now.

На production — usually False.

XComs

Пройти value между tasks:

def extract(**context):
    context['ti'].xcom_push(key='rows', value=1000)

def transform(**context):
    rows = context['ti'].xcom_pull(key='rows', task_ids='extract')

Но large data — через DB / files, не XCom.

Типичный pipeline

Для analyst:

1. Extract — pull from API / DB
2. Transform — clean, aggregate в Python / SQL
3. Load — write в analytics DB
4. Validate — data quality checks
5. Refresh dashboards — Metabase / Tableau cache

extract_sales = BigQueryOperator(
    task_id='extract_sales',
    sql='SELECT ... FROM transactions WHERE date = {{ ds }}'
)

transform = PythonOperator(...)

load = PostgresOperator(
    task_id='load_warehouse',
    sql='INSERT INTO warehouse.daily_sales ...'
)

validate = PythonOperator(...)

extract_sales >> transform >> load >> validate

Templated queries

{{ ds }} — execution date. Airflow renders.

sql = """
    SELECT * FROM events
    WHERE date = '{{ ds }}'
"""

Troubleshooting

Task fails

• Check logs в Airflow UI
• Retry automatic (если configured)
• Manual re-run

Skipped tasks

Upstream failed → downstream skipped unless trigger_rule specified.

Long-running

Task can be slow. Задайте execution_timeout.

UI

Web UI на http://localhost:8080:

• DAG list
• Graph view
• Timeline (gantt)
• Logs
• Manual trigger

Production considerations

Idempotency

Task должен быть idempotent: если запустить дважды, result same.

-- Idempotent (upsert)
INSERT ON CONFLICT DO UPDATE

-- Not idempotent (can duplicate)
INSERT

Test в dev

Unit tests для Python callables. Integration tests для DAG structure.

Monitoring

SLA misses → alerts (email, Slack).

Failed tasks → PagerDuty или on-call.

Airflow vs альтернативы

Airflow

• Largest community
• Flexible
• Python-based
• Complex для simple use cases

Dagster

• Modern alternative
• Better UX
• Typed assets
• Smaller community

Prefect

• Python-first
• Simpler setup
• Growing

Airflow — default choice enterprise.

На собесе

«Airflow experience?» Какие DAGs, operators, scale.

«Что такое DAG?» Directed Acyclic Graph of tasks с dependencies.

«Как handle failures?» Retries, alerts, manual re-run.

«Idempotency почему важна?» Re-run без side-effects.

Связанные темы

• Airflow шпаргалка
• Как настроить Airflow DAG
• Airflow vs Dagster
• Batch vs stream processing
• dbt шпаргалка

FAQ

Надо знать для junior analyst?

Advantage, не mandatory. Для middle+ — expected.

Без programming возможно?

No. Python knowledge required.

Alternative для simple cases?

Cron jobs. Airflow — когда workflow complex.

---

Тренируйте аналитику — откройте тренажёр с 1500+ вопросами для собесов.