Apache Beam на собеседовании Data Engineer

Что такое Beam

Унifit модель batch + streaming, portable между runners.

Idea. Write once, run anywhere. Code на Beam → execute на Spark / Flink / Dataflow / direct.

В практике Beam — main use на GCP Dataflow.

PCollection и PTransform

PCollection. Distributed dataset (bounded или unbounded).

PTransform. Operation на PCollection (Map, Filter, GroupBy).

import apache_beam as beam

with beam.Pipeline() as p:
    (p
     | 'Read' >> beam.io.ReadFromText('input.txt')
     | 'Split' >> beam.Map(lambda x: x.split(','))
     | 'Filter' >> beam.Filter(lambda x: int(x[1]) > 100)
     | 'Write' >> beam.io.WriteToText('output.txt'))

| — pipe operator. Builds DAG, executed by runner.

Runners

DirectRunner. Local, для testing.

DataflowRunner. GCP managed.

SparkRunner. На Spark cluster.

FlinkRunner. На Flink.

Apache Samza, Apache Apex. Less common.

Same code — different runner — different execution.

Windowing

Beam — first-class event time windowing.

(p | 'Read' >> beam.io.ReadFromKafka(...)
   | 'Window' >> beam.WindowInto(beam.window.FixedWindows(60))
   | 'Aggregate' >> beam.CombinePerKey(sum))

Window types: Fixed, Sliding, Session, Global.

Triggers. Когда emit results — early (speculative), on-time, late.

Применение

GCP Dataflow. Heavy use на Google Cloud.

Cross-cloud. Companies wanting portability.

Streaming + batch unified. Same code оба.

В РФ — почти не используется. Spark / Flink direct — стандарт.

Связанные темы

• Spark Structured Streaming для DE
• Apache Flink для DE
• Cloud ETL tools для DE
• Spark vs Flink для DE
• Подготовка к собесу Data Engineer

FAQ

Это официальная информация?

Нет. Статья основана на документации Apache Beam.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.