Spark + Iceberg на собеседовании Data Engineer

Setup

spark.conf.set("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.iceberg.type", "hive")
spark.conf.set("spark.sql.catalog.iceberg.uri", "thrift://hive-metastore:9083")
spark.conf.set("spark.sql.catalog.iceberg.warehouse", "s3://my-warehouse/")

Iceberg jar на classpath.

Catalog

USE iceberg.warehouse;

CREATE TABLE events (
  id BIGINT, ts TIMESTAMP, user_id BIGINT
) USING iceberg
PARTITIONED BY (days(ts));

Read / write

df = spark.read.format("iceberg").load("iceberg.warehouse.events")

# Or SQL
df = spark.sql("SELECT * FROM iceberg.warehouse.events WHERE date(ts) = '2026-05-07'")

# Write
new_df.writeTo("iceberg.warehouse.events").append()

ACID гарантии — multiple writers без conflict (если не overlapping data).

Schema evolution

ALTER TABLE iceberg.warehouse.events ADD COLUMN device_type STRING;
ALTER TABLE iceberg.warehouse.events RENAME COLUMN ts TO event_ts;

Без перезаписи data. Iceberg metadata tracks column IDs.

Compaction

Many small files → poor performance.

CALL iceberg.system.rewrite_data_files('warehouse.events');

Combines small files в large. Optimization regular.

Expire old snapshots:

CALL iceberg.system.expire_snapshots(
  'warehouse.events', 
  TIMESTAMP '2026-04-01' 
);

Reclaims storage.

Связанные темы

• Iceberg deep для DE
• Spark RDD vs DataFrame для DE
• Lakehouse Iceberg Delta для DE
• Spark Catalog для DE
• Подготовка к собесу Data Engineer

FAQ

Это официальная информация?

Нет. Статья основана на документации Apache Iceberg / Spark.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.