Spark Catalog API на собеседовании Data Engineer

Catalog в Spark

Spark catalog — namespace для databases, tables, views.

spark.catalog.listDatabases()
spark.catalog.listTables('my_db')
spark.catalog.tableExists('my_db.my_table')

TempView

Session-scoped, in-memory only. Видна только в session.

df.createOrReplaceTempView('my_view')
spark.sql('SELECT * FROM my_view')

После spark.stop() — view исчезает.

Global temp view. Cross-session, but cluster restart — gone.

df.createGlobalTempView('global_view')
spark.sql('SELECT * FROM global_temp.global_view')

Persistent table

Saved в catalog с metadata (Hive / Iceberg / etc).

df.write.saveAsTable('my_db.my_table')
spark.sql('SELECT * FROM my_db.my_table')

Persists между sessions. Catalog tracks location.

Catalog implementations

In-memory (default). Catalog в memory driver. Lost при stop.

Hive Metastore. Standard для big data. Persistent, shared across Spark / Trino / Hive.

Iceberg REST. Modern catalog для Iceberg.

Glue Catalog (AWS). Managed Hive-compatible.

Unity Catalog (Databricks). Modern lakehouse catalog.

Polaris (Snowflake). Iceberg-compatible.

Configure через spark.sql.catalogImplementation=hive.

Iceberg / Delta integration

Iceberg:

spark.sql("""
  CREATE TABLE my_db.events (
    id BIGINT, ts TIMESTAMP
  ) USING iceberg
  PARTITIONED BY (days(ts))
""")

Spark + Iceberg — modern lakehouse.

Delta Lake:

df.write.format("delta").save("/path/to/table")
spark.sql("CREATE TABLE my_db.events USING delta LOCATION '/path/to/table'")

Time travel, ACID — встроено.

Связанные темы

• Hive Metastore для DE
• Iceberg deep для DE
• Spark RDD vs DataFrame для DE
• Lakehouse Iceberg Delta для DE
• Подготовка к собесу Data Engineer

FAQ

Это официальная информация?

Нет. Статья основана на документации Spark / Iceberg / Delta.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.