HDFS на собеседовании Data Engineer

HDFS архитектура

Hadoop Distributed File System. Distributed FS для big files на commodity hardware.

Properties:

• Append-only (мостly).
• Immutable files после close.
• Optimized для large sequential reads.
• Не подходит для many small files (NameNode bottleneck).

Blocks

Файлы разбиваются на blocks (default 128 MB).

big_file.parquet (1 GB)
  → block 1 (128 MB)
  → block 2 (128 MB)
  ...

Каждый block stored на multiple DataNodes (replication).

NameNode и DataNode

NameNode. Master. Stores metadata (file → blocks → DataNodes).

• In-memory (RAM hungry).
• Single point of failure (без HA setup).

DataNode. Worker. Stores actual block data.

• Periodically reports блоки to NameNode.
• Replication coordinated by NameNode.

Replication

Default replication factor — 3.

Block A:
  Replica 1 — DataNode-1 (rack 1)
  Replica 2 — DataNode-2 (rack 2)
  Replica 3 — DataNode-3 (rack 2)

Rack-aware placement. 1 копия в local rack, 2 в other rack — survives rack failure.

При DataNode failure — NameNode replicates lost blocks.

Federation и HA

HA. Active + Standby NameNode. ZooKeeper coordinates.

Federation. Multiple NameNodes, каждый owns namespace. Scales metadata beyond single NameNode RAM.

EC (Erasure Coding). Alternative replication. RAID-like — saves storage at cost of recompute. 50%+ storage savings.

Связанные темы

• Hadoop и MapReduce для DE
• S3 и object storage для DE
• Spark RDD vs DataFrame для DE
• Hive Metastore для DE
• Подготовка к собесу Data Engineer

FAQ

HDFS vs S3?

S3 — managed, durable, cheap. HDFS — self-hosted, low-latency. В 2026 lake часто на S3-compatible (HDFS — legacy).

Это официальная информация?

Нет. Статья основана на документации Apache Hadoop.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.