Yandex Cloud data services на собеседовании Data Engineer

Object Storage

S3-compatible. Hot / Cold / Cold Archive tiers.

import boto3
s3 = boto3.client('s3', endpoint_url='https://storage.yandexcloud.net')

Tools (Spark, dbt) work с S3 API.

Managed ClickHouse

Managed CH cluster. Single-host или distributed (sharded + replicated).

Pros vs self-host:

• Auto backups, monitoring.
• Auto-failover.
• Easy scaling.

Cons: vendor lock, limited config tuning.

В РФ — стандарт DWH replacement Snowflake / BigQuery.

Managed Greenplum

Managed GP. Поддержка Postgres-compatible MPP.

Use case: big enterprise needing transactions + analytics.

Часто Сбер / банки используют — replicated к Yandex Cloud.

DataTransfer

Managed CDC / replication.

Sources: Postgres, MySQL, MongoDB, Kafka, S3.

Targets: ClickHouse, Greenplum, Object Storage, Postgres.

Postgres OLTP → DataTransfer (CDC) → ClickHouse OLAP.

Real-time replication.

DataProc

Managed Hadoop / Spark cluster.

Pros: Spark / Hive / HBase ready. Auto-scaling.

Cons: Spark compute сильно дороже на CH workload.

DataSphere

ML platform — JupyterLab, training infrastructure, deployment.

Notebook → train model on GPU → deploy as endpoint.

В РФ — alternative SageMaker / Vertex AI / Databricks.

Связанные темы

• S3 и object storage для DE
• DWH ClickHouse для DE
• Greenplum для DE
• CDC и Debezium для DE
• Подготовка к собесу Data Engineer

FAQ

Это официальная информация?

Нет. Статья основана на документации Yandex Cloud.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.