Snowpipe на собеседовании Data Engineer

Что такое Snowpipe

Snowflake's continuous data ingestion. Files в storage → autoload в table.

Vs traditional COPY. Manual / scheduled — slow.

Snowpipe — within minutes файла arrival.

Auto-ingest

S3 sends event notification → SQS → Snowpipe.

CREATE PIPE my_pipe AUTO_INGEST=TRUE AS
COPY INTO my_table
FROM @my_stage
FILE_FORMAT = (TYPE = PARQUET);

S3 PutObject → automatic load.

Manual REST

client = SnowpipeClient(...)
client.insert_files(['file1.parquet', 'file2.parquet'])

App-controlled. Когда auto-ingest не applies.

Snowpipe Streaming

2023+. Direct row-level ingest без files.

SnowflakeStreamingIngestClient.insert_rows([row1, row2, ...])

Sub-second latency. Replaces Kafka → S3 → Snowpipe pipeline.

Cost

Per file overhead. Many small files — pricey.

Optimize:

• Aggregate files (target 100MB-1GB).
• Compress.

Snowpipe credit-based pricing.

Связанные темы

• Snowflake vs BigQuery для DE
• S3 и object storage для DE
• CDC vs batch loading для DE
• Streaming SQL для DE
• Подготовка к собесу Data Engineer

FAQ

Это официальная информация?

Нет. Статья основана на документации Snowflake.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.