Schema Registry на собеседовании Data Engineer

Что такое Schema Registry

Centralized schema management для Kafka messages.

Producer → registry: register schema → assigns ID.
Producer → Kafka: [magic byte][schema_id][payload].
Consumer → reads schema_id → fetches schema из registry → decodes.

Catches schema drift автоматически.

Subjects и schemas

Subject. Logical name (typically <topic>-value или <topic>-key).

Schema. Specific version под subject.

Subject: orders-value
  v1: {id: long, amount: double}
  v2: {id: long, amount: double, status: string}  # added field

Каждая registered schema — unique ID globally.

Compatibility levels

Configurable per-subject.

• BACKWARD (default). New schema reads old data. Old consumers OK.
• FORWARD. Old schema reads new data.
• FULL. Both.
• TRANSITIVE. Compat all previous versions, не just N-1.
• NONE. No checks.

При registering breaking schema → registry rejects.

Producer / consumer integration

Confluent serializers. Auto-register schemas, auto-encode с schema ID.

from confluent_kafka.schema_registry.avro import AvroSerializer

producer.produce(topic='orders', value=order_data, on_delivery=cb)
# Behind scenes — fetches/registers schema, prefixes message с ID.

Consumer mirror — fetches schema by ID, decodes properly.

Альтернативы

Apicurio Registry. Open source.

AWS Glue Schema Registry. AWS-native.

Karapace. Open source Confluent SR-compatible.

В РФ Confluent Schema Registry available open source (community edition).

Связанные темы

• Schema evolution для DE
• Kafka на собесе DE
• Kafka Connect для DE
• CDC и Debezium для DE
• Подготовка к собесу Data Engineer

FAQ

Это официальная информация?

Нет. Статья основана на документации Confluent Schema Registry.

Тренируйте Data Engineering — откройте тренажёр с 1500+ вопросами для собесов.