Mapping Kafka and AMQP

The following diagram shows how tenefit.cloud uses topic patterns to map between AMQP messages and Kafka messages.

Kafka-AMQP mapping

Transforming AMQP messages to Kafka messages

  • AMQP topic patterns containing $key cause Kafka messages to include a key matching the value for $key extracted from the AMQP topic name.

  • Publishing an AMQP message with an empty payload results in a Kafka message with a null value. Kafka topics configured for log compaction treat this as a tombstone for the corresponding message key.

  • Kafka headers prefixed with $amqp are reserved for internal use, such as retention of message type information. AMQP messages published by AMQP clients should not include user properties with names prefixed by $amqp.

Transforming Kafka messages to AMQP messages

  • AMQP subscribers receive Kafka messages filtered by message key. AMQP topic patterns containing $key deliver only Kafka messages with key matching $key.

  • A Kafka message with a null or empty value is delivered as an AMQP message with an empty payload to AMQP subscribers.

  • Kafka header names prefixed by $amqp are reserved for internal use, such as injection of message type information. They are removed before transformation and are not included in AMQP messages delivered to AMQP clients. Kafka messages produced by a Kafka producer should not contain headers with names prefixed by $amqp.