Mapping Kafka and MQTT

The following diagram shows how tenefit.cloud uses topic patterns to map between MQTT messages and Kafka messages.

Kafka-MQTT mapping

Transforming MQTT messages to Kafka messages

  • MQTT topic patterns containing $key cause Kafka messages to include a key matching the value for $key extracted from the MQTT topic name.

  • Publishing an MQTT message with an empty payload results in a Kafka message with a null value. Kafka topics configured for log compaction treat this as a tombstone for the corresponding message key.

  • Kafka headers prefixed with $mqtt are reserved for internal use. MQTT messages published by MQTT clients should not include user properties with names prefixed by $mqtt.

Transforming Kafka messages to MQTT messages

  • MQTT subscribers receive Kafka messages filtered by message key. MQTT topic patterns containing $key deliver only Kafka messages with key matching $key.

  • A Kafka message with a null or empty value is delivered as an MQTT message with an empty payload to MQTT subscribers.

  • Kafka header names prefixed by $mqtt are reserved for internal use. They are removed before transformation and are not included in MQTT messages delivered to MQTT clients. Kafka messages produced by a Kafka producer should not contain headers with names prefixed by $mqtt.