During message sending and consumption, Kafka, producers, and consumers collaborate to ensure service reliability. In addition, developers must use Kafka message queues properly to improve the efficiency and accuracy of message sending and consumption.

Best practices for message producers and consumers are as follows:

Acknowledging Message Production and Consumption

Message Production (Sending)

The producer decides whether to re-send a message based on the Kafka response after sending a message.

The producer waits for the response of the messaging API to determine if the message is successfully sent. If an exception occurs when sending the message, the producer will not receive a success response and must decide whether to re-send the message. If a success response is received, it indicates that the message has been stored in Kafka.

Message Consumption

The consumer acknowledges successful message consumption.

The produced messages are sequentially stored in Kafka. During consumption, messages stored in Kafka are obtained in sequence. Consumers obtain messages, consume them, and record the status (successful or failed). The status is then submitted to Kafka. Based on the consumption status, Kafka determines whether to consume the next batch of messages or re-send the messages that failed to be consumed.

During this process, the message consumption status may not be successfully submitted due to an exception. In this case, the corresponding messages will be re-obtained by the consumers in the next message consumption request.

Idempotent Transferring of Message Production and Consumption

To guarantee lossless messaging, Distributed Message Service (Kafka) implements a series of reliability measures. For example, the message synchronization storage mechanism is used to prevent the system and server from being abnormally restarted or powered off. The ACK mechanism is used to deal with exceptions that occur during message transmission.

Considering extreme conditions such as network exceptions, you can use Kafka to design message sending and consumption in addition to acknowledging message production and consumption.

If message sending cannot be acknowledged, the producer needs to re-send the message.

After consuming a message that has been processed, the consumer needs to notify Kafka that consumption is successful and ensure that the message is not processed repeatedly.

Producing and Consuming Messages in Batches

To improve the message sending and consumption efficiency, you are advised to use the batch message sending and consumption mode, which can effectively lower the number of API calls and minimize service fees.

This way, resources are saved.

A maximum of 10 messages can be sent in batches. The total size cannot exceed 512 KB.

Batch production (sending) of messages can be flexibly used. When there are a large number of concurrent messages, the messages are sent in batches. When the number of concurrent messages is small, the messages are sent one by one. In this way, the number of API calls is reduced and the real-time message sending is ensured.

When consuming messages in batches, consumers need to process and acknowledge messages in the sequence of receiving messages. Therefore, when a message in the batch fails to be consumed, the consumer does not need to consume the remaining messages, and directly submit consumption acknowledgment of the successfully consumed messages.

Using Consumer Groups to Facilitate O&M

You can use Kafka as a message management system. Reading message content from queues is helpful for fault locating and service debugging.

When problems occur during message production and consumption, you can create different consumer groups to locate and analyze problems or debug services for interconnecting with other services. You can create a consumer group to consume messages in queues and analyze the consumption process. This does not affect the message processing of other services.