The delivery semantics for Kafka is at least once. This means that a message is delivered at least once to ensure that the message is not lost. However, this does not ensure that the message is not duplicated. When a network error occurs or the client restarts, a small number of messages may be repeatedly delivered. If the consumer is sensitive to message duplication, such as in the online transaction scenario, consumption idempotence must be implemented.
If your application is a database application, you can perform the following operations to implement an idempotence check:
(1) When you send a message, specify a key as a unique transaction ID.
(2) When you consume a message, check whether the key is consumed. If the key is consumed, skip the message. If the key is not consumed, consume the message once.
If your application is not sensitive to the duplication of a few messages, an idempotence check is not required.