1. RocketMQ cannot avoid duplicated messages because:
(1) The offset for acknowledgment is synchronized to the server every 5 seconds.
(2) To prevent message loss, the SDK always submits the minimum offset of the messages that have not been acked. For example, for unordered messages with an offset of 1, 2, 3, 4, 5, messages 1, 3, 4, 5 are consumed and acked, and message 2 is not acked. Offset 2 is the last offset submitted for ack. If the client restarts at this time, consumption will start from offset 2.
(3) Network interactions do not necessarily mean that all interaction data is delivered. To prevent data loss, it is necessary to retry, which may also lead to duplication.
2. If the business is very sensitive to duplicate consumption, you can avoid duplication using either method shown below:
(1) You can deduplicate messages at the business level by msgId. If the key field is unique to the business, you can also use the key field for deduplication.
(2) You can implement consumption idempotence on the business logic. Specifically, if the same message is processed multiple times, its impact on the service is idempotent.