Consumer offsets are reset in the following scenarios:
(1) No offset is committed to the broker. For example, the consumer is first connected to the client.
(2) Messages are pulled from an invalid offset. For example, the maximum offset in a partition is 10, but the client starts to pull messages from Offset 11.
Resetting a Consumer Offset:
On Java clients, you can set the auto.offset.reset parameter to one of the following values to specify how to reset a consumer offset:
(1) latest: resets the consumer offset to the maximum offset.
(2) latest: resets the consumer offset to the maximum offset.
(3) none: does not reset the consumer offset.
Configuration Suggestion
(1) We recommend that you set this parameter to latest instead of earliest. This prevents consumers from consuming messages from the very beginning due to an invalid offset.
(2) If you commit the offset by calling the commit(offsets) function, you can set this parameter to none.
(3) Pulling Large Messages
Cautions for Pulling Large Messages
During message consumption, clients proactively pull messages from the broker. If you want to pull large messages, you can control the pulling rate by configuring the following parameters:
(1) max.poll.records: If each message exceeds 1 MB in size, we recommend that you set this parameter to 1.
(2) fetch.max.bytes: Set this parameter to a value that is slightly greater than the size of a single message.
(3) max.partition.fetch.bytes: Set this parameter to a value that is slightly greater than the size of a single message.
Clients pull large messages one by one.