Background
A Kafka consumer group is a set of consumers that consume messages on one or more topics. Consumer group is an important concept in implementing message parallel processing and load balancing.
Features of consumer groups include:
l Distributed consumption: Messages in one topic can be consumed by multiple consumer groups, and consumers within each consumer group can process messages in parallel. This distributed consumption model allows multiple consumers to process messages at the same time, improving the throughput and efficiency of message processing.
l Load balancing: Within a consumer group, each consumer is responsible for consuming one or more partitions of a topic. Kafka automatically assigns partitions to ensure that the number of partitions consumed by each consumer is roughly equal, enabling load balancing.
l Consumer coordinator: Each consumer group has a consumer coordinator to manage consumer registration, partition assignments, and offset submissions. The consumer coordinator interacts with the Kafka cluster to coordinate consumer behaviors.
l Offset management: Each consumer in a consumer group saves the offset of the partition that it consumes. The offset represents the offsets of the messages that the consumer has consumed in the partition. Kafka automatically manages the submission and maintenance of offsets to ensure that consumers can continue to consume from where they last consumed.
By using consumer groups, Kafka implements efficient message consumption and load balancing. Consumer groups allow multiple consumers to process messages in parallel, improving the throughput and concurrency performance of message processing. At the same time, the load balancing mechanism within a consumer group ensures that the number of partitions consumed by each consumer is almost the same, thus avoiding resource waste and uneven load distribution.
Before You Begin
You have purchased and deployed a Distributed Message Service (Kafka).
Procedure
(1) Click Consumer Group Management, and click Create Consumer Group.
(2) Click Create Consumer Group, enter a name for the consumer group, and click Create.
(3) The created consumer group will appear in the consumer group list. Click Topic Subscription to subscribe to the topic in Step 2, and you can use Consumption Dialing Test to verify whether the consumption is normal.
