1.1.1 FAQs for Operation

How Long Are Messages Retained on Kafka?

Messages are retained on Kafka for 72 hours and will be deleted after this period.

How Many Topics Can Kafka Create?

Kafka Basic Edition can create 50 topics, and Kafka Advanced Edition can create 100 topics.

Is It Possible to Consume Data That Has Already Been Consumed?

Consumers use an underlying blocking queue and immediately consume data produced by the producer. This causes a serious problem. If you restart a consumer, no data can be obtained though you can see that all the data exist in the log file. In other words, once the data is consumed, it cannot be consumed by the consumer with the same groupid. In this case, you can reset the offset of the consumer group in the console within 3 days.

Is It Necessary to Create a Consumer Group in Advance?

Though consumer groups and subscribed topics can be automatically created when a business app client is connected, we recommend you to create a consumer group in advance.

The error message "Not authorized to access group" is displayed.

You can resolve this problem by creating a consumer group.

Why Does a PHP Client Experience a Long Delay When It Sends a Message?

The long sending delay is caused by the PHP language feature. The PHP client reinitializes a KafkaProducer object each time it sends a message. The initialization involves various operations, such as connecting to each broker and updating metadata. The sending delay is greater than 100 ms in a VPC and 500 ms on the Internet. In contrast, a Java client re-uses KafkaProducer and has a small sending delay.

Where Can I Find Examples of Producing Consumption Messages?

Best practices - producer practices and consumer practices.

How Can I Test Message Sending?

You can test message sending in the Kafka console. Log in to the console, go to the Topics page, and send messages in the Actions column of a topic to test whether the cluster is running properly.

How Can I Determine Whether a Message Is Sent on the Client?

If the callback is successful, the message is sent. Most clients return Callback or Future after a message is sent. If the callback is successful, the message is sent.

You can also check whether the message is properly sent in the console in the following ways:

Click Monitoring Information to check the number of produced instance messages.

Click Message Query to view messages by time.

Does the Callback Setting on the Java Client Affect the Message Sending Speed?

It depends on the processing time in the callback and the setting of max.in.flight.requests.per.connection.

(1) Processing time in the callback: To reduce the processing time in a callback, do not frequently perform time-consuming processing in the callback. You can process multiple callbacks after a specific number of ACKs are accumulated or process them in another asynchronous thread to avoid blocking the completion of callbacks.

(2) Setting of max.in.flight.requests.per.connection: Before the producer receives a response from the server, the maximum number of messages that can be sent is determined by the setting of max.in.flight.requests.per.connection. A higher value requires a larger memory, but it will also improve throughput.

Which Open Source Broker Versions Are Supported?

Broker versions 2.3.1 and 2.8.2 are supported. After an instance is created, the broker version cannot be upgraded or customized.

How Do I Select Storage Space for an Instance?

Disk size = Average traffic × Message retention period × 3 (replicas). We recommend you optimize the topics when migrating to the cloud to reduce costs.

What Problems May Occur if I Upgrade Brokers?

If you upgrade brokers, messages may be out of order, clients may be disconnected from the brokers, or messages cannot be evenly distributed to partitions.

The following problems may occur:

During the upgrade, all brokers in the Kafka cluster are restarted one by one. Services are not interrupted during the restart of brokers. However, the messages consumed within 5 minutes after each broker is restarted may be out of order in specific partitions.

l During the restart of brokers, clients that are connected to the brokers may be disconnected. If the clients support automatic reconnection, other brokers automatically take over the services.

l During the upgrade and restart of brokers, the volumes of messages processed by different partitions are also uneven. You need to evaluate the impacts of the upgrade on your business.

It takes about 5 to 15 minutes to upgrade all the brokers. If you have multiple instances, you can first upgrade a test cluster. After you upgrade the test cluster and verify the upgrade results, you can upgrade the production cluster.

Why Am I Unable to Change the Region of an Instance?

After an instance is purchased and deployed, its region is closely associated with its physical resources and cannot be changed. If you need to change the region of an instance, release the instance and purchase a new instance.

How Can I Perform a Quick Test on the Kafka Broker?

Before You Begin

A Kafka instance is created and deployed, and it is in the Running state.

Procedure

To perform a quick test on the Kafka broker, perform the following steps:

(1) Create a topic.

(2) Log in to the console, go to the Topics page, and then click Send Message in the Actions column of a topic to send messages.

(3) On the Topics page, check the partition status.

(4) On the message query page, query messages by time or offset.

Are Delayed Messages Supported?

No, Distributed Message Service (Kafka) does not support delayed messages as Apache Kafka.

Is Data Compression Supported?

Yes, the Distributed Message Service (Kafka) broker can send and receive compressed data.

To use this feature, you need to set compression-related parameters on a Kafka client. When you set compression-related parameters on the Kafka client, note the following points:

Compression format: Formats such as Snappy, LZ4, and GZIP are supported. The GZIP format consumes a large quantity of CPU resources. Therefore, we recommend that you use Snappy or LZ4.

Scenarios: Generally, CPU resources are more expensive than traffic and storage resources. Therefore, we recommend that you use compression only in scenarios that require a high compression ratio, such as logs, but not in other scenarios.

CPU consumption: Compression occupies extra CPU resources, more than 20% on average. You can test the extra CPU consumption based on the actual scenario.

How to Release Instances?

You can unsubscribe from an instance on the console if it is no longer needed.

Why Is the Total Number of Topics (Partitions) Limited?

A large number of topics (partitions) will greatly reduce the cluster performance and stability.

In Kafka, messages are stored and scheduled by partition. If messages are stored in a large number of partitions, the cluster performance and stability are greatly reduced.

Why Am I Unable to Reduce Partitions for a Topic?

Due to the limit of Apache Kafka, data will be lost if you reduce partitions for a topic.

Is the Compact Log Cleanup Policy Supported?

Yes, Kafka instances of open-source version 2.2.0 or later support the compact log cleanup policy.

How Do I Check the IP Addresses from Which Messages Are Consumed?

You can check the consumption details of a consumer group in the Kafka console.

Where Can I Find Consumer Practices?

Best practices - consumer practices.

How Can I Modify Consumer Offsets in the Console?

The mechanism used to commit consumer offsets depends on the consumer SDK that you use. Generally, the following two mechanisms are supported:

Automatic committing: The SDK commits the consumer offset of the latest consumed message plus 1 at the specified intervals.

Manual committing: You call the commit(offsets) function to commit the consumer offset of the latest consumed message plus 1 on the client.

You can reset the offset of a consumer group in the Kafka console.

Why Do I Fail to View the Subscriptions of a Consumer Group in the Kafka Console?

Issue Description

I have started the consumption thread of a consumer group, but no subscriptions to topics are displayed in the Kafka console.

Possible Reasons

The configured subscription is not successful because the client settings are invalid or a network exception occurs on the client.

You use the assign mode to manually specify a specific partition to which the consumer subscribes, but you do not commit the consumer offset.

Solutions

Troubleshoot the issues on the client.

Troubleshoot the network issues on the client.

Commit the consumer offset.

Why Is a Partition Consumed by Multiple Consumption Threads?

When a consumer client uses the StickyAssignor policy to consume messages, multiple consumer threads consume the same partition. As a result, messages cannot be processed as expected.

Possible Reasons

The version of your client is earlier than 2.3. Clients whose version is earlier than 2.3 may assign the same partition to multiple consumption threads for consumption.

Solutions

We recommend that you update the version of your client to 2.3 or later or use another partition assignment policy.

If you use the StickyAssignor policy, assignment issues such as the duplicate assignment of partitions can occur in some cases. We recommend that you do not use this assignment policy unless necessary.

Why Is a Group in the Deleting State for a Long Time?

Groups are deleted in asynchronous mode. Typically, it takes about 1 to 2 minutes to delete a group.

Issue Description

After you delete a group on the consumer group management page in the Kafka console, the group has been in the Deleting state for a long time. Deleting

Possible Reasons

The group has active subscriptions.

One or more consumer threads in the group are committing consumer offsets.

Solutions

If the group subscribes to a topic, or one or more consumer threads of the group are committing consumer offsets, cancel the subscription and then delete the group again.

Why Is the Accumulated Messages Value of a Group 0 or Empty?

When you view the consumption status of a group, the Consumer Offset value is not equal to the Maximum Offset value, but the Accumulated Messages value is 0 or empty.

Possible Reasons

You have not committed a consumer offset, or the consumer offset that you committed has expired.

You have committed a consumer offset and the consumer offset is in effect, but some messages are deleted due to expiration, which causes the displayed consumer offset to be smaller than or equal to the minimum offset.

Why Cannot I Log In to the Instance Where Distributed Message Service (Kafka) Is Deployed?

Distributed Message Service (Kafka) is fully hosted and maintained by the Kafka professional team. You do not need to log in to the instance where the Kafka is deployed. Some basic cluster information is transparently transmitted through the monitoring and alerting feature.

Will a Kafka Instance Be Restarted After Its Enterprise Project Is Modified?

No.

Do Kafka Instances Support Batch Topic Import?

You can create topics in batches. You can download the batch creation template and fill in the required information. For details, see the operation guide for Batch Creating Topics.

What Should I Do if Kafka Storage Space Is Used Up Because Consumed Messages Are Not Deleted?

Messages are not deleted immediately after being consumed and are still stored on disks. As a result, Kafka storage space may be used up.

Can I Delete Unnecessary Topics in a Consumer Group?

Yes. You can delete unnecessary topics in a consumer group as follows:

(1) Log in to the management console.

(2) Go to the Kafka console.

(3) In the operation column of the instance list page, click Manage in the target instance row.

(4) Click Topic Management. On the Topic Management page, click More. A pop-up window will appear as shown below.

(5) In the topic row, click Delete and select OK.

Do I Need to Create Consumer Groups, Producers, and Consumers for Kafka Instances?

No, you are not required to create them. Based on your needs, you can create consumer groups as follows:

(1) Log in to the management console.

(2) Go to the Kafka console.

(3) In the operation column of the instance list page, click Manage in the target instance row.

(4) Click Consumer Group Management to enter the consumer group management page.

(5) Click Create Consumer Group, enter a name for the consumer group, and click Create.

Note: When a consumer group business application is connected, the client can create the consumer group automatically.

What Is the Maximum Size of a Message that Can Be Created?

1 MB by default.

Why Do Messages Still Exist After the Retention Period Elapses?

Possible reasons:

l An error in message storage configuration: The message retention time is not configured correctly. Check whether the log.retention.ms or log.retention.hours parameter in the Kafka configuration file is correct.

l A consumer group delay or failure: If a message is sent to a consumer group, but a consumer in the consumer group has a delay or failure that leads to untimely consumption of the message, the message may still exist after the retention period elapses.

l An issue in message processing logic: There is an issue in the message processing logic on the consumer that prevents messages from being processed or consumed in a timely manner. It may be a code bug, processing logic error, or other issue.

l Inconsistency across multiple partitions or topics: If your messages are distributed across multiple partitions or topics, and the message retention time for some partitions or topics is different from others, the messages may exist in some partitions or topics after the retention period elapses.

To solve this problem, you can take the following measures:

l Check the Kafka configuration file to make sure the message retention time is configured correctly.

l Check the status of consumer groups to ensure that consumers can consume messages in a timely manner.

l Check the message processing logic on the consumer to ensure that messages are processed and consumed correctly.

l Check the configuration of partitions or topics to ensure that they have consistent message retention time.

1.1.2 Rules for Configuring Client Parameters

This topic describes the rules for configuring client parameters for Kafka. We recommend that you specify proper values for these parameters based on your business requirements. The following tables describe the parameters for clients:

Table 1. Producer client parameters

Parameter	Description
retries	The number of retries that are allowed for a message that fails to be sent.
retry.backoff.ms	The interval between retries when a message fails to be sent. We recommend that you set this parameter to 1000. Unit: milliseconds.
acks	The persistence mechanism for sending messages. To improve message sending performance, we recommend that you set this parameter to 1. If you set this parameter to 0, no response is returned from the broker. In this mode, the message sending performance is high, but the risk of data loss is also high. If you set this parameter to 1, a response is returned when data is written to the leader. In this mode, the message sending performance and the risk of data loss are moderate. If a failure occurs on the leader, data loss may occur. If you set this parameter to all, a response is returned when data is written to the leader and synchronized to the followers. In this mode, the message sending performance is low, but the risk of data loss is low. Data loss occurs only when the leader and the followers fail at the same time.
batch.size	The volume of cached messages to be delivered to each partition. When the volume of cached messages reaches the specified upper limit, a network request is triggered. Then, the producer sends the messages to the broker in a batch. If you set the batch.size parameter to a small value, the message sending performance and stability may be affected. We recommend that you use the default value 16384. Unit: bytes.
linger.ms	The maximum storage duration for each message in the cache. If a message is stored in the cache for a period of time that exceeds the maximum storage duration, the producer immediately sends the message to the broker while ignoring the batch.size parameter. We recommend that you set the linger.ms parameter to a value that ranges from 100 to 1000 based on your business requirements. Unit: milliseconds.
partitioner.class	The strategy that you use for partitioning. We recommend that you use the sticky partitioning strategy to improve message sending performance. If the version of your producer client is 2.4 or later, the sticky partitioning strategy is used by default.
buffer.memory	The size of the memory pool on the client. If you set this parameter to a small value, the time that is consumed to apply for memory may be long. This causes low message sending performance or message sending timeout. We recommend that you set this parameter to a value that is equal to or greater than the value of Value of the batch.size parameter × Number of partitions × 2. Unit: bytes.

Table 2. Consumer client parameters

Parameter	Description
fetch.min.bytes	The minimum amount of data that the consumer can pull from the broker. Before you configure this parameter, you must evaluate the number of messages to be sent by the producer. If you set this parameter to an excessively large value, message delay may be increased when the consumer consumes messages. If you set this parameter to an excessively small value, the frequency at which the consumer pulls messages may be increased. Unit: bytes.
fetch.max.wait.ms	The maximum amount of time that the broker waits to return a response. Unit: milliseconds. If you use local storage and configure the fetch.min.bytes parameter, the broker returns a response after the value that is specified by the fetch.min.bytes parameter is reached. After the time that is specified by the fetch.max.wait.ms parameter elapses, a response is returned regardless of whether there is enough data. If you use cloud storage, the broker immediately returns a response once new data is sent in instead of waiting until the time that is specified by the fetch.min.bytes parameter elapses.
max.partion.fetch.bytes	The maximum amount of data that each partition returns. Unit: bytes.
session.timeout.ms	The interval at which the consumer sends heartbeats. If no heartbeat is sent within the interval, the broker determines that the consumer is dead, and a rebalance is triggered. During the rebalance, the client stops consuming data and waits for the rebalance to complete. We recommend that you set this parameter to 3000060000. Unit: milliseconds. Valid value: 6000300000.
max.poll.records	The maximum number of messages returned for each poll. If you set this parameter to an excessively large value, you must process the message consumption logic at the earliest opportunity. If the message consumption logic is processed at a low speed, data in the next poll is affected. As a result, no heartbeat is sent during the interval that is specified by the session.timeout.ms parameter, and a rebalance is triggered. We recommend that you set this parameter to a value that is smaller than the value of <Number of messages consumed by each thread per second> × <Number of threads consuming messages> × <Value specified by session.timeout.ms>. In Java Client 0.10.1 and later versions, a separate thread is used to send heartbeats. If you use a Java Client of earlier versions or a client in another language, you must specify proper values for parameters that are related to data processing time and heartbeat sending interval. This helps prevent frequent rebalances from affecting normal consumption.
max.poll.interval.ms	The maximum interval between invocations of polls. This parameter is required only in Java client 0.10.1 and later versions. If the consumer does not request polls during the interval that is specified by the session.timeout.ms parameter, the broker determines that the consumer is dead, and a rebalance is triggered even if a heartbeat is sent. Therefore, you must set this parameter to a proper value. We recommend that you set this parameter to a value that is greater than the value of <Time consumed to consume a message> × <Number of data records pulled>. In most cases, you can use the default value. Unit: milliseconds. Valid values: 300000.
enable.auto.commit	Specifies whether to automatically commit the consumer offset. Valid values: true: automatically commit the consumer offset by default false: do not commit the consumer offset automatically Default value: true
auto.commit.interval.ms	The interval at which the consumer offsets are automatically committed. Default value: 1000. Unit: milliseconds.
auto.offset.reset	The policy that is used to reset the consumer offset. latest: resets the consumer offset to the maximum offset. earliest: resets the consumer offset to the earliest offset. none: does not reset the consumer offset. We recommend that you set this parameter to latest instead of earliest. This prevents consumers from consuming messages from the very beginning due to an invalid offset. If you commit the offset by calling the commit(offsets) function, you can set this parameter to none.

1.1.3 How to Determine and Handle Message Accumulation?

l Checking whether message accumulation is normal

Log in to the Distributed Message Service (Kafka) console. On the Consumer Group Management page, locate the target consumer group and go to the Message Accumulation page.

(1) If the number of accumulated messages fluctuates within an expected stable range, the client is pulling new messages. In this case, message accumulation is normal.

(2) If the number of accumulated messages gradually increases and the consumer offset remains unchanged, the consumer thread on the client is blocked, and the client stops consuming messages and committing consumer offsets to the broker. In this case, message accumulation is abnormal.

(3) If the number of accumulated messages gradually increases and consumer offsets are being committed, the client is still consuming messages, but the message consumption rate is lower than the message production rate. In this case, message accumulation is abnormal. In most cases, abnormal message accumulation is caused by slow message consumption or a blocked consumption thread. We recommend that you do not specify long durations for related parameters in the consumption logic.

l How to handle abnormal message accumulation

After you confirm that abnormal message accumulation exists based on the preceding operations, you can print the message consumption duration or check the execution of the thread based on the stack information. This way, you can increase the rate of message consumption to prevent abnormal message accumulation.

1.1.4 Why Do Rebalances Frequently Occur on My Consumer Client?

l Possible Reasons

The Kafka client version is too low or the Consumer does not have a separate thread to send heartbeat messages.

For clients earlier than version 0.10.2: The consumer does not have a separate thread to send heartbeat messages. Heartbeat messages are sent by using the poll interface. If the consumption response is stuck, the request to send heartbeat messages times out, and a rebalance occurs.

For clients of version 0.10.2 and later: No messages are pulled after the time specified by the max.poll.interval.ms parameter elapses. This causes the client to disconnect from the message queue and triggers a rebalance. The default value of the max.poll.interval.ms parameter is 5 minutes.

l Solutions

You must understand the following parameters:

session.timeout.ms: specifies the timeout period of requests to send heartbeat messages. You can specify a custom value based on your business requirements.

max.poll.records: specifies the maximum number of messages returned for each poll.

For clients earlier than version 0.10.2: Heartbeat messages are sent by using the poll interface. The consumer does not have a separate thread to send heartbeat messages.

For clients of version 0.10.2 and later: You can set the max.poll.interval.ms parameter to prevent the clients from not consuming messages for a long time.

(1) The following items provide references for you to set parameter values:

session.timeout.ms: For clients earlier than version 0.10.2, set a value larger than the time required to consume a batch of messages but smaller than 30 s. We recommend that you set the value to 25s. For clients of version 0.10.2 and later, keep the default value of 10 s.

max.poll.records: Set a value far smaller than the number of consumed messages per thread per second multiplied by the number of threads multiplied by the value of the max.poll.interval.ms parameter.

max.poll.interval.ms: Set a value larger than the value of max.poll.records divided by the product of the number of consumed messages per thread per second multiplied by the number of threads.

(2) Improve the consumption speed of the client and allocate a separate thread to process the consumption logic.

(3) Reduce the number of topics subscribed to by a group to no more than five. We recommend that one group subscribes to only one topic.

(4) Upgrade the client to version 0.10.2 or later.

1.1.5 Why Does a Consumer Slowly Pull a Message From the Broker or Fail to Pull a Message From the Broker?

Issue Description

Topics to which the consumer subscribes receive messages, and the consumption does not reach the latest offset. However, the consumer slowly pulls messages from the broker or fails to pull messages from the broker. Such issues occur more frequently when the consumption is performed over the Internet.

Possible Reasons

The consumption traffic reaches the network bandwidth.

The traffic for a single message exceeds the network bandwidth.

The traffic for the messages pulled by the consumer at a time exceeds the network bandwidth.

Description

The following parameters determine the number and size of messages that a consumer can pull at a time:

max.poll.records: the maximum number of messages that the consumer can pull at a time.

fetch.max.bytes: the maximum number of bytes of messages that the consumer can pull at a time.

max.partition.fetch.bytes: the maximum number of bytes of messages that the consumer can pull from a single partition at a time.

Solutions

(1) Log in to the Kafka console to query messages.

If messages are returned, perform the subsequent steps.

(2) On the Instance Details page, click Monitoring in the left-side navigation pane to check whether the consumption traffic reaches the network bandwidth.

If the consumption traffic reaches the network bandwidth, increase the network bandwidth.

(3) Check whether the traffic for a single message in the topic exceeds the network bandwidth.

If the traffic exceeds the network bandwidth, increase the network bandwidth or reduce the size of the message.

(4) Check whether the traffic for the messages pulled by the consumer at a time exceeds the network bandwidth

Description

If the traffic exceeds the network bandwidth, adjust the configuration of the following parameters:

fetch.max.bytes: Set the parameter to a value smaller than the network bandwidth.

max.partition.fetch.bytes: Set the parameter to a value smaller than the limit value. The limit value is calculated by using the following formula: Network bandwidth/Number of partitions that are subscribed to.

1.1.6 Why Is It Not Recommended to Use a Sarama Go Client to Send and Receive Messages?

Issue Description

If you use a Go client that is developed based on the Sarama library, the following issues may occur:

After you add a partition for a topic, the client cannot detect the new partition or consume messages in the partition. You must restart the client to enable the client to consume messages from the new partition.

If the client subscribes to more than two topics, the client may fail to consume messages from specific partitions.

If the policy for resetting consumer offsets of the client is set to Oldest(earliest), the client may start to consume messages from the earliest offset after the client breaks down or the broker version is updated. This issue occurs because the out_of_range class is implemented in the code of the client.

Solutions

We recommend that you replace the Go client that is developed based on the Sarama library with a Go client that is developed by using Confluent at the earliest opportunity.

For information about the demo of the Go client that is developed by using Confluent, see kafka-confluent-go-demo.

1.1.7 Why Are the Messages That Are Sent to a Topic Unevenly Distributed Among the Partitions of the Topic?

Issue Description

The messages that are sent to a topic are unevenly distributed among the partitions of the topic. Some partitions have more messages, some partitions have fewer messages, and some partitions even have no messages.

Possible Reasons

Messages are sent only to the specified partitions. As a result, the unspecified partitions have no messages.

Messages are sent to a specified partition based on a specified message key. As a result, messages are unevenly distributed among partitions.

The partition assignment policy is re-coded, but the policy logic has defects. As a result, messages are unevenly distributed among partitions.

1.1.8 Why Can a Group That Does Not Appear in the Console Consume Messages?

A group does not appear in the Kafka console. However, consumer threads in this group are consuming messages.

Possible Reasons

l A consumer uses the assign mode to consume messages. In this case, consumer threads may be able to consume messages even if you did not create a group for the consumer in the console.

l A consumer uses the subscribe mode to consume messages. After the group to which the consumer belongs is deleted, consumer threads can continue to consume messages when the consumer threads are not stopped or rebalancing is not triggered.

Solutions

l When a consumer uses the assign mode to consume messages, create a group in the Kafka console in advance.

l We recommend that you use existing groups when possible, because a large number of created groups may affect the stability of the cluster.

l Before you delete a group, make sure that all consumer threads in the group are stopped.

1.1.9 Does the NFS Attached to a Consumer Affect the Message Consumption Speed?

A consumer uses the main consumption thread to synchronize the pulled messages to the network file system (NFS) that is attached to the consumer. This decreases the message consumption speed of the consumer and blocks message processing.

Possible Reasons

An NFS cannot run fast enough.

An NFS is a shared network storage system that supports simultaneous access from multiple consumers. However, the consumers compete for resources. The more consumers involved, the lower performance the NFS can provide.

Solutions

We recommend that you use two independent threads to consume and store messages. One is used for message pulling, and the other is used for message storage. After the thread for message pulling processes a message, the thread forwards the processing results to the other thread for storage and continues message consumption. This ensures consumption efficiency.

You can also attach an Elastic Volume Service (EVS) disk to each consumer to independently store processing results. This prevents performance degradation due to the competition for NFS resources among consumers. To store the processing results of different consumers in the same NFS, you can use an asynchronous tool or thread to forward the processing results that are stored in EVS disks to the NFS. This prevents message processing from being blocked by synchronous storage in the NFS. Asynchronous processing is an effective way to prevent performance degradation due to competitive resource access.