How Do I Ensure the Reliability of Message Publishing?
l acks=0: no response is returned from the broker before the producer successfully writes the message. In this mode, the message sending performance is high, but the risk of data loss is also high.
l acks=1: A response is returned to the producer when data is written to the leader.
l acks=all / -1: A response is returned to the producer when data is written to the leader and synchronized to the followers. In this mode, the message sending performance is low, but the risk of data loss is low.
How Do I Ensure the Order of Messages?
l In Kafka, each partition is independent and order can only be guaranteed within a partition.
l Local order: It is an order for a single user, and is not a global order. You can use the user ID as the key to ensure that a certain user is assigned to a specific partition (which may cause partition skew).
l To realize a global order, use the same key for all messages, so they will be assigned to the same partition. This method applies to scenarios with a small amount of temporary data, and the performance may deteriorate when the amount of messages is large.
How Do I Choose the Number of Partitions?
l You can specify the number of partitions when creating a topic, or manually increase the number after the topic is created. The number of partitions can only be increased. The amount of data between partitions becomes uneven if you increase the number of partitions.
l The number of partitions directly determines the concurrent processing ability of the topic, but a larger number of partitions does not necessarily mean a better performance. The number of partitions has an impact on message latency.
l Generally, we recommend you to select Broker Num * Consumer Num so that on average each consumer reads the number (Broker Num) of partitions at the same time, and this evenly distributes partition pressure across each broker.
How to Select Storage Space?
Kafka instances support multi-replica storage (at least three replicas). The storage space is consumed by all replicas. When creating an instance, specify its storage space based on the expected service message size and the number of replicas.
For example, if the estimated message size is 100 GB, the disk capacity must be at least: 100 GB x 3 + 100 GB (reserved space).
How Do I Select the Bandwidth for an Instance?
The network bandwidth of a Kafka instance is the unidirectional (read or write) maximum bandwidth. We recommend that you reserve 30% bandwidth to ensure that your applications run smoothly.
l 100MB/s: recommended when the service traffic is less than 70 MB
l 300MB/s: recommended when the service traffic is less than 210 MB
Does Kafka Support Disk Encryption?
No. Distributed Message Service (Kafka) does not support disk encryption.
Does Kafka Expansion Affect Services?
The expansion process of Kafka may have a certain impact on services, depending on your expansion strategy and implementation method. Possible effects:
l Re-partition and re-allocation: To expand a Kafka cluster, you may need to re-partition data and re-allocate replicas. Kafka needs to re-allocate replicas, re-partition data, and may need to reload and rebalance consumer groups. This process may lead to performance degradation and delay increase for a period of time.
l Increased network and disk load: The nodes and replicas added during Kafka expansion may increase the network and disk load. The replication and synchronization of data consume network bandwidth, and the added nodes and replicas may increase the write and read pressure on the disks.
Is Data Migrated During Capacity Expansion?
Data migration is not required during expansion.
How to Select and Configure a Security Group?
A security group is a network security mechanism used to control network access to VM instances in a cloud computing platform. It is a virtual firewall that defines inbound and outbound traffic rules. You can use security groups to protect VM instances. You can select security groups on the page for purchasing instances.
Does a Kafka Instance Support Access Across Regions?
Yes. You can access a Kafka instance across regions over a public network after purchasing and binding an elastic IP with a public IP address or by using Express Connect.
Do Kafka Instances Support Cross-Subnet Access?
Yes. Cross-subnet access is supported within the same VPC. By default, subnets in the same VPC can communicate with each other.
Can I Change the Private Network Addresses of a Kafka Instance?
The connection address cannot be changed or specified.
How Do I Obtain the Public Access Address After Public Access Is Enabled?
Log in to the Kafka console. In the action column of the instance list page, click Set Public IP of the target instance to view the specific address.
Does Kafka Support Authentication on Clients by the Broker?
Kafka does not support authentication on clients by the broker.
Are SSL Certificates Used in Different Instances the Same?
In Kafka, the SSL certificates used by instances are usually different. An SSL certificate is a security mechanism used to encrypt and verify communications between Kafka instances. Each Kafka instance needs a certificate to ensure the security of communication and perform authentication.
An SSL certificate has a public key and a private key. The public key is used to encrypt the communication process and the private key is used to decrypt the communication process. To ensure security, the private key must be confidential and can only be accessible by Kafka instances.
In a Kafka cluster, each instance has a certificate and a private key pair. This offers each instance with an independent encryption and authentication mechanism. If multiple instances share the same certificate and private key, the private key of an instance may be accessed by other instances, compromising the security of communication.
To secure communication between Kafka instances, we recommend you to generate a separate SSL certificate and private key for each instance. This provides each instance with an independent encryption and authentication mechanism, Improving the security of the entire Kafka clust