Distributed Message Service (Kafka), or DMS (Kafka), is a distributed, high-throughput, and high-availability message queuing service that provides fully managed services based on open-source Kafka. It solves the long-standing pain points of open-source products. Users only need to focus on business development, without caring about O&M and deployment. With the features of low cost, greater elasticity and larger reliability, it is widely used in log collection, monitoring data aggregation, streaming data processing, online and offline analysis and other big data fields. So it is an integral part of the big data ecosystem.

Announcements on Kafka User Guide

Due to length limitations, this documentation introduces Distributed Message Service (Kafka) and its differences from Apache Kafka, such as the specifications, console operations, API calls, and client access to instances of eSurfing Cloud DMS (Kafka).

For more information about the basic knowledge of Kafka or technical details about producing and consuming messages, see the official Apache Kafka website.

Architecture

l Broker: A message-oriented middleware or processing node. A Kafka node is a broker, and multiple brokers constitute a Kafka cluster.

l Topic: A collection of messages of the same type.

l Partition: A topic can be physically divided into multiple partitions, and each partition is an ordered queue.

l Segment: A partition is physically composed of multiple segments.

l Offset: Each partition consists of an ordered, immutable series of messages that are successively appended to the partition. Each message in the partition has a sequential serial number called an offset, which is used to uniquely identify a message in the partition.

l Producer: A message and data generator that calls APIs to produce messages and publishes messages to Kafka topics.

l Consumer: A message subscriber, also known as a message consumer, is responsible for reading and consuming messages from a Kafka broker.

l Consumer Group: The collection of a type of consumers. This type of consumers usually consumes the same type of messages with the same consumption logic. The relationship between a consumer group and a topic is N:N, meaning that the same consumer group can subscribe to multiple topics, and the same topic can also be subscribed to by multiple consumer groups.

For more information, see Explanation of Terms.

Comparison with Open-Source Kafka

Compared with self-built Kafka based on the open-source version, Distributed Message Service (Kafka) has advantages in terms of low O&M costs, partition scale, message query, access control through ACLs, visual configuration, O&M monitoring, cluster inspection, stability, reliability, security, and ease of use. For more information, see Comparison with Open-Source Kafka.

Benefits

DMS (Kafka) is an ideal choice for large-scale data processing and real-time stream processing because it is highly available, secure, reliable, and fully managed. For more information, see Benefits.

Features

The features of Distributed Message Service (Kafka) are mainly reflected in the following aspects:

Messaging Capabilities

Message Broadcasting: Deliver the same message to all consumers within a consumer group.

Message backtracking: You can reset the consumption progress based on time.

Automatic message deletion: When the disk is full, data outside the protection period is automatically deleted to ensure service availability.

Automatic failover: Automatic load balancing of production and consumption and automatic active/standby failover when a message node fails to guarantee business continuity.

Queue Capacity

Feature high throughput, and realize asynchronous replication of multiple copies of messages.

Feature high reliability, and realize synchronous replication of multiple copies of messages.

Visual Management

Application user management: Multiple applications can call the same message service to manage the access permissions of the application user.

Topic management: You can manage topics in an instance and perform operations such as creation and deletion.

Consumer group management: You can manage consumer groups in an instance.

Broker monitoring: Broker details and multi-dimensional monitoring metrics are provided.

Topic monitoring: Topic details and multi-dimensional monitoring metrics are provided.

Security Protection

Provide traceable tenant management operation records.

Provide user authentication, SASL-authorized access mechanism, and enterprise-level security protection.

For more information, see Features.

Application Scenarios

Distributed Message Service (Kafka) can be applied to industries such as the Internet of Things (IoT), telecommunications, e-commerce, and financial services, and is typically used in scenarios such as streaming computing, and log aggregation. For more information, see Application Scenarios.

Limitations

Distributed Message Service (Kafka) has limits on objects such as instances and topics. Be careful not to exceed the limits to prevent the program from exiting unexpectedly. For more information, see Limits.