Message Production
l Message compression: Larger messages are consumed and then sent to the server, maximizing bandwidth utilization.
l Delayed messages: After you set the consumption delay, messages delivered to the server can only be consumed after the preset time elapses.
l Transactional messages: Based on the preset transactions, transactional messages ensure the eventual consistency of data between distributed systems.
Message Consumption
l Ordered consumption: Normal ordered and strictly ordered messages.
l Cluster consumption: A topic can be consumed by one or more consumer groups, and consumer instances in the consumer group can evenly share the messages.
l Consumption position setting: You can set the position where the consumer group starts the first consumption, namely, from the queue head, queue tail, and the position specified by the client.
l Message tracking: You can track your consumption progress over time and reset the consumption progress of the subscription group on a certain topic to a past or future time point.
Comprehensive O&M
l Application user management: You can isolate cluster tenants and manage the permissions for applications to access a cluster.
l Topic management: You can manage topics in an instance and perform operations such as creation and deletion.
l Subscription group management: You can manage the subscription groups in an instance.
l Producer and consumer management: You can view the producer and consumer information under the current instance and update it in real time.
l Message query: Query by message ID, message logical offset, and message key
l Comprehensive O&M: node status detection, node start and stop; instance status detection, instance start and stop; SLA monitoring.
Ordered Messages
Ordered messages: Messages that are consumed in the exact order that they are created. RocketMQ has two types of ordered messages: globally ordered messages and locally ordered messages (also known as normal ordered messages, or partitioned ordered messages).
Normal ordered messages: Under normal circumstances, ordered messages can be guaranteed. However, once the broker is restarted due to a communication fault, the total number of queues changes, and the queues will change after Hash Modulo and positioning. As a result, a temporary message order inconsistency occurs. If a temporary message disorder is acceptable for the business in case of a cluster fault (such as a broker shutdown or restart), normal ordered messages are applicable.
Strictly ordered messages: The message order is guaranteed in both normal and abnormal conditions, but a failover is not supported. Specifically, if one machine in the broker cluster fails, the entire cluster is unavailable (or the queue corresponding to the hash value is affected), and the service availability is greatly compromised. If the server is deployed in synchronous dual-write mode, the slave node automatically switches to the master node. However, the service will still be unavailable for a few minutes.
Transaction Message
DMS RocketMQ provides distributed transaction similar to X/Open XA. After half-transaction messages are sent, they are judged based on the preset transaction. Satisfactory messages will be confirmed by the server, and unsatisfactory ones will not be received by the server to ensure the eventual consistency of the produced messages and local transactions in the distributed scenarios.
Half messages: They are temporarily undeliverable messages. The sender has successfully sent a message to the message queue server, but the server has not received a secondary confirmation for the message from the producer. The message is marked as "temporarily undeliverable", and is called a half message.
Message check: Due to network crashes, producer application restarts, and other reasons, the secondary confirmation for a transaction message is lost. When discovering that a message remains as a "half-message" for a long time through scanning, DMS RocketMQ actively inquires with the message producer about the final status of the message (commit or rollback). The entire process is called message tracking.
Delayed Messages
The producer sends the messages to the RocketMQ server and sets the consumption delay. The messages can only be consumed by consumers after the preset time elapses. If you want to send a delayed message, specify a period of time that starts from the point in time when the message is sent by a producer. The broker sends the message to consumers after the specified period of time elapses. You can use delayed messages to implement delayed scheduling and triggering in distributed scenarios.
Clustering Consumption/Broadcasting Consumption
Broadcasting consumption: In broadcasting consumption mode, when multiple consumers in a consumer group consume a message, RocketMQ ensures that the message is consumed by each consumer at least once.
In clustering consumption mode, the consumers in one or multiple group consume messages from a specific topic. Each consumer group has an independent consumption progress, which is saved on the server. The consumers in a consumer group can evenly consume messages to achieve load balancing. For example, a topic has nine messages and a consumer group has three different consumers (possibly three processes or three machines). Each consumer consumes only three messages. This consumption mode achieves point-to-point consumption and broadcasting consumption in JMS, and is recommended because it meets the requirements of most scenarios.
Message Retry
Ordered messages must be received in order. When a consumer fails to consume a message, the RocketMQ will automatically perform a retry on the message at an interval of one second. At this point, other messages have to wait for consumption. When you are using ordered messages, it is recommended to ensure that the application can promptly monitor and handle consumption failures to avoid blocking.
For unordered messages, RocketMQ allows a maximum of 16 retries per message by default.
The intervals for retries are shown as follows:
Nth Retry | Time Since the Last Retry |
1 | 10s |
2 | 30s |
3 | 1 minute |
4 | 2 minutes |
5 | 3 minutes |
6 | 4 minutes |
7 | 5 minutes |
8 | 6 minutes |
9 | 7 minutes |
10 | 8 minutes |
11 | 9 minutes |
12 | 10 minutes |
13 | 20 minutes |
14 | 30 minutes |
15 | 1 hour |
16 | 2 hours |
If a message fails after 16 retries, it will no longer be delivered. If calculated strictly according to the listed retry intervals, a message will be retried 16 times in the next 4 hours and 46 minutes if the consumption fails. Messages beyond this time range will not be re-delivered.
Filtering Messages
After a consumer subscribes to a topic, RocketMQ delivers all messages in the topic to the consumer. If the consumer needs only some messages from the topic, you can configure filter conditions to get only desired message subsets, avoiding receiving a large number of invalid messages.
Message filtering involves the following steps:
l Producer: Before a producer initializes messages, the producer attaches attributes and tags that are used to match the filtering conditions configured by a consumer to the messages.
l Consumer: During message initialization and consumption, a consumer needs to inform the server of the messages to which the consumer wants to subscribe in a specific topic. This is also known as reporting the filtering conditions.
l Broker: The RocketMQ server matches messages based on the expressions of the reported filtering conditions and delivers the messages that meet the filtering conditions to the consumer.
RocketMQ supports the tag-based filtering and attribute-based SQL filtering methods.
l Tag-based filtering: Messages are matched based on the tags that are defined by producers. Consumers use the tags to specify the messages that they want to consume.
l Attribute-based SQL filtering: Messages are matched based on the key-value pair that is specified by producers. When producers send messages, the producers can specify multiple attributes for each message. Then, the consumer can specify attributes in SQL expressions to receive specific messages.