What Are Big Keys and Hot Keys?
Definitions of Big Keys and Hot Keys
Term | Definition |
Big keys: | There are two types of big keys: * If the size of a single String key exceeds 1 MB, the key is defined as a big key. * If the number of elements in a set key exceeds 3,000, the key is defined as a big key. |
Hot keys: | A key is defined as a hot key if the number of requests per second exceeds 3,000; |
What Is the Impact of Big Keys or Hot Keys?
Type | Impacts |
Big keys: | * Latency of client command execution increases during a query. * A shard uses a larger memory than other shards, and resources are unbalanced. * Frequently reading data from big keys exhausts the outbound bandwidth of the instance, triggering flow control and affecting services.* If a shard uses up the memory, keys on this shard are evicted, and memory overflow may occur. |
Hot keys: | *Hot keys occupy CPU resources and slow down other requests and the overall performance.* Cache breakdown may occur. If Redis cannot handle the pressure on hot keys, requests will hit the database. The database may break down as its load increases dramatically, affecting other services.* If a cluster shard has hot keys, there may be performance bottlenecks on the shard, increasing request pressure. |
How Do I Avoid Big Keys and Hot Keys?
Type | Suggested |
Big keys: | * Keep the size of strings within 1 MB.* Keep the quantity of set members within 3,000.* Use proper data structures. For example, you can split complex data structures such as JSON into multiple small keys. * Set a proper value for the expiration time of keys. |
Hot keys: | * Use the local cache for clients. Frequently used data is obtained from the local cache first. In this way, the pressure on Redis can be shared. |
How Do I Analyze the Hot Keys of a Redis 3.0 Instance?
Redis 3.0 does not support hot key analysis on the console. Alternatively, you can use the following methods to analyze hot keys:
Methods | Description |
Identify hot keys at the business layer | This method allows you to add code to the service layer to record requests that were sent to Redis instances and asynchronously analyze the collected statistics.* Advantages: Locates hot keys at the service layer quickly. Disadvantages: Requires familiarity with the service system logic. |
Run the MONITOR command. | You can run the MONITOR command to monitor requests between clients and the Redis server in real time. You can use the MONITOR command with redirection operators to record the monitored commands in a log file. You can then identify hot keys by analyzing the request records in the log file.* Advantages: The operation is simple and easy. * Disadvantages: The MONITOR command continuously outputs all processed commands, affecting the performance of Redis servers. Be careful when running the MONITOR command in production environments. |
How Do I Detect Big Keys and Hot Keys in Advance?
Methods | Description |
Using the analysis tool on the DCS console | See relevant sections on top key analysis and offline full key analysis. |
Identify big keys and hot keys by using the bigkeys and hotkeys parameters in redis-cli.
| *Check for big keys: The bigkeys parameter provided by Redis allows redis-cli to traverse all keys in a Redis instance and return the statistics of keys and the biggest keys of six data types: STRING, LIST, HASH, SET, ZSET and STREAM. Sample command: redis-cli -h <connection address of the instance> -p <port> -a <password> --bigkeys. *Check for hot keys: Since Redis 4.0, the hotkeys parameter is provided to help you quickly identify hot keys. Sample command: redis-cli -h <connection address of the instance> -p <port> -a <password> --hotkeys. You can obtain details of hot keys in the summary of results. |
Analyze a specified key by using built-in commands of Redis.
| The following section lists low-risk commands for analyzing keys of various data types to determine whether a key is a big key. For a string key, you can run the STRLEN command. This command returns the length (number of bytes) of a string value stored in the key. For a list key, you can run the LLEN command. This command returns the length of a list value stored in the key. For a hash key, you can run the HLEN command. This command returns the number of members in the key. For a set key, you can run the SCARD command. This command returns the number of members in the key. For a ZSET key, you can run the ZCARD command. This command returns the number of members in the key. For a stream key, you can run the XLEN command. This command returns the number of members in the key. |
Identify hot keys at the business layer | This method allows you to add code to the business layer to record requests that were sent to Redis instances and asynchronously analyze the collected statistics. |
How Does DCS Delete Expired Keys?
Mechanisms for Deleting Expired Keys:
1. Lazy free deletion: Before a read/write command is executed, Redis checks whether the key to be accessed has expired. If it has expired, it will be deleted and a response will be returned indicating that the key does not exist. If the key has not expired, the command execution resumes.
2. Scheduled deletion: Redis periodically deletes expired keys using a scheduled task. The task is executed at certain intervals. Each time the task is executed, a random collection of keys is checked, and expired keys are deleted.
The two mechanisms can be used together to timely clean up expired data in Redis and optimize cache performance.
How Do I Query Expired Keys?
Run the TTL key_name command to query the expiration time of the key named key_name. In the command output:
-1 indicates that the key is not configured with expiration time and is stored permanently.
-2 indicates that the key has expired or has been deleted.
An integer whose value is >=0 indicates the expiration time of the key.
On the DCS console, you can delete expired keys immediately or regularly. For details, see section on how to delete expired keys.
How Long Are Keys Stored? How Do I Set Key Expiration?
How Long Are Keys Stored?
Keys that do not have an expiration date are stored permanently.
Keys that have an expiration date are deleted after they expire. Keys cannot be accessed when they expire.
You can run the TTL KEY_NAME command to query the expiration time of a key.
How Do I Set Key Expiration?
You can run the expire or pexpire command to set the key expiration time. For example, if you run expire key_x 100, key_x will expire in 100 seconds. expire sets key expiration in seconds, and pexpire sets key expiration in milliseconds.
You can also run the SETEX key seconds value command.
You can query the expiration time of a key through TTL.
Why Does Memory Usage Decrease After Big Key Analysis Is Performed on Redis?
Due to the lazy free deletion mechanism, keys are not deleted immediately upon expiration, unless they are accessed. During a big key analysis, all keys are traversed, and the expired keys will be deleted. As a result, memory usage decreases after a big key analysis is completed.