Elasticsearch is an efficient distributed search and analytics engine, applicable to storing, retrieving, and analyzing large-scale data. To ensure optimal performance, stability, and scalability, here are some best practices for Elasticsearch.
1. Cluster Planning
Number of Nodes
The number and distribution of nodes play a key role in constructing an Elasticsearch cluster. An odd number of nodes is typically preferred for better handling of master node elections and failovers. For example, a cluster with three master nodes can maintain functionality even if two nodes fail.
Single-node clusters should be avoided as they can lead to single points of failure.
Shards and Replicas
Shards are horizontal partitions of data within the cluster. When creating an index, select an appropriate number of shards based on data amount and query load, ideally no more than the number of nodes. Then, to assure data availability, set a sufficient number of replicas, but avoid setting too many to prevent resource waste.
Example:
PUT /my_index
{
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
}
}
2. Index Design
Settings for Shards and Replicas
When creating an index, select an appropriate number of shards based on data amount and query load.
For frequently accessed indexes, consider increasing the number of replicas to enhance query performance.
Index Fields
Make wise choices for field types and avoid unnecessarily complex types. For example, use the keyword type for exact values, and the text type for full-text search. Example:
PUT /my_index
{
"mappings": {"properties": {
"product_name": {
"type": "text"
},
"category": {
"type": "keyword"
},
"price": {
"type": "float"
}
}
}
}
Disable fields that are not required to be indexed to reduce index size and improve performance.
3. Write Operations
Bulk Write
Use the bulk API for write operations to minimize network overhead. In high-throughput write scenarios, consider moderately increasing the Refresh interval to reduce the number of index refresh operations and enhance write performance.
Example:
POST /my_index/_doc/_bulk
{ "index": {}}
{ "field1": "value1" }
{ "index": {}}
{ "field2": "value2" }
Index Refresh
Avoid frequent index refreshes. Appropriately adjust the Refresh interval to balance between write and query performance.
Example:
PUT /my_index/_settings
{
"refresh_interval": "30s"
}
4. Query Optimization
Query Performance
Use the query DSL for complex searches to take full advantage of the capabilities of Elasticsearch.
Enhance query performance by using index aliases and index templates.
Example:
GET /my_index/_search
{
"query": {
"bool": {
"must": [
{ "match": { "product_name": "apple" }},
{ "range": { "price": { "gte": 100 }}}
]
}
}
}
Pagination Query
For pagination queries involving a large amount of data, use Scroll to avoid performance issues associated with deep pagination.
Example:
POST /my_index/_search?scroll=5m
{
"query": { "match_all": {}},
"size": 100
}
5. Hardware and Monitoring
Hardware Selection
Select high-performance hardware, particularly fast disks and large memory.
Use SSD hard drives to enhance indexing and query performance.
Monitoring and Alarm
Establish monitoring metrics such as cluster health, node status, and resource usage.
Monitor the cluster continuously with monitoring tools and set up alarms for timely issue response.
6. Security
Authentication and Authorization
Enable security features and authenticate using usernames and passwords.
Restrict user access using roles and permissions.
Encrypted Communication
Enable TLS/SSL encryption to secure data during transmission.
7. Maintenance
Backup and Recovery
Ensure regular backups of index data to facilitate swift data recovery in case of unforeseen events.