Troubleshooting Method
According to the criteria of nine aspects to consider, the processing steps of connection failure are organized as follows:
Excluding database instance exceptions:
Check whether the DDS system fails and confirm whether the instance status is normal.
Verify that no other process or user has locked the instance or table.
Using correct client connection method:
Make sure to use the correct version of the MongoDB client, version 4.0 or later is recommended.
For Intranet connection, ensure that DDS and ECS instance are in the same region and VPC.
For Internet connection, make sure to purchase or use the existing EIP and bind it to the DDS instance.
Using correct SSL method to connect securely:
Confirm the status of the SSL switch and use the correct connection command based on the switch status.
Make sure that the correct SSL certificate file path is provided.
Excluding connection command error:
Check whether the connection address, port, username, and password are correctly configured.
In SSL mode, check whether SSL-related parameters are correctly set in the connection command.
Excluding network failure:
For Intranet access, check whether the ECS instance and the DDS instance are in the same region and VPC. If not, establish a VPC peering connection.
Check the security group rules to ensure that the relevant ports of the DDS instance are allowed to pass.
For Internet access, check the security group and network ACL rules to ensure that access to the relevant ports of the DDS is allowed.
Test whether ECS can ping the port of DDS instance address.
Cross-CIDR block access (configure IP mapping):
Confirm whether cross-CIDR block access configurations are required, and configure the network accordingly.
Excluding the case where number of instance connections is full:
Check the number of connections to the instance to ensure that the number of connections does not reach the upper limit.
Excluding the case where disk space is full:
Check the instance disk utilization to ensure that the disk is not full, which may cause the instance to be unable to connect properly.
Excluding the issue of CPU usage spikes:
Check the CPU utilization of the instance to ensure that the CPU is running properly to avoid affecting data read/write processing and connection.
Troubleshooting Steps
Confirm database instance status:
• Check whether the DDS instance is running normally through the DDS console.
Verify connection string:
• Check whether the connection string is correct, including the correct host name (or IP address), port number, user name and password.
Check network connection:
• Make sure your network connection is working properly and try to access other websites or services to verify if the network is working.
Check security groups and firewall rules:
• Make sure security groups and firewall rules allow ports related to the DDS instance, especially if Internet access is involved.
Verify SSL configuration (if used):
• If SSL is used for the connection, check whether SSL is enabled in the Basic Information of the DDS console and upload the certificate to the ECS instance.
• The connection command is as follows:
./mongo --ssl --host <DB_HOST> --port <DB_PORT> -u <DB_USER> -p --authenticationDatabase <AUTH_DB> --sslCAFile <CA_FILE_PATH> --sslAllowInvalidHostnames
• Where, the meaning of each parameter is as follows:
– --ssl: Enable SSL connection.
– --host <DB_HOST>: Specifies the host name or IP address of the MongoDB database.
– --port <DB_PORT>: Specifies the port number of the MongoDB database.
– -u <DB_USER>: Specify the MongoDB username to be used for authentication.
– -p: Indicates entering a password in the command line. After entering this command, you will be prompted to enter the password of MongoDB user.
– --authenticationDatabase <AUTH_DB>: Specifies the database to be used for authentication. Specify here the database where the authenticated user is located.
– --sslCAFile <CA_FILE_PATH>: Specifies the path of the Certificate Authority (CA) file for the SSL connection. This file is used to verify the legitimacy of the server-side certificate.
– --sslAllowInvalidHostnames: Allows connection to an SSL server without verifying the server's hostname. This option is used to ignore host name mismatch errors in SSL certificates when connecting.
You need to replace <DB_HOST>, <DB_PORT>, <DB_USER>, <AUTH_DB>, and <CA_FILE_PATH> with the actual MongoDB database's hostname, port number, username, authentication database, and path to the SSL CA file.
Confirm user permissions:
• Ensure that the username and password you use have permission to connect to the DDS instance.
Try using other client tools:
• If the connection failure is due to a client tool issue, try connecting using a different client tool to troubleshoot the client tool itself.
View error logs:
• Check the error logs or messages generated when the connection fails, which may help identify the cause of the problem.
Check number of connections and resource usage:
• Confirm whether the number of connections to the document database service instance has reached the upper limit, and whether the disk and CPU usage are normal.
Verify Intranet/Internet access configuration:
• For Intranet access, confirm whether the DDS instance and the DDS instance are in the same region and VPC. If not, check the VPC peering connection configuration.
• For Internet access, ensure that the security group and network ACL rules of the ECS instance and the DDS instance allow corresponding ports to pass, and verify whether ECS can ping the port of the DDS instance address.
Exclude other factors:
• Exclude other factors that may affect the connection, such as blocked processes, and resource contention.
How to deal with connection failures caused by reaching the maximum limit of instance connections
Issue Description
• Error when connecting to DDS via Python:
– pymongo.errors.ServerSelectionTimeoutError: Indicates that the MongoDB driver for Java (pymongo) is unable to select an available DDS server, resulting in connection timeout.
• Error when connecting to DDS via Java:
– com.mongodb.MongoTimeoutException: Indicates that the MongoDB driver for Java is unable to connect to the DDS instance within the specified time, resulting in connection timeout.
Possible Reasons
• Sharded Cluster:
– There are too many read and write requests, which exceeds the maximum number of connections in the sharded cluster.
– Some sharded servers are overloaded and unable to handle more connection requests.
– The sharding key is improperly designed, resulting in unbalanced load in some sharded clusters.
• Replica Set:
– There are too many read and write requests, which exceeds the maximum number of connections to the replica set.
– The primary node in the replica set is overloaded and cannot handle more connection requests.
• Single-Node Instance:
– There are too many read and write requests, which exceeds the maximum number of connections to the single-node instance.
Solution
• Check whether service connections are valid in time, optimize instance connections, and release unnecessary connections.
• Check the value of the net.maxIncomingConnections parameter and the instance specification, modify the parameter value, or expand the database specification.
• View monitoring information, CPU, memory, disk, connection count, and other metrics on the Basic Information page on the DDS console. You can set alarm policies in the Alarm Center of the console to identify risks in advance when alarms occur.
Solution
Optimize instance connections, and release unnecessary connections.
• You can temporarily release all connections by restarting the instance.
• You can query the current number of connections to a node, as well as the current source of connections, analyze how many connections have been established for each terminal and DDS instance, and make corresponding adjustments.
db.serverStatus().connections
{ "current" : 7, "available" : 398, "totalCreated" : 818364 }
#You need to pay attention to the following parameters and corresponding values:
current: The number of connections currently established.
available: The number of connections currently available.Modify the parameter value, or expand the database specification.
• Adjust the value of connPoolMaxConnsPerHost on the DDS parameter setting page, and restart the instance to take effect after modification.
• If the number of adjusted connections is too large, it may cause the service to crash. In this case, the only way to increase the number of connections carried is to increase the number of connections by changing the instance specification.
How to retrieve and control the current number of connections
Query and modify the value of the current number of connections
The maximum number of connections varies depending on the specifications of the purchased DDS instance.
Log in to the DDS Management Console to query the current number of connections:
On the Instance list page, click the ID of the instance whose running status is Running.
Go to the instance details page.
Click on the Parameter Settings menu bar, and the value of connPoolMaxConnsPerHost parameter allows you to view the configuration value of current instance connection count.
The number of connections can be limited by modifying the connPoolMaxConnsPerHost parameter.
What to do if ECS and DDS are placed in a disconnected VPC environment
For how to switch the VPC of the ECS instance to the same VPC as the DDS instance, see ECS - User Guide - NIC - Switching VPC.
Does the application need to support automatic database reconnection
Yes, the application should support automatic database reconnection after DDS is restarted. The restart of DDS may be scheduled maintenance operation or may be due to abnormal situation, in either case, the database connection may be interrupted. To ensure the continuity and reliability of the application, it should have the ability to automatically reconnect.
It is also recommended that your application uses long connections to the database to reduce resource consumption and improve performance.
How to create and log in to ECS
For details about how to create and log in to ECS, see ECS - Getting Started - Create ECS and ECS - Getting Started - Log in to ECS.
• The ECS is used to connect to the DDS instance and must be in the same VPC subnet as the instance to be connected.
• When you create an ECS, select the operating system, such as centOS 7.6, and bind an EIP to the ECS.
• Configure the security group correctly so that the ECS can directly connect to DDS.