What is DTS?
eSurfing Cloud DTS works as the base for data transfer and incorporates data migration and real-time data synchronization. It helps you migrate database data to the cloud rapidly and stably without interrupting business. In addition, DTS provides real-time data synchronization channels to realize database disaster recovery or active-active disaster recovery. DTS provides multiple features, such as data migration and real-time synchronization across multiple types of data sources, data masking, and enterprise-grade encryption. By using a self-developed synchronization engine, DTS provides a synchronization performance more than four times that provided by the native synchronization feature of databases.
What are a region and an availability zone?
Region
A region is the physical location in which a document database service is deployed.
Only availability zones within a region can connect to each other over an internal network.
The public cloud provides data centers in different regions for which document database services can be used. By activating document database services in different regions, you can align your application designs with the requirements of specific users, laws in different regions, or other requirements.
Availability Zone
Each region includes different zones known as availability zones. An availability zone is a physical zone that has an independent power grid and network. Availability zones are interconnected over an internal network and physically isolated. An availability zone is not affected by failures in other availability zones. To connect to other availability zones within the same region, an availability zone provides cost-effective and low-latency network connections. By using a document database service deployed in an independent availability zone, you can prevent a failure in another availability zone from affecting your application. DTS allows you to deploy your backup set instance across three availability zones. This means you can deploy the primary, standby, and hidden nodes across the availability zones, which achieves disaster recovery across availability zones.
Does DTS allow HA instance migration for relational database services?
Currently, if the following requirements are met, DTS allows high availability (HA) instance migration from a MySQL database instance.
The global transaction identifier (GTID) mode is enabled for your source database instance.
The virtual IP addresses (VIPs) of your HA instances are provided.
What are required for source databases by DTS?
In the current public preview, DTS supports data migration/synchronization between MySQL and data migration between PostgreSQL. DTS has different requirements for source databases in different scenarios:
Data migration between MySQL: DTS supports RDS MySQL databases and self-managed databases of MySQL 5.6/5.7/8.0. The version of a self-managed database must be equal to or earlier than that of a destination database.
Data migration between PostgreSQL databases: DTS supports RDS PostgreSQL databases and self-managed PostgreSQL 12 databases. The version of a source database must be equal to or earlier than that of a destination database.
Data synchronization between MySQL: DTS supports RDS MySQL databases and self-managed databases of MySQL 5.6/5.7/8.0. The version of a source database must be equal to or earlier than that of a destination database.
What are required for destination databases by DTS?
In the current public preview, DTS supports data migration/synchronization between MySQL and data migration between PostgreSQL. DTS has different requirements for destination databases in different scenarios:
Data migration between MySQL: DTS supports RDS MySQL databases. The version of a destination database must be equal to or later than that of a source self-managed database.
Data migration between PostgreSQL databases: DTS supports RDS PostgreSQL databases. The version of a source database must be equal to or earlier than that of a destination database.
Data synchronization between MySQL: DTS supports RDS MySQL databases. The version of a destination database must be equal to or later than that of a source database.
Does DTS use concurrency technologies?
DTS uses concurrency technologies for full data migration, including:
Concurrent multi-table migrations. DTS can migrate data in multiple tables based on your configuration.
Concurrent data writes to a destination database from a single table. When DTS migrates a single table, it uses multiple threads to concurrently write data in batches to the destination database. This helps solve the imbalance of read and write speed.
Does DTS support data migration between the public cloud and a private cloud?
Currently, DTS supports data migration from a private cloud to eSurfing Cloud (inbound data migration). It does not support data migration from eSurfing Cloud to a private cloud (outbound data migration).
Does DTS support migration from MySQL to PostgreSQL?
No, DTS does not support this type of migration. In the current public preview, DTS supports the following links:
Data migration: MySQL -> MySQL and PostgreSQL -> PostgreSQL.
Data synchronization: MySQL -> MySQL.
Does DTS support replicating data within a specific time range?
No, DTS does not support this operation.
Does DTS support resumable transfer?
Yes, DTS supports this feature.
Full data migration provided by DTS supports table-level and row-level resumable transfer under certain conditions. When a task is paused or restarts unexpectedly, a migrated table will not be migrated again. If relevant conditions are met, DTS resumes the migration from the row with the breakpoint by default.
Incremental data migration provided by DTS supports resumable transfer by recording offsets. When a task is paused or restarts unexpectedly, DTS starts to migrate incremental data from the last recorded offset.
What is the difference between data migration and data synchronization?
Data migration: migrates data to eSurfing Cloud. For example, you can migrate an on-premises database, a self-managed database hosted on ECS, or a third-party cloud database to an eSurfing Cloud database. A data migration task is a one-off job. After your data migration task is complete, you can release the DTS instance.
Data synchronization: synchronizes data between two data sources in real time. It is suitable for scenarios such as active geo-redundancy, disaster recovery, and real-time data warehousing. A data synchronization task is a continuous job. After you create such a task, the task continuously synchronizes data to keep data consistency between two data sources.
What Can I Do If Data Bloat Occurs When I Use DTS?
When a write operation is performed on a database, write amplification occurs. This means the volume of data written to a disk is larger than that of the migrated data. Write amplification is caused by multiple techniques used in modern database systems, such as Copy on Write (COW) and logging. Therefore, you are recommended to reserve 1.5 times the size of your source database data for the server disk space of your destination database.
How Does DTS Select a Read-only RDS Instance?
DTS cannot automatically select a read-only instance. The instance from which DTS reads data during migration depends on the IP address and port or instance information configured for your source database.
How Does DTS Affect Source and Destination Databases?
Full data migration: Full data migration has large effects on source and destination databases. It occupies disk I/O and network bandwidth resources of all the servers of many databases. Select a proper period for full data migration after a careful assessment. You are recommended to migrate full data during off-peak hours.
Incremental data migration: Incremental data migration only has small effects on source and destination databases.
Do I Need to Stop the Source Database Service When I Use DTS?
No, you do not need to stop the source database service.
DTS provides the hot migration feature by combining full data migration and incremental data migration. DTS maximizes business continuity and minimizes its effects on the source database service. Therefore, you do not need to stop the source database service.
What Are the Factors that Affect the DTS Task Speed and How Can I Estimate the Time Required to Run My Task?
Factors of Affecting the Migration Speed
If the source database provides a higher read throughput, the migration is faster, which means less time is required to run the migration task.
If the destination database provides a higher write throughput, the migration is faster, which means less time is required to run the migration task.
If more network bandwidth resources are available, the migration is faster, which means less time is required to run the migration task.
If higher network quality and a lower network latency are ensured, the migration is faster, which means less time is required to run the migration task.
If a DTS instance of a higher specification is used, the migration is faster, which means less time is required to run the migration task.
If the source database generates incremental data at a higher speed, the DTS task takes longer time to keep up with the increment.
If you separate one DTS task into multiple migration tasks by table, you can improve the overall migration performance.
Estimation of Migration Time
Multiple factors affect the migration speed, making it challenging to have a one-size-fits-all method to estimate the migration time.
Can I Change the Objects to be Migrated/Synchronized in a DTS task?
Data migration task: If you create a task and you do not start it, you can change the objects to be migrated. If the task is already started, you cannot change the objects.
Data synchronization task: If you create a task and you do not start it, you can change the objects to be synchronized. If the task is already started, you cannot change the objects.
Can DTS Synchronize Data Between Different Databases in One Instance?
Yes, DTS can synchronize data between different databases in one instance. DTS provides the object name mapping feature for real-time synchronization tasks. You can change the object names for the destination database so that each object has different names in your source database and destination database.
What Operations on Source/Destination Databases Can Affect the DTS Task Status?
The following part takes eSurfing Cloud RDS MySQL as an example to describe the operations that may affect the DTS task status:
Database instance exception: DTS automatically retries to establish a connection. If the retry fails, you can restart your DTS task and resume migration after your database instance is recovered.
Database instance update: A database kernel update restarts an instance, causing that DTS is disconnected from your database instance within a short period. During the period, DTS retries to establish a connection. If the retry fails, you can restart your DTS task and resume migration after your database instance is recovered.
Database instance restart: A database instance restart causes that DTS is temporarily disconnected from your database instance. During the period, DTS retries to establish a connection. If the retry fails, you can restart your DTS task and resume migration after your database instance is recovered.
Database instance specification change: A specification change restarts a database, causing that DTS is temporarily disconnected from your database instance. During the period, DTS retries to establish a connection. If the retry fails, you can restart your DTS task and resume migration after your database instance is recovered.
Limit on the number of connections for a session: A DTS task requires a certain number of connections between the source and the destination. If the number of database connections does not meet the requirements, the DTS task may fail. If it fails, you can restart the task and resume migration after you change the number of database connections.
Change in the user password for a database instance: If you change a database user password, DTS may fail to connect to your database instance. If it fails to connect to your database instance, you can restart the task and resume migration after you pause your task and change the password to the original one.
Modification of the user permissions on a database instance: If you modify the user permissions on a database, DTS may not have the required permissions, causing the task to fail. In this case, you can restart the task and resume migration after you grant the required permissions to the user that runs the task.
Deletion of source database logs: If source database binlogs are deleted, DTS cannot obtain the logs generated after the synchronization offset from the source database. This causes the task to fail. For example, the incremental data synchronization step of the task fails. In this case, you can recreate a task that synchronizes data from the new log offset.
Modification of database parameters: A DTS task checks the necessary parameters of source and destination databases during the precheck phase before the task starts. After the precheck is completed, you are not recommended to modify the database parameters before the task ends. This avoids a task failure caused by parameter modifications. If the task fails due to parameter modifications, you can restart the task and resume migration after you reset the modified parameters to the original settings.
Network jitter: If DTS cannot connect to your database due to network jitter, DTS automatically retries to establish a connection. If the retry fails, the DTS task status changes to Abnormal. In this case, you can restart the task and resume migration after the network is recovered.
Does DTS Supports Using the Standby Database of a Read-only Instance as the Source Database?
DTS does not need to write data to a source database. However, in incremental data migration/synchronization scenarios, DTS needs to read the incremental data logs of the source database, such as MySQL binlogs. In these scenarios, you must ensure the integrity of logs in the read-only instance.