Manage Transfer

Create a transfer

  1. In the list of services, select Transfer.

  2. Select the Transfer tab.

  3. Click Create transfer.

  4. Under Endpoints:

    1. Select the source endpoint in Source or create a new one.

    2. Select the target endpoint in Target or create a new one.

  5. Under Basic settings:

    1. Enter the transfer Name.

    2. (optional) Enter the transfer Description.

  6. Under Transfer settings select the Transfer type:

    1. Snapshot to make a one-time transfer of the database snapshot.

    2. Increment continuously retrieves changes from the source database and applies them to the target database.

    3. Snapshot and increment to transfer the data and keep the target database in sync with the source database.

  7. Specify Runtime properties:

    1. Set the Number of operations. This setting is designed to handle Apache Kafka® sources and sets the number of workers. Each worker in this case can copy multiple producers.

    2. In Serverless runtimeCopy settings, enter the values for the following settings:

      1. Jobs count sets the number of parallel instances of a container with a transfer. Increasing this number will speed up your transfer execution. You can specify up to 8 instances.

      2. Process count specifies the number of processes within each container. You can run up to 10 processes.

      The average download speed of data transfers is between 1 and 10 mb/s.

    1. Select VM performance:

      • small - 2 CPUs

      • medium - 4 CPUs

      • large - 8 CPUs

    2. Select the Minimum logging level:

      Level Description
      TRACE Detailed diagnostics information
      DEBUG Debugging information
      INFO Diagnostics information for statistics
      WARNING Warning about a non-critical malfunction. Please investigate further
      ERROR Error report
      FATAL Possible system failure, service shutdown
    3. Under Settings, specify how to select a network in which to run a virtual machine instance:

      • Automatic

        Allows the service to set the network automatically based on the target resource location.

      • Manual

        Select a VPC Network ID from the drop-down list.

    Read more about runtimes here: Runtime types in Transfer

  8. If needed, you can enable the Regular Copy schedule.

    1. Select the desired Interval.

    2. Add one or more Incremental tables. With incremental tables, you can transfer only the data that has changed. Each of these tables has the following fields:

      • Table namespace corresponds to the database schema (as in PostgreSQL), name, or dataset. Refer to the Airbyte - destination namespace configuration to learn more about namespaces.

      • Table name in the source to compare your target table with.

      • Cursor column is the name of a column that contains some value that indicates whether the table should be incremented. A common example of a column that contains a cursor is a column with timestamps. Refer to the Airbyte - Incremental Sync - Append to learn more about cursor definitions and usage examples.

  9. Set up Data transformations if you need to modify a list of tables to transfer. For instance, you don't want to transfer a column with passwords. In Transformer list, click Add transformer to create a set of transformation rules. Add this column's name to the Exclude columns section. In Transformer[number]Columns filter, configure transformations for tables and columns:

    • Include tables restricts the set of tables to transfer, whereas Exclude tables allow transferring all data except the specified tables. Set these table names as regular expressions.

    • Include columns sets the list of columns to transfer and Exclude columns makes the list of columns that will not be transferred. Set these table names as regular expressions.

    Each transformer is a separate set of rules and you can combine different rules within each set. For example, you can set a table in Include tables and an Exclude column. In this case, the service will ignore the specified Exclude columns only for the included table. If you combine Exclude tables and Include columns, only the specified columns will be transferred from all tables except those specified in theExclude tables field.

  10. Click Submit.

Activate the transfer

  1. In the list of services, select Transfer.

  2. Select the Transfer tab.

  3. Click the icon near the transfer name and select Activate.

Transfer data from the AWS networks

Transfer requires access to the resources in AWS that are usually protected by a set of tools. Refer to the following guide to see how to peer networks and configure DoubleCloud and AWS to successfully perform a transfer: Transfer data from a PostgreSQL RDS instance to ClickHouse®.

Update a transfer

  1. In the list of services, select Transfer.

  2. Open the Transfers tab.

  3. Select the transfer to update.

  4. Click Edit at the top-right of the page.

  5. Edit Basic and Transfer settings (the Source and Target endpoints cannot be changed at this point).

  6. Click Submit.

Delete a transfer

  1. In the list of services, select Transfer.

  2. Open the Transfers tab.

  3. Select the transfer to delete.

  4. Click Delete at the top-right of the page.

  5. Confirm deletion.

  6. Click Delete.