The ClickHouse® connector

You can use this connector both for source and target endpoints.

Source endpoint configuration

  1. Select the connection type:

    Connection to a database with a cluster identifier in DoubleCloud.

    Specify the connection properties:

    1. Choose Managed Cluster you want to connect to.

    2. Specify a User of the database.

    3. Enter the Password of the database user.

    4. Set the name of an existing Database in the cluster.

    Connection to a database with an explicit host network address and port.

    1. Under OnPremise cluster, specify the connection properties:

      1. Under Shards:

        1. Click Add Shard.

        2. Specify Shard ID identifier;

        3. Under Hosts, click Add Host and enter the domain name (FQDN) or IP-address of the host in the textbox.

      2. Click SSL if you need to secure your connection.

      3. To encrypt the data transmission, click Chose a file under PEM Certificate.

      4. Specify HTTP Port for HTTP interface connections.

        Note

        • Optional fields have default values if these fields are specified.

        • Complex types recording is supported (array, tuple etc).

      5. Specify Native port for clickhouse-client connections.

    2. Specify the User of the database for connection to Transfer service.

    3. Enter the Password for the database user.

    4. Specify the Database name in the cluster.

  2. Specify additional settings:

    • Included tables. The Transfer service will transfer only the data from these tables. Specify regular expressions to select tables.

    • Excluded tables. The data from the tables on this list won't be transferred. Specify regular expressions to select tables.

Target endpoint configuration

  1. Select the connection type:

    Connection to a database with a cluster ID in DoubleCloud.

    Specify the connection properties:

    1. Choose Managed Cluster you want to connect to.

    2. Specify a User of the database.

    3. Enter the Password of the database user.

    4. Set the name of an existing Database in the cluster.

    Connection to a database with an explicit host network address and port.

    1. Under OnPremise cluster, specify the connection properties:

      1. Under Shards:

        1. Click Add Shard.

        2. Specify Shard ID identifier;

        3. Under Hosts, click Add Host and enter the domain name (FQDN) or IP-address of the host in the textbox.

      2. Click SSL if you need to secure your connection.

      3. To encrypt the data transmission, click Chose a file under PEM Certificate.

      4. Specify HTTP Port for HTTP interface connections.

        Note

        • Optional fields have default values if these fields are specified.

        • Complex types recording is supported (array, tuple etc).

      5. Specify Native port for clickhouse-client connections.

    2. Specify the User of the database for connection to Transfer service.

    3. Enter the Password for the database user.

    4. Specify the name of a Database in the cluster.

  2. Specify additional settings:

    1. Under Alt names:

      1. Click Add Alt names.

      2. Provide the name you want to change in From table name.

      3. Write a new name in To table name. If the Target endpoint has the table with the same name, the data will be written into the existing table.

      Tip

      You can merge data from several Source endpoint tables into a single one at your Target endpoint. To perform this, create Alt Name instances with different From table name values and the same To table name ones. This feature will work only if the data schemes are the same.

    2. Specify Sharding field.

    3. If you need Sharding by a transfer ID, click this checkbox.

      Warning

      If both Sharding field and Sharding by transfer ID are active, sharding is performed by transfer ID only.

    4. Under Explicit sharding function you can specify routes of data transfer to specific shards.

      1. Click Add Mapping rule.

      2. Specify the data in Column value.

      3. Enter the Shard name to which you want to transfer data.

    Note

    When you use an Explicit sharding function, specify the data type in Sharding field.

    1. If you need to Upload data in JSON format, click this checkbox.

    2. Specify Flush interval in seconds.

    Warning

    Flush interval field must contain an expression with a number and an s postfix, for example, 7s.

    1. Select a Cleanup policy. This policy allows you to select a way to clean up data in the target database when you activate, reactivate or reload the transfer:

      • Disabled: Do not clean. Select this option if you only perform replication without copying data.

      • Drop: Fully delete the tables included in the transfer (default). Use this option to always transfer the latest version of the table schema to the target database from the source.

      • Truncate: Execute the TRUNCATE command for a target table each time you run a transfer.