The ClickHouse® connector

You can use this connector both for source and target endpoints.

Source endpoint

  1. Select the Connection type:

    Managed cluster

    Connection to a database with a cluster identifier in DoubleCloud.

    Specify the connection properties:

    1. Choose Managed Cluster you want to connect to.

    2. Provide the name of an existing Database in the cluster.

    3. Specify a User of the database.

    4. Enter the Password for the database user.

    Custom installation

    Connection to a database with an explicit host network address and port.

    1. Under Shards:

      1. Click + Shard.

      2. Specify the Shard identifier;

      3. Under Hosts, click + Host and enter the domain name (FQDN) or IP-address of the host in the textbox.

    2. Specify the HTTP Port for HTTP interface connections.

      Tip

      • Optional fields have default values if these fields are specified.

      • Complex types recording is supported (array, tuple etc).

    3. Specify the Native port for clickhouse-client connections.

    4. Click SSL if you need to secure your connection.

    5. To encrypt the data transmission, click Upload file under CA Certificate.

  2. Specify the Database name in the cluster.

  3. Specify the User of the database for connection to Transfer service.

  4. Enter the Password for the database user.

  5. Specify additional settings:

    • Included tables. The Transfer service will transfer only the data from these tables. Specify regular expressions to select tables.

    • Excluded tables. The data from the tables on this list won't be transferred. Specify regular expressions to select tables.

To create a ClickHouse® source endpoint with API, use the endpoint.ClickhouseSource model.

ClickHouse® type Transfer type
Int64 int64
Int32 int32
Int16 int16
Int8 int8
UInt64 uint64
UInt32 uint32
UInt16 uint16
UInt8 uint8
float
Float64 double
FixedString, String string
IPv4, IPv6, Enum8, Enum16 utf8
boolean
Date date
DateTime datetime
DateTime64 timestamp
REST... any

Target endpoint

  1. Select the connection type:

    Managed cluster

    Connection to a database with a cluster ID in DoubleCloud.

    Specify the connection properties:

    1. Choose the Managed Cluster to which you want to connect.

    2. Specify a User of the database and their Password.

    3. Enter the name of an existing Database in the cluster.

    Custom installation

    Connection to a database with an explicit host network address and port.

    Under Custom installation, specify the connection properties:

    1. Under Shards:

      1. Click + Shard.

      2. Specify the Shard identifier.

      3. Under Hosts, click + Host and enter the domain name (FQDN) or IP-address of the host in the text field.

    2. Specify HTTP Port for HTTP interface connections.

      Note

      • Optional fields have default values if these fields are specified.

      • Complex types recording is supported (array, tuple etc).

    3. Specify Native port for clickhouse-client connections.

    4. Click SSL if you need to secure your connection.

    5. To encrypt the data transmission, click Upload file under CA Certificate and provide a file in the PEM format.

  2. Specify the name of a Database in your cluster.

  3. Specify the User of the database for connection to the Transfer service.

  4. Enter the Password for the database user.

  5. Specify Advanced settings:

    1. Under Rename tables:

      1. Click + Table.

      2. Provide the name you want to change in Source table name.

      3. Write a new name in Target table name. If the Target endpoint has the table with the same name, the data will be written into the existing table.

      Merge multiple tables with renaming

      You can merge data from several Source endpoint tables into a single one at your Target endpoint. To perform this, create Rename table instances with different Source table name values and the same Target table name ones.

      For this feature to work, ensure the data schemas on the source and target tables are the same.

    2. Specify Sharding field.

    3. If you need Sharding by a transfer ID, click this checkbox.

      Warning

      If both Sharding field and Sharding by transfer ID are active, sharding is performed by transfer ID only.

    4. Under Explicit sharding function you can specify routes of data transfer to specific shards.

      1. Click Add Mapping rule.

      2. Specify the data in Column value.

      3. Enter the Shard name to which you want to transfer data.

      When you use an Explicit sharding function, specify the data type in Sharding field.

    5. If you need to Upload data in JSON format, click this checkbox.

    6. Specify Flush interval in seconds.

      Warning

      The Flush interval field must contain an expression with a number and an s postfix, for example, 7s.

    7. Select a Cleanup policy. This policy allows you to select a way to clean up data in the target database when you activate, reactivate or reload the transfer:

      • Disabled: Do not clean. Select this option if you only perform replication without copying data.

      • Drop: Fully delete the tables included in the transfer (default). Use this option to always transfer the latest version of the table schema to the target database from the source.

      • Truncate: Execute the TRUNCATE command for a target table each time you run a transfer.

To create a ClickHouse® target endpoint with API, use the endpoint.ClickhouseTarget model.

Transfer type ClickHouse® type
int64 Int64
int32 Int32
int16 Int16
int8 Int8
uint64 UInt64
uint32 UInt32
uint16 UInt16
uint8 UInt8
float Float64
double Float64
string String
utf8 String
boolean UInt8
date Date
datetime DateTime
timestamp DateTime64
any String

See also