Manage target endpoints in Transfer

Create a target endpoint

Note

The Transfer service uses custom-built connectors for the following sources: ClickHouse®, PostgreSQL, MySQL, MongoDB, Amazon S3.

Other connectors are based on Airbyte .

  1. Go to the Transfer service page .

  2. Select Endpoints tab, click Create endpoint, and select Target.

  3. Select the Target type you want to transfer the data from.

  4. Under Basic settings:

    • Enter the Name of the endpoint.
    • (optional) Enter the Description of the endpoint.
  5. Specify endpoint parameters under Endpoint settings:

  6. Click Submit.

You can create a target endpoint using the Transfer endpoint resource of the DoubleCloud Terraform provider.

Example provider and resource configuration:

# main.tf

terraform {
  required_providers {
    doublecloud = {
      source    = "registry.terraform.io/doublecloud/doublecloud"
    }
  }
}

provider "doublecloud" {
  authorized_key = file("authorized_key.json")
}

resource "doublecloud_transfer_endpoint" "example-target-endpoint" {
  name       = "example-target-endpoint"
  project_id = DOUBLECLOUD_PROJECT_ID     # Replace with your project ID
  settings {
    clickhouse_target {
      clickhouse_cleanup_policy = "DROP"
      connection {
        address {
            cluster_id = "chcexampleexampleexa"
        }
        database = "default"
        password = CLICKHOUSE_PASSWORD    # Replace with the ClickHouse user password
        user = CLICKHOUSE_USER            # Replace with the ClickHouse user
      }
    }
  }
}

To learn how to get the authorized_key.json file, refer to Create an API key. You can find the DoubleCloud project ID on the project settings page.

Tip

This example contains an example ClickHouse® target endpoint configuration. For configuration parameters for other target types, refer to the Transfer endpoint resource schema .

To create a target endpoint, use the EndpointService create method and pass the following parameters:

  • project_id - The ID of the project in which you want to create your endpoint. You can get this value on your project's information page.

  • name - the new endpoint name. Must be unique within the project.

  • settings - the relevant endpoint settings according to your endpoint type containing _target as reflected in the doublecloud.transfer.v1.EndpointSettings model.

This example shows how to create a ClickHouse® target endpoint.

import json
import logging

import doublecloud

from doublecloud.transfer.v1.endpoint.clickhouse_pb2 import (
 ClickhouseConnection,
 ClickhouseConnectionOptions,
 ClickhouseTarget,
)
from doublecloud.transfer.v1.endpoint.common_pb2 import Secret
from doublecloud.transfer.v1.endpoint_service_pb2 import CreateEndpointRequest
from doublecloud.transfer.v1.endpoint_service_pb2_grpc import EndpointServiceStub

def create_ch_dst_endpoint(sdk, project_id, name):
   svc = sdk.client(EndpointServiceStub)
   operation = svc.Create(
      CreateEndpointRequest(
         project_id=project_id,
         name=f"ch-dst-{name}",
         settings=EndpointSettings(
               clickhouse_target=ClickhouseTarget(
                  connection=ClickhouseConnection(
                     connection_options=ClickhouseConnectionOptions(
                           mdb_cluster_id="<your_cluster_id>",
                           database="<database_name>",
                           user="<username>",
                           password=Secret(raw="<cluster_password>"),
                     )
                  )
               )
         ),
      )
   )
   return operation

For more in-depth examples, check out DoubleCloud API Python SDK repository .

func createCHDstEndpoint(ctx context.Context, dc *dc.SDK, flags *cmdFlags) (*operation.Operation, error) {
   op, err := dc.WrapOperation(dc.Transfer().Endpoint().Create(ctx, &transfer.CreateEndpointRequest{
      ProjectId: *flags.projectID,
      Name:      fmt.Sprint("s3-dst-", *flags.name),
      Settings: &transfer.EndpointSettings{
         Settings: &transfer.EndpointSettings_ClickhouseTarget{
            ClickhouseTarget: &endpoint.ClickhouseTarget{
               Connection: &endpoint.ClickhouseConnection{
                  Connection: &endpoint.ClickhouseConnection_ConnectionOptions{
                     ConnectionOptions: &endpoint.ClickhouseConnectionOptions{
                        Address:  &endpoint.ClickhouseConnectionOptions_MdbClusterId{MdbClusterId: "cluster_id"},
                        Database: "default",
                        User:     "user",
                        Password: &endpoint.Secret{Value: &endpoint.Secret_Raw{Raw: "98s*%^P!3Bw38"}},
                     },
                  },
               },
            },
         },
      },
   }))
   if err != nil {
      return op, err
   }
   err = op.Wait(ctx)
   return op, err
}

For more in-depth examples, check out DoubleCloud API Go SDK repository .

Tip

If you are going to transfer data from a database located in an AWS network, see Access DoubleCloud resources with resources in your AWS VPC to prepare for a successful transfer.

Update a target endpoint

  1. Go to the Transfer page in the console and open the Endpoints tab.

  2. Select the endpoint to update.

  3. Click Edit at the top-right of the page.

  4. Edit the Basic parameters and the Endpoint parameters. You can't change the Source type at this point.

  5. Click Submit.

Warning

You can't change the target type. If you want to transfer data to a different type database, create a new target endpoint.

resource "doublecloud_transfer_endpoint" "example-target-endpoint" {
  name = "example-target-endpoint"
  project_id = DOUBLECLOUD_PROJECT_ID     # Replace with your project ID
  settings {
    postgres_source {
      ...                                 # Update the desired parameters
    }
  }
}

After you finish editing the configuration, update the endpoint resource using the terraform apply command.

To update a target endpoint, use the EndpointService update method and pass the following parameters:

  • endpoint_id - the ID of the endpoint you want to update. To find the endpoint ID, get a list of endpoints in the project.

  • name - the new endpoint name. Must be unique within the project.

  • settings - the relevant endpoint settings according to your endpoint type as reflected in the doublecloud.transfer.v1.EndpointSettings model.

Delete a target endpoint

Warning

Before you delete an endpoint, delete all the transfers that use it.

To delete an endpoint:

  1. Go to the Transfer page in the console and open the Endpoints tab.

  2. Select the endpoint to delete.

  3. At the top-right of the page, click Delete.

  4. In the opened window, confirm your action and click Delete.

To delete an endpoint:

  1. Comment out or delete the endpoint resource from the Terraform configuration files.

  2. Apply the configuration:

    terraform apply
    

Use the EndpointService delete method and pass the endpoint ID in the endpoint_id request parameter.

To find the endpoint ID, get a list of transfers in the project.

github-mark-white

View this example on GitHub

from doublecloud.transfer.v1.endpoint_service_pb2 import DeleteEndpointRequest


def delete_endpoint(svc, endpoint_id: str):
   return svc.Delete(DeleteEndpointRequest(endpoint_id=endpoint_id))

For more in-depth examples, check out DoubleCloud API Python SDK repository .

func deleteEndpoint(ctx context.Context, dc *dc.SDK, endpointId string) (*operation.Operation, error) {
   op, err := dc.WrapOperation(dc.Transfer().Endpoint().Delete(ctx, &transfer.DeleteEndpointRequest{EndpointId: endpointId}))
   if err != nil {
      log.Fatal(err)
   }
   err = op.Wait(ctx)
   return op, err
}

For more in-depth examples, check out DoubleCloud API Go SDK repository .