Create and manage target endpoints in Transfer

Create a target endpoint

Note

Transfer uses custom-built connectors for the following sources: ClickHouse®, PostgreSQL, MySQL, MongoDB, and Amazon S3.

Other connectors are based on Airbyte .

Console

Terraform

API

Python

Go to the Transfer page .
Click Create → Target endpoint.
In Target type, select the type of your target database.
Under Basic settings:
- Enter the Name of the endpoint.
- (optional) Enter the Description of the endpoint.
Specify endpoint parameters under Endpoint settings:
Click Submit.

You can create a target endpoint using the Transfer endpoint resource of the DoubleCloud Terraform provider.

Example provider and resource configuration:

# main.tf

terraform {
  required_providers {
    doublecloud = {
      source    = "registry.terraform.io/doublecloud/doublecloud"
    }
  }
}

provider "doublecloud" {
  authorized_key = file("authorized_key.json")
}

resource "doublecloud_transfer_endpoint" "example-target-endpoint" {
  name       = "example-target-endpoint"
  project_id = DOUBLECLOUD_PROJECT_ID     # Replace with your project ID
  settings {
    clickhouse_target {
      clickhouse_cleanup_policy = "DROP"
      connection {
        address {
            cluster_id = "chcexampleexampleexa"
        }
        database = "default"
        password = CLICKHOUSE_PASSWORD    # Replace with the ClickHouse user password
        user = CLICKHOUSE_USER            # Replace with the ClickHouse user
      }
    }
  }
}

To learn how to get the authorized_key.json file, refer to Create an API key. You can find the DoubleCloud project ID on the project settings page.

Tip

This example contains an example ClickHouse® target endpoint configuration. For configuration parameters for other target types, refer to the Transfer endpoint resource schema .

To create a target endpoint, use the EndpointService create method and pass the following parameters:

project_id - The ID of the project in which you want to create your endpoint. You can get this value on your project's information page.
name - the new endpoint name. Must be unique within the project.
settings - the relevant endpoint settings according to your endpoint type containing _target as reflected in the doublecloud.transfer.v1.EndpointSettings model.

This example shows how to create a ClickHouse® target endpoint.

import json
import logging

import doublecloud

from doublecloud.transfer.v1.endpoint.clickhouse_pb2 import (
 ClickhouseConnection,
 ClickhouseConnectionOptions,
 ClickhouseTarget,
)
from doublecloud.transfer.v1.endpoint.common_pb2 import Secret
from doublecloud.transfer.v1.endpoint_service_pb2 import CreateEndpointRequest
from doublecloud.transfer.v1.endpoint_service_pb2_grpc import EndpointServiceStub

def create_ch_dst_endpoint(sdk, project_id, name):
   svc = sdk.client(EndpointServiceStub)
   operation = svc.Create(
      CreateEndpointRequest(
         project_id=project_id,
         name=f"ch-dst-{name}",
         settings=EndpointSettings(
               clickhouse_target=ClickhouseTarget(
                  connection=ClickhouseConnection(
                     connection_options=ClickhouseConnectionOptions(
                           mdb_cluster_id="<your_cluster_id>",
                           database="<database_name>",
                           user="<username>",
                           password=Secret(raw="<cluster_password>"),
                     )
                  )
               )
         ),
      )
   )
   return operation

For more in-depth examples, check out DoubleCloud API Python SDK repository .

func createCHDstEndpoint(ctx context.Context, dc *dc.SDK, flags *cmdFlags) (*operation.Operation, error) {
   op, err := dc.WrapOperation(dc.Transfer().Endpoint().Create(ctx, &transfer.CreateEndpointRequest{
      ProjectId: *flags.projectID,
      Name:      fmt.Sprint("s3-dst-", *flags.name),
      Settings: &transfer.EndpointSettings{
         Settings: &transfer.EndpointSettings_ClickhouseTarget{
            ClickhouseTarget: &endpoint.ClickhouseTarget{
               Connection: &endpoint.ClickhouseConnection{
                  Connection: &endpoint.ClickhouseConnection_ConnectionOptions{
                     ConnectionOptions: &endpoint.ClickhouseConnectionOptions{
                        Address:  &endpoint.ClickhouseConnectionOptions_MdbClusterId{MdbClusterId: "cluster_id"},
                        Database: "default",
                        User:     "user",
                        Password: &endpoint.Secret{Value: &endpoint.Secret_Raw{Raw: "98s*%^P!3Bw38"}},
                     },
                  },
               },
            },
         },
      },
   }))
   if err != nil {
      return op, err
   }
   err = op.Wait(ctx)
   return op, err
}

For more in-depth examples, check out DoubleCloud API Go SDK repository .

Tip

If you are going to transfer data from a database located in an AWS network, see Access DoubleCloud resources with resources in your AWS VPC to prepare for a successful transfer.

Update a target endpoint

Console

Terraform

API

Go to the Transfer page in the console and switch to the Endpoints tab.
Select the endpoint you want to update.
Click Edit at the top right of the page.
Edit settings under Basic parameters and Endpoint parameters.
Click Submit.

Warning

You can’t change the target type. If you want to transfer data to a different type database, create a new target endpoint.

resource "doublecloud_transfer_endpoint" "example-target-endpoint" {
  name = "example-target-endpoint"
  project_id = DOUBLECLOUD_PROJECT_ID     # Replace with your project ID
  settings {
    postgres_source {
      ...                                 # Update the desired parameters
    }
  }
}

After you finish editing the configuration, update the endpoint resource using the terraform apply command.

To update a target endpoint, use the EndpointService update method and pass the following parameters:

endpoint_id - the ID of the endpoint you want to update. To find the endpoint ID, get a list of endpoints in the project.
name - the new endpoint name. Must be unique within the project.
settings - the relevant endpoint settings according to your endpoint type as reflected in the doublecloud.transfer.v1.EndpointSettings model.

Delete a target endpoint

Warning

Before you delete an endpoint, delete all the transfers that use it.

Console

Terraform

API

Python

To delete an endpoint:

Go to the Transfer page in the console and switch to the Endpoints tab.
Select the endpoint you want to delete.
At the top right of the page, click Delete.
In the dialog that opened, confirm your action and click Delete.

To delete an endpoint:

Comment out or delete the endpoint resource from the Terraform configuration files.
Apply the configuration:
```
terraform apply
```

Use the EndpointService delete method and pass the endpoint ID in the endpoint_id request parameter.

To find the endpoint ID, get a list of transfers in the project.

View this example on GitHub

from doublecloud.transfer.v1.endpoint_service_pb2 import DeleteEndpointRequest


def delete_endpoint(svc, endpoint_id: str):
   return svc.Delete(DeleteEndpointRequest(endpoint_id=endpoint_id))

For more in-depth examples, check out DoubleCloud API Python SDK repository .

func deleteEndpoint(ctx context.Context, dc *dc.SDK, endpointId string) (*operation.Operation, error) {
   op, err := dc.WrapOperation(dc.Transfer().Endpoint().Delete(ctx, &transfer.DeleteEndpointRequest{EndpointId: endpointId}))
   if err != nil {
      log.Fatal(err)
   }
   err = op.Wait(ctx)
   return op, err
}

For more in-depth examples, check out DoubleCloud API Go SDK repository .

Create and manage source endpoints

Create and manage transfers