Transfer API models

Option

Description

TRANSFER_STATUS_UNSPECIFIED

CREATING

Transfer does some work before running

CREATED

Transfer created but not started by user

RUNNING

Transfer currently doing replication work

STOPPING

Transfer shutdown

STOPPED

Transfer stopped by user

ERROR

Transfer stopped by system

SNAPSHOTTING

Transfer copy snapshot

DONE

Transfer reach terminal phase

Option

Description

TRANSFER_TYPE_UNSPECIFIED

SNAPSHOT_AND_INCREMENT

Snapshot and increment

SNAPSHOT_ONLY

Snapshot

INCREMENT_ONLY

Increment

Option

Description

CLICKHOUSE_CLEANUP_POLICY_UNSPECIFIED

CLICKHOUSE_CLEANUP_POLICY_DISABLED

CLICKHOUSE_CLEANUP_POLICY_DROP

CLICKHOUSE_CLEANUP_POLICY_TRUNCATE

Option

Description

CLEANUP_POLICY_UNSPECIFIED

DISABLED

Don't cleanup

DROP

Drop

TRUNCATE

Truncate

Option

Description

COLUMN_TYPE_UNSPECIFIED

INT64

INT32

INT16

INT8

UINT64

UINT32

UINT16

UINT8

DOUBLE

BOOLEAN

STRING

UTF8

ANY

DATETIME

Option

Description

OBJECT_TRANSFER_STAGE_UNSPECIFIED

BEFORE_DATA

Before data transfer

AFTER_DATA

After data transfer

NEVER

Don't copy

Option

Description

KAFKA_MECHANISM_UNSPECIFIED

KAFKA_MECHANISM_SHA256

KAFKA_MECHANISM_SHA512

Option

Description

ACTION_BREAKDOWN_UNSPECIFIED

ACTION_CANVAS_COMPONENT_NAME

ACTION_CAROUSEL_CARD_ID

ACTION_CAROUSEL_CARD_NAME

ACTION_DESTINATION

ACTION_DEVICE

ACTION_REACTION

ACTION_TARGET_ID

ACTION_TYPE

ACTION_VIDEO_SOUND

ACTION_VIDEO_TYPE

Option

Description

BREAKDOWN_UNSPECIFIED

AD_FORMAT_ASSET

AGE

APP_ID

BODY_ASSET

CALL_TO_ACTION_ASSET

COUNTRY

DESCRIPTION_ASSET

DEVICE_PLATFORM

DMA

FREQUENCY_VALUE

GENDER

HOURLY_STATS_AGGREGATED_BY_ADVERTISER_TIME_ZONE

HOURLY_STATS_AGGREGATED_BY_AUDIENCE_TIME_ZONE

IMAGE_ASSET

IMPRESSION_DEVICE

LINK_URL_ASSET

PLACE_PAGE_ID

PLATFORM_POSITION

PRODUCT_ID

PUBLISHER_PLATFORM

REGION

SKAN_CONVERSION_ID

TITLE_ASSET

VIDEO_ASSE

Option

Description

FIELD_UNSPECIFIED

ACCOUNT_CURRENCY

ACCOUNT_ID

ACCOUNT_NAME

ACTION_VALUES

ACTIONS

AD_BID_VALUE

AD_CLICK_ACTIONS

AD_ID

AD_IMPRESSION_ACTIONS

AD_NAME

ADSET_BID_VALUE

ADSET_END

ADSET_ID

ADSET_NAME

ADSET_START

AGE_TARGETING

ATTRIBUTION_SETTING

AUCTION_BID

AUCTION_COMPETITIVENESS

AUCTION_MAX_COMPETITOR_BID

BUYING_TYPE

CAMPAIGN_ID

CAMPAIGN_NAME

CANVAS_AVG_VIEW_PERCENT

CANVAS_AVG_VIEW_TIME

CATALOG_SEGMENT_ACTIONS

CATALOG_SEGMENT_VALUE

CATALOG_SEGMENT_VALUE_MOBILE_PURCHASE_ROAS

CATALOG_SEGMENT_VALUE_OMNI_PURCHASE_ROAS

CATALOG_SEGMENT_VALUE_WEBSITE_PURCHASE_ROAS

CLICKS

CONVERSION_RATE_RANKING

CONVERSION_VALUES

CONVERSIONS

CONVERTED_PRODUCT_QUANTITY

CONVERTED_PRODUCT_VALUE

COST_PER_15_SEC_VIDEO_VIEW

COST_PER_2_SEC_CONTINUOUS_VIDEO_VIEW

COST_PER_ACTION_TYPE

COST_PER_AD_CLICK

COST_PER_CONVERSION

COST_PER_DDA_COUNTBY_CONVS

COST_PER_ESTIMATED_AD_RECALLERS

COST_PER_INLINE_LINK_CLICK

COST_PER_INLINE_POST_ENGAGEMENT

COST_PER_ONE_THOUSAND_AD_IMPRESSION

COST_PER_OUTBOUND_CLICK

COST_PER_THRUPLAY

COST_PER_UNIQUE_ACTION_TYPE

COST_PER_UNIQUE_CLICK

COST_PER_UNIQUE_CONVERSION

COST_PER_UNIQUE_INLINE_LINK_CLICK

COST_PER_UNIQUE_OUTBOUND_CLICK

CPC

CPM

CPP

CREATED_TIME

CTR

DATE_START

DATE_STOP

DDA_COUNTBY_CONVS

DDA_RESULTS

ENGAGEMENT_RATE_RANKING

ESTIMATED_AD_RECALL_RATE

ESTIMATED_AD_RECALL_RATE_LOWER_BOUND

ESTIMATED_AD_RECALL_RATE_UPPER_BOUND

ESTIMATED_AD_RECALLERS

ESTIMATED_AD_RECALLERS_LOWER_BOUND

ESTIMATED_AD_RECALLERS_UPPER_BOUND

FREQUENCY

FULL_VIEW_IMPRESSIONS

FULL_VIEW_REACH

GENDER_TARGETING

IMPRESSIONS

INLINE_LINK_CLICK_CTR

INLINE_LINK_CLICKS

INLINE_POST_ENGAGEMENT

INSTANT_EXPERIENCE_CLICKS_TO_OPEN

INSTANT_EXPERIENCE_CLICKS_TO_START

INSTANT_EXPERIENCE_OUTBOUND_CLICKS

INTERACTIVE_COMPONENT_TAP

LABELS

LOCATION

MOBILE_APP_PURCHASE_ROAS

OBJECTIVE

OPTIMIZATION_GOAL

OUTBOUND_CLICKS

OUTBOUND_CLICKS_CTR

PLACE_PAGE_NAME

PURCHASE_ROAS

QUALIFYING_QUESTION_QUALIFY_ANSWER_RATE

QUALITY_RANKING

QUALITY_SCORE_ECTR

QUALITY_SCORE_ECVR

QUALITY_SCORE_ORGANIC

REACH

SOCIAL_SPEND

SPEND

TOTAL_POSTBACKS

UNIQUE_ACTIONS

UNIQUE_CLICKS

UNIQUE_CONVERSIONS

UNIQUE_CTR

UNIQUE_INLINE_LINK_CLICK_CTR

UNIQUE_INLINE_LINK_CLICKS

UNIQUE_LINK_CLICKS_CTR

UNIQUE_OUTBOUND_CLICKS

UNIQUE_OUTBOUND_CLICKS_CTR

UNIQUE_VIDEO_CONTINUOUS_2_SEC_WATCHED_ACTIONS

UNIQUE_VIDEO_VIEW_15_SEC

UPDATED_TIME

VIDEO_15_SEC_WATCHED_ACTIONS

VIDEO_30_SEC_WATCHED_ACTIONS

VIDEO_AVG_TIME_WATCHED_ACTIONS

VIDEO_CONTINUOUS_2_SEC_WATCHED_ACTIONS

VIDEO_P100_WATCHED_ACTIONS

VIDEO_P25_WATCHED_ACTIONS

VIDEO_P50_WATCHED_ACTIONS

VIDEO_P75_WATCHED_ACTIONS

VIDEO_P95_WATCHED_ACTIONS

VIDEO_PLAY_ACTIONS

VIDEO_PLAY_CURVE_ACTIONS

VIDEO_PLAY_RETENTION_0_TO_15S_ACTIONS

VIDEO_PLAY_RETENTION_20_TO_60S_ACTIONS

VIDEO_PLAY_RETENTION_GRAPH_ACTIONS

VIDEO_THRUPLAY_WATCHED_ACTIONS

VIDEO_TIME_WATCHED_ACTIONS

WEBSITE_CTR

WEBSITE_PURCHASE_ROAS

WISH_BID

Option

Description

MSSQL_REPLICATION_METHOD_UNSPECIFIED

STANDARD

CDC

Option

Description

UNEXPECTED_FIELD_BEHAVIOR_UNSPECIFIED

UNEXPECTED_FIELD_BEHAVIOR_IGNORE

UNEXPECTED_FIELD_BEHAVIOR_INFER

UNEXPECTED_FIELD_BEHAVIOR_ERROR

Field

Description

id

string

project_id

string

name

string

description

string

labels

list

labels[]

doublecloud.transfer.v1.Endpoint.LabelsEntry

settings

doublecloud.transfer.v1.EndpointSettings

Field

Description

key

string

value

string

Field

Description

settings

one of: mysql_source / postgres_source / kafka_source / mongo_source / clickhouse_source / s3_source / aws_cloudtrail_source / big_query_source / facebook_marketing_source / google_ads_source / amazon_ads_source / instagram_source / linkedin_ads_source / mssql_source / redshift_source / snowflake_source / mysql_target / postgres_target / clickhouse_target / kafka_target / mongo_target

mysql_source

doublecloud.transfer.v1.endpoint.MysqlSource

postgres_source

doublecloud.transfer.v1.endpoint.PostgresSource

kafka_source

doublecloud.transfer.v1.endpoint.KafkaSource

mongo_source

doublecloud.transfer.v1.endpoint.MongoSource

clickhouse_source

doublecloud.transfer.v1.endpoint.ClickhouseSource

s3_source

doublecloud.transfer.v1.endpoint.airbyte.S3Source

aws_cloudtrail_source

doublecloud.transfer.v1.endpoint.airbyte.AWSCloudTrailSource

big_query_source

doublecloud.transfer.v1.endpoint.airbyte.BigQuerySource

facebook_marketing_source

doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource

google_ads_source

doublecloud.transfer.v1.endpoint.airbyte.GoogleAdsSource

amazon_ads_source

doublecloud.transfer.v1.endpoint.airbyte.AmazonAdsSource

instagram_source

doublecloud.transfer.v1.endpoint.airbyte.InstagramSource

linkedin_ads_source

doublecloud.transfer.v1.endpoint.airbyte.LinkedinAdsSource

mssql_source

doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource

redshift_source

doublecloud.transfer.v1.endpoint.airbyte.RedshiftSource

snowflake_source

doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource

mysql_target

doublecloud.transfer.v1.endpoint.MysqlTarget

postgres_target

doublecloud.transfer.v1.endpoint.PostgresTarget

clickhouse_target

doublecloud.transfer.v1.endpoint.ClickhouseTarget

kafka_target

doublecloud.transfer.v1.endpoint.KafkaTarget

mongo_target

doublecloud.transfer.v1.endpoint.MongoTarget

Field

Description

project_id

string

name

string

description

string

labels

list

labels[]

doublecloud.transfer.v1.CreateEndpointRequest.LabelsEntry

settings

doublecloud.transfer.v1.EndpointSettings

Field

Description

key

string

value

string

Field

Description

endpoint_id

string

Field

Description

endpoint_id

string

Field

Description

project_id

string Identifier of the folder containing the endpoints to be listed.

page

doublecloud.v1.Paging The maximum number of endpoints to be sent in the response message. If the folder contains more endpoints than page_size, next_page_token will be included in the response message. Include it into the subsequent ListEndpointRequest to fetch the next page. Defaults to 100 if not specified. The maximum allowed value for this field is 500.

Field

Description

endpoints

list The list of endpoints. If there are more endpoints in the folder, then next_page_token is a non-empty string to be included into the subsequent ListEndpointsRequest to fetch the next endpoints page.

endpoints[]

doublecloud.transfer.v1.Endpoint

next_page

doublecloud.v1.NextPage Opaque value identifying the next endpoints page. This field is empty if there are no more endpoints in the folder. Otherwise, it is non-empty and should be included in the subsequent ListEndpointsRequest to fetch the next endpoints page.

Field

Description

endpoint_id

string Identifier of the endpoint to be updated.

name

string The new endpoint name. Must be unique within the folder.

description

string The new description for the endpoint.

labels

list

labels[]

doublecloud.transfer.v1.UpdateEndpointRequest.LabelsEntry

settings

doublecloud.transfer.v1.EndpointSettings The new endpoint name. Must be unique within the folder.

Field

Description

key

string

value

string

Field

Description

operation_id

string

Field

Description

id

string

project_id

string

name

string

description

string

labels

list

labels[]

doublecloud.transfer.v1.Transfer.LabelsEntry

source

doublecloud.transfer.v1.Endpoint

target

doublecloud.transfer.v1.Endpoint

status

doublecloud.transfer.v1.TransferStatus

type

doublecloud.transfer.v1.TransferType

warning

string

Field

Description

key

string

value

string

Field

Description

transfer_id

string

Field

Description

source_id

string

target_id

string

name

string

description

string

labels

list

labels[]

doublecloud.transfer.v1.CreateTransferRequest.LabelsEntry

project_id

string

type

doublecloud.transfer.v1.TransferType

Field

Description

key

string

value

string

Field

Description

transfer_id

string

Field

Description

transfer_id

string The ID of the Transfer

Field

Description

transfer_id

string The ID of the Transfer

Field

Description

transfer_id

string

Field

Description

transfer_id

string The ID of Transfer

Field

Description

transfer_id

string

Field

Description

project_id

string Identifier of the folder containing the transfers to be listed.

page

doublecloud.v1.Paging The maximum number of transfers to be sent in the response message. If the folder contains more transfers than page_size, next_page_token will be included in the response message. Include it into the subsequent ListTransfersRequest to fetch the next page. Defaults to 100 if not specified. The maximum allowed value for this field is 500.

Field

Description

transfers

list The list of transfers. If there are more transfers in the folder, then next_page_token is a non-empty string to be included into the subsequent ListTransfersRequest to fetch the next transfers page.

transfers[]

doublecloud.transfer.v1.Transfer

next_page_token

string Opaque value identifying the next transfers page. This field is empty if there are no more transfers in the folder. Otherwise it is non-empty and should be included in the subsequent ListTransfersRequest to fetch the next transfers page.

Field

Description

connection_string

string String to use in clients. i.e. url for config

user

string User for HTTP basic auth.

password

string Password for HTTP basic auth.

Field

Description

transfer_id

string The ID of the Transfer

Field

Description

metrics

list The list of the transfer metrics

metrics[]

doublecloud.transfer.v1.TransferMetrics.Metric

Field

Description

help

string

type

string

value

string

Field

Description

transfer_id

string Identifier of the transfer to be updated.

description

string The new description for the transfer.

labels

list

labels[]

doublecloud.transfer.v1.UpdateTransferRequest.LabelsEntry

name

string The new transfer name. Must be unique within the folder.

Field

Description

key

string

value

string

Field

Description

connection

one of: connection_options

connection_options

doublecloud.transfer.v1.endpoint.ClickhouseConnectionOptions

Field

Description

address

one of: mdb_cluster_id / on_premise

mdb_cluster_id

string

on_premise

doublecloud.transfer.v1.endpoint.OnPremiseClickhouse

database

string Database

user

string

password

doublecloud.transfer.v1.endpoint.Secret

Field

Description

name

string

hosts

list

hosts[]

string

Field

Description

sharding

one of: column_value_hash / custom_mapping / transfer_id

column_value_hash

doublecloud.transfer.v1.endpoint.ClickhouseSharding.ColumnValueHash

custom_mapping

doublecloud.transfer.v1.endpoint.ClickhouseSharding.ColumnValueMapping

transfer_id

google.protobuf.Empty

Field

Description

column_name

string

Field

Description

column_name

string

mapping

list

mapping[]

doublecloud.transfer.v1.endpoint.ClickhouseSharding.ColumnValueMapping.ValueToShard

Field

Description

column_value

doublecloud.transfer.v1.endpoint.ColumnValue

shard_name

string

Field

Description

connection

doublecloud.transfer.v1.endpoint.ClickhouseConnection

include_tables

list While list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.

include_tables[]

string

exclude_tables

list Exclude list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.

exclude_tables[]

string

Field

Description

connection

doublecloud.transfer.v1.endpoint.ClickhouseConnection

clickhouse_cluster_name

string

alt_names

list Alternative table names in target

alt_names[]

doublecloud.transfer.v1.endpoint.AltName

sharding

doublecloud.transfer.v1.endpoint.ClickhouseSharding

cleanup_policy

doublecloud.transfer.v1.endpoint.ClickhouseCleanupPolicy

Field

Description

shards

list

shards[]

doublecloud.transfer.v1.endpoint.ClickhouseShard

http_port

int64

native_port

int64

tls_mode

doublecloud.transfer.v1.endpoint.TLSMode

Field

Description

from_name

string Source table name

to_name

string Target table name

Field

Description

name

string

type

doublecloud.transfer.v1.endpoint.ColumnType

key

bool

required

bool

path

string

Field

Description

value

one of: string_value

string_value

string

Field

Description

schema

one of: fields / json_fields

fields

doublecloud.transfer.v1.endpoint.FieldList

json_fields

string

Field

Description

fields

list Column schema

fields[]

doublecloud.transfer.v1.endpoint.ColSchema

Field

Description

Field

Description

value

one of: raw Raw secret value

raw

string Raw secret value

Field

Description

ca_certificate

string CA certificate

X.509 certificate of the certificate authority which issued the server's certificate, in PEM format. When CA certificate is specified TLS is used to connect to the server.

Field

Description

tls_mode

one of: disabled / enabled

disabled

google.protobuf.Empty

enabled

doublecloud.transfer.v1.endpoint.TLSConfig

Field

Description

provider

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Provider

table

doublecloud.transfer.v1.endpoint.DeltaResultTable

settings

doublecloud.transfer.v1.endpoint.DeltaSettings

Field

Description

Field

Description

region

string

Field

Description

security

one of: sasl / no_auth Authentication with SASL

sasl

doublecloud.transfer.v1.endpoint.KafkaSaslSecurity Authentication with SASL

no_auth

doublecloud.transfer.v1.endpoint.NoAuth No authentication

Field

Description

connection

one of: cluster_id / on_premise Managed Service for Kafka cluster ID

cluster_id

string Managed Service for Kafka cluster ID

on_premise

doublecloud.transfer.v1.endpoint.OnPremiseKafka Connection options for on-premise Kafka

Field

Description

user

string User name

password

doublecloud.transfer.v1.endpoint.Secret Password for user

mechanism

doublecloud.transfer.v1.endpoint.KafkaMechanism SASL mechanism for authentication

Field

Description

connection

doublecloud.transfer.v1.endpoint.KafkaConnectionOptions Connection settings

auth

doublecloud.transfer.v1.endpoint.KafkaAuth Authentication settings

topic_name

string Full source topic name

parser

doublecloud.transfer.v1.endpoint.Parser Data parsing rules

Field

Description

connection

doublecloud.transfer.v1.endpoint.KafkaConnectionOptions Connection settings

auth

doublecloud.transfer.v1.endpoint.KafkaAuth Authentication settings

topic_settings

doublecloud.transfer.v1.endpoint.KafkaTargetTopicSettings Target topic settings

serializer

doublecloud.transfer.v1.endpoint.Serializer Data serialization format settings

Field

Description

topic_name

string Topic name

save_tx_order

bool Save transactions order Not to split events queue into separate per-table queues.

Field

Description

topic_settings

one of: topic / topic_prefix Full topic name

topic

doublecloud.transfer.v1.endpoint.KafkaTargetTopic Full topic name

topic_prefix

string Topic prefix

Analogue of the Debezium setting database.server.name. Messages will be sent to topic with name <topic_prefix>..<table_name>.

Field

Description

broker_urls

list Kafka broker URLs

broker_urls[]

string

tls_mode

doublecloud.transfer.v1.endpoint.TLSMode TLS settings for broker connection. Disabled by default.

Field

Description

database_name

string

collection_name

string

Field

Description

connection

one of: connection_options

connection_options

doublecloud.transfer.v1.endpoint.MongoConnectionOptions

Field

Description

address

one of: on_premise

on_premise

doublecloud.transfer.v1.endpoint.OnPremiseMongo

user

string User name

password

doublecloud.transfer.v1.endpoint.Secret Password for user

auth_source

string Database name associated with the credentials

Field

Description

connection

doublecloud.transfer.v1.endpoint.MongoConnection

collections

list List of collections for replication. Empty list implies replication of all tables on the deployment. Allowed to use * as collection name.

collections[]

doublecloud.transfer.v1.endpoint.MongoCollection

excluded_collections

list List of forbidden collections for replication. Allowed to use * as collection name for forbid all collections of concrete schema.

excluded_collections[]

doublecloud.transfer.v1.endpoint.MongoCollection

secondary_preferred_mode

bool Read mode for mongo client

Field

Description

connection

doublecloud.transfer.v1.endpoint.MongoConnection

database

string Database name

cleanup_policy

doublecloud.transfer.v1.endpoint.CleanupPolicy

Field

Description

hosts

list

hosts[]

string

port

int64

tls_mode

doublecloud.transfer.v1.endpoint.TLSMode

replica_set

string

Field

Description

connection

one of: on_premise Connection options for on-premise MySQL

on_premise

doublecloud.transfer.v1.endpoint.OnPremiseMysql Connection options for on-premise MySQL

Field

Description

view

doublecloud.transfer.v1.endpoint.ObjectTransferStage Views

CREATE VIEW ...

routine

doublecloud.transfer.v1.endpoint.ObjectTransferStage Routines

CREATE PROCEDURE ... ; CREATE FUNCTION ... ;

trigger

doublecloud.transfer.v1.endpoint.ObjectTransferStage Triggers

CREATE TRIGGER ...

tables

doublecloud.transfer.v1.endpoint.ObjectTransferStage

Field

Description

connection

doublecloud.transfer.v1.endpoint.MysqlConnection Database connection settings

database

string Database name

You can leave it empty, then it will be possible to transfer tables from several databases at the same time from this source.

service_database

string Database for service tables

Default: data source database. Here created technical tables (__tm_keeper, __tm_gtid_keeper).

user

string User for database access.

password

doublecloud.transfer.v1.endpoint.Secret Password for database access.

include_tables_regex

list

include_tables_regex[]

string

exclude_tables_regex

list

exclude_tables_regex[]

string

timezone

string Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.

object_transfer_settings

doublecloud.transfer.v1.endpoint.MysqlObjectTransferSettings Schema migration

Select database objects to be transferred during activation or deactivation.

Field

Description

connection

doublecloud.transfer.v1.endpoint.MysqlConnection Database connection settings

security_groups

list Security groups

security_groups[]

string

database

string Database name

Allowed to leave it empty, then the tables will be created in databases with the same names as on the source. If this field is empty, then you must fill below db schema for service table.

user

string User for database access.

password

doublecloud.transfer.v1.endpoint.Secret Password for database access.

sql_mode

string Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.

skip_constraint_checks

bool Disable constraints checks

Recommend to disable for increase replication speed, but if schema contain cascading operations we don't recommend to disable. This option set FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0.

timezone

string Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.

cleanup_policy

doublecloud.transfer.v1.endpoint.CleanupPolicy Cleanup policy

Cleanup policy for activate, reactivate and reupload processes. Default is DISABLED.

service_database

string Database schema for service table

Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper).

Field

Description

hosts

list

hosts[]

string

port

int64 Database port

tls_mode

doublecloud.transfer.v1.endpoint.TLSMode TLS settings for server connection. Disabled by default.

Field

Description

data_schema

doublecloud.transfer.v1.endpoint.DataSchema

null_keys_allowed

bool Allow null keys, if no - null keys will be putted to unparsed data

add_rest_column

bool Will add _rest column for all unknown fields

Field

Description

parser

one of: json_parser / tskv_parser

json_parser

doublecloud.transfer.v1.endpoint.GenericParserCommon

tskv_parser

doublecloud.transfer.v1.endpoint.GenericParserCommon

Field

Description

hosts

list

hosts[]

string

port

int64 Will be used if the cluster ID is not specified.

tls_mode

doublecloud.transfer.v1.endpoint.TLSMode TLS settings for server connection. Disabled by default.

Field

Description

connection

one of: on_premise Connection options for on-premise PostgreSQL

on_premise

doublecloud.transfer.v1.endpoint.OnPremisePostgres Connection options for on-premise PostgreSQL

Field

Description

sequence

doublecloud.transfer.v1.endpoint.ObjectTransferStage Sequences

CREATE SEQUENCE ...

sequence_owned_by

doublecloud.transfer.v1.endpoint.ObjectTransferStage Owned sequences

CREATE SEQUENCE ... OWNED BY ...

sequence_set

doublecloud.transfer.v1.endpoint.ObjectTransferStage

table

doublecloud.transfer.v1.endpoint.ObjectTransferStage Tables

CREATE TABLE ...

primary_key

doublecloud.transfer.v1.endpoint.ObjectTransferStage Primary keys

ALTER TABLE ... ADD PRIMARY KEY ...

fk_constraint

doublecloud.transfer.v1.endpoint.ObjectTransferStage Foreign keys

ALTER TABLE ... ADD FOREIGN KEY ...

default_values

doublecloud.transfer.v1.endpoint.ObjectTransferStage Default values

ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...

constraint

doublecloud.transfer.v1.endpoint.ObjectTransferStage Constraints

ALTER TABLE ... ADD CONSTRAINT ...

index

doublecloud.transfer.v1.endpoint.ObjectTransferStage Indexes

CREATE INDEX ...

view

doublecloud.transfer.v1.endpoint.ObjectTransferStage Views

CREATE VIEW ...

materialized_view

doublecloud.transfer.v1.endpoint.ObjectTransferStage Materialized views

CREATE MATERIALIZED VIEW ...

function

doublecloud.transfer.v1.endpoint.ObjectTransferStage Functions

CREATE FUNCTION ...

trigger

doublecloud.transfer.v1.endpoint.ObjectTransferStage Triggers

CREATE TRIGGER ...

type

doublecloud.transfer.v1.endpoint.ObjectTransferStage Types

CREATE TYPE ...

rule

doublecloud.transfer.v1.endpoint.ObjectTransferStage Rules

CREATE RULE ...

collation

doublecloud.transfer.v1.endpoint.ObjectTransferStage Collations

CREATE COLLATION ...

policy

doublecloud.transfer.v1.endpoint.ObjectTransferStage Policies

CREATE POLICY ...

cast

doublecloud.transfer.v1.endpoint.ObjectTransferStage Casts

CREATE CAST ...

Field

Description

connection

doublecloud.transfer.v1.endpoint.PostgresConnection Database connection settings

database

string Database name

user

string User for database access.

password

doublecloud.transfer.v1.endpoint.Secret Password for database access.

include_tables

list Included tables

If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.

include_tables[]

string

exclude_tables

list Excluded tables

If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.

exclude_tables[]

string

slot_byte_lag_limit

int64 Maximum lag of replication slot (in bytes); after exceeding this limit replication will be aborted.

service_schema

string Database schema for service tables (__consumer_keeper, __data_transfer_mole_finder). Default is public

object_transfer_settings

doublecloud.transfer.v1.endpoint.PostgresObjectTransferSettings Select database objects to be transferred during activation or deactivation.

Field

Description

connection

doublecloud.transfer.v1.endpoint.PostgresConnection Database connection settings

security_groups

list Security groups

security_groups[]

string

database

string Database name

user

string User for database access.

password

doublecloud.transfer.v1.endpoint.Secret Password for database access.

cleanup_policy

doublecloud.transfer.v1.endpoint.CleanupPolicy Cleanup policy for activate, reactivate and reupload processes. Default is truncate.

Field

Description

key

string Name of the serializer parameter

value

string Value of the serializer parameter

Field

Description

serializer

one of: serializer_auto / serializer_json / serializer_debezium Select the serialization format automatically

serializer_auto

doublecloud.transfer.v1.endpoint.SerializerAuto Select the serialization format automatically

serializer_json

doublecloud.transfer.v1.endpoint.SerializerJSON Serialize data in json format

serializer_debezium

doublecloud.transfer.v1.endpoint.SerializerDebezium Serialize data in debezium format

Field

Description

Field

Description

serializer_parameters

list Settings of sterilization parameters as key-value pairs

serializer_parameters[]

doublecloud.transfer.v1.endpoint.DebeziumSerializerParameter

Field

Description

Field

Description

client_id

string

client_secret

string

scope

string

refresh_token

string

start_date

string

region

string

profiles

list

profiles[]

double

Field

Description

aws_key_id

string

aws_secret_key

string

aws_region_name

string

start_date

string

Field

Description

project_id

string

dataset_id

string

credentials_json

string

Field

Description

start_date

string

account_id

string

end_date

string

access_token

string

include_deleted

bool

fetch_thumbnail_images

bool

custom_insights

list

custom_insights[]

doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.InsightConfig

Field

Description

name

string

fields

list

fields[]

doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.Field

breakdowns

list

breakdowns[]

doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.Breakdown

action_breakdowns

list

action_breakdowns[]

doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.ActionBreakdown

Field

Description

customer_id

string

start_date

string

end_date

string

custom_queries

list

custom_queries[]

doublecloud.transfer.v1.endpoint.airbyte.GoogleAdsSource.CustomQuery

login_customer_id

string

conversion_window_days

double

Field

Description

developer_token

string

client_id

string

client_secret

string

access_token

string

refresh_token

string

Field

Description

query

string

table_name

string

Field

Description

start_date

string

access_token

string

Field

Description

start_date

string

account_ids

list

account_ids[]

int64

credentials

doublecloud.transfer.v1.endpoint.airbyte.LinkedinAdsSource.Credentials

Field

Description

credentials

one of: oauth / access_token

oauth

doublecloud.transfer.v1.endpoint.airbyte.LinkedinAdsSource.Credentials.OAuth

access_token

string

Field

Description

client_id

string

client_secret

string

refresh_token

string

Field

Description

host

string

port

int64

database

string

username

string

password

string

replication_method

doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.MSSQLReplicationMethod

ssl_method

doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLConfig

Field

Description

ssl_method

one of: unencrypted / encrypted_trust_server_certificate / encrypted_verify_certificate

unencrypted

doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLUnencrypted

encrypted_trust_server_certificate

doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLEncryptedTrusted

encrypted_verify_certificate

doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLEncryptedVerifyCert

Field

Description

Field

Description

host_name_in_certificate

string

Field

Description

Field

Description

host

string

port

int64

database

string

schemas

list

schemas[]

string

username

string

password

string

Field

Description

dataset

string

path_pattern

string

schema

string

format

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Format

provider

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Provider

Field

Description

Field

Description

delimiter

string

quote_char

string

escape_char

string

encoding

string

double_quote

bool

newlines_in_values

bool

block_size

int64

additional_reader_options

string

advanced_options

string

Field

Description

format

one of: csv / parquet / avro / jsonl

csv

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Csv

parquet

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Parquet

avro

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Avro

jsonl

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Jsonl

Field

Description

newlines_in_values

bool

unexpected_field_behavior

doublecloud.transfer.v1.endpoint.airbyte.S3Source.Jsonl.UnexpectedFieldBehavior

block_size

int64

Field

Description

buffer_size

int64

columns

list

columns[]

string

batch_size

int64

Field

Description

bucket

string

aws_access_key_id

string

aws_secret_access_key

string

path_prefix

string

endpoint

string

use_ssl

bool

verify_ssl_cert

bool

Field

Description

host

string

role

string

warehouse

string

database

string

schema

string

jdbc_url_params

string

credentials

doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource.Credentials

Field

Description

credentials

one of: oauth / basic_auth

oauth

doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource.Credentials.OAuth

basic_auth

doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource.Credentials.BasicAuth

Field

Description

username

string

password

string

Field

Description

client_id

string

client_secret

string

access_token

string

refresh_token

string