Transfer API models

DBTTransformer.Operation

Option	Description
OPERATION_UNSPECIFIED
OPERATION_BUILD
OPERATION_COMPILE
OPERATION_DEBUG
OPERATION_PARSE
OPERATION_RUN
OPERATION_SEED
OPERATION_SNAPSHOT
OPERATION_TEST

Flavor

Option	Description
FLAVOR_UNSPECIFIED
SMALL
MEDIUM
LARGE
TINY

RegularSnapshotScheduleInterval

Option	Description
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_UNSPECIFIED
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_15MIN
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_30MIN
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_HOUR
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_2HOUR
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_3HOUR
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_6HOUR
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_8HOUR
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_12HOUR
REGULAR_SNAPSHOT_SCHEDULE_INTERVAL_DAY

TransferStatus

Option	Description
TRANSFER_STATUS_UNSPECIFIED
CREATING	Transfer does some work before running
CREATED	Transfer created but not started by user
RUNNING	Transfer currently doing replication work
STOPPING	Transfer shutdown
STOPPED	Transfer stopped by user
ERROR	Transfer stopped by system
SNAPSHOTTING	Transfer copy snapshot
DONE	Transfer reach terminal phase

TransferType

Option	Description
TRANSFER_TYPE_UNSPECIFIED
SNAPSHOT_AND_INCREMENT	Snapshot and increment
SNAPSHOT_ONLY	Snapshot
INCREMENT_ONLY	Increment

endpoint.ClickhouseCleanupPolicy

Option	Description
CLICKHOUSE_CLEANUP_POLICY_UNSPECIFIED
CLICKHOUSE_CLEANUP_POLICY_DISABLED
CLICKHOUSE_CLEANUP_POLICY_DROP
CLICKHOUSE_CLEANUP_POLICY_TRUNCATE

endpoint.CleanupPolicy

Option	Description
CLEANUP_POLICY_UNSPECIFIED
DISABLED	Don't cleanup
DROP	Drop
TRUNCATE	Truncate

endpoint.ColumnType

Option	Description
COLUMN_TYPE_UNSPECIFIED
INT32
INT16
INT8
UINT64
UINT32
UINT16
UINT8
DOUBLE
BOOLEAN
STRING
UTF8
ANY
DATETIME
INT64

endpoint.ObjectTransferStage

Option	Description
OBJECT_TRANSFER_STAGE_UNSPECIFIED
BEFORE_DATA	Before data transfer
AFTER_DATA	After data transfer
NEVER	Don't copy

endpoint.CoralogixSeverity

Option	Description
CORALOGIX_SEVERITY_UNSPECIFIED
CORALOGIX_SEVERITY_DEBUG
CORALOGIX_SEVERITY_VERBOSE
CORALOGIX_SEVERITY_INFO
CORALOGIX_SEVERITY_WARN
CORALOGIX_SEVERITY_ERROR
CORALOGIX_SEVERITY_CRITICAL

endpoint.KafkaCompressionEncoding

Option	Description
KAFKA_COMPRESSION_ENCODING_UNSPECIFIED
KAFKA_COMPRESSION_ENCODING_GZIP
KAFKA_COMPRESSION_ENCODING_SNAPPY
KAFKA_COMPRESSION_ENCODING_LZ4
KAFKA_COMPRESSION_ENCODING_ZSTD

endpoint.KafkaMechanism

Option	Description
KAFKA_MECHANISM_UNSPECIFIED
KAFKA_MECHANISM_SHA256
KAFKA_MECHANISM_SHA512

endpoint.MetricaStreamType

Option	Description
METRICA_STREAM_TYPE_UNSPECIFIED
METRICA_STREAM_TYPE_VISITS
METRICA_STREAM_TYPE_HITS_V2

endpoint.ObjectStorageCodec

Option	Description
OBJECT_STORAGE_CODEC_UNSPECIFIED
UNCOMPRESSED
GZIP

endpoint.ObjectStorageReaderFormat.Jsonl.UnexpectedFieldBehavior

Option	Description
UNEXPECTED_FIELD_BEHAVIOR_UNSPECIFIED
UNEXPECTED_FIELD_BEHAVIOR_IGNORE
UNEXPECTED_FIELD_BEHAVIOR_INFER
UNEXPECTED_FIELD_BEHAVIOR_ERROR

endpoint.ObjectStorageSerializationFormat

Option	Description
OBJECT_STORAGE_SERIALIZATION_FORMAT_UNSPECIFIED
OBJECT_STORAGE_SERIALIZATION_FORMAT_JSON
OBJECT_STORAGE_SERIALIZATION_FORMAT_CSV
OBJECT_STORAGE_SERIALIZATION_FORMAT_RAW
OBJECT_STORAGE_SERIALIZATION_FORMAT_PARQUET

endpoint.ObjectStorageUnparsed

Option	Description
OBJECT_STORAGE_UNPARSED_UNSPECIFIED
OBJECT_STORAGE_UNPARSED_RETRY
OBJECT_STORAGE_UNPARSED_FAIL
OBJECT_STORAGE_UNPARSED_CONTINUE

endpoint.ProtoMessagePackageType

Option	Description
PROTO_MESSAGE_PACKAGE_TYPE_UNSPECIFIED
PROTOSEQ
REPEATED
SINGLE_MESSAGE

endpoint.airbyte.FacebookMarketingSource.ActionBreakdown

Option	Description
ACTION_BREAKDOWN_UNSPECIFIED
ACTION_CANVAS_COMPONENT_NAME
ACTION_CAROUSEL_CARD_ID
ACTION_CAROUSEL_CARD_NAME
ACTION_DESTINATION
ACTION_DEVICE
ACTION_REACTION
ACTION_TARGET_ID
ACTION_TYPE
ACTION_VIDEO_SOUND
ACTION_VIDEO_TYPE

endpoint.airbyte.FacebookMarketingSource.Breakdown

Option	Description
BREAKDOWN_UNSPECIFIED
AD_FORMAT_ASSET
AGE
APP_ID
BODY_ASSET
CALL_TO_ACTION_ASSET
COUNTRY
DESCRIPTION_ASSET
DEVICE_PLATFORM
DMA
FREQUENCY_VALUE
GENDER
HOURLY_STATS_AGGREGATED_BY_ADVERTISER_TIME_ZONE
HOURLY_STATS_AGGREGATED_BY_AUDIENCE_TIME_ZONE
IMAGE_ASSET
IMPRESSION_DEVICE
LINK_URL_ASSET
PLACE_PAGE_ID
PLATFORM_POSITION
PRODUCT_ID
PUBLISHER_PLATFORM
REGION
SKAN_CONVERSION_ID
TITLE_ASSET
VIDEO_ASSE

endpoint.airbyte.FacebookMarketingSource.Field

Option	Description
FIELD_UNSPECIFIED
ACCOUNT_CURRENCY
ACCOUNT_ID
ACCOUNT_NAME
ACTION_VALUES
ACTIONS
AD_BID_VALUE
AD_CLICK_ACTIONS
AD_ID
AD_IMPRESSION_ACTIONS
AD_NAME
ADSET_BID_VALUE
ADSET_END
ADSET_ID
ADSET_NAME
ADSET_START
AGE_TARGETING
ATTRIBUTION_SETTING
AUCTION_BID
AUCTION_COMPETITIVENESS
AUCTION_MAX_COMPETITOR_BID
BUYING_TYPE
CAMPAIGN_ID
CAMPAIGN_NAME
CANVAS_AVG_VIEW_PERCENT
CANVAS_AVG_VIEW_TIME
CATALOG_SEGMENT_ACTIONS
CATALOG_SEGMENT_VALUE
CATALOG_SEGMENT_VALUE_MOBILE_PURCHASE_ROAS
CATALOG_SEGMENT_VALUE_OMNI_PURCHASE_ROAS
CATALOG_SEGMENT_VALUE_WEBSITE_PURCHASE_ROAS
CLICKS
CONVERSION_RATE_RANKING
CONVERSION_VALUES
CONVERSIONS
CONVERTED_PRODUCT_QUANTITY
CONVERTED_PRODUCT_VALUE
COST_PER_15_SEC_VIDEO_VIEW
COST_PER_2_SEC_CONTINUOUS_VIDEO_VIEW
COST_PER_ACTION_TYPE
COST_PER_AD_CLICK
COST_PER_CONVERSION
COST_PER_DDA_COUNTBY_CONVS
COST_PER_ESTIMATED_AD_RECALLERS
COST_PER_INLINE_LINK_CLICK
COST_PER_INLINE_POST_ENGAGEMENT
COST_PER_ONE_THOUSAND_AD_IMPRESSION
COST_PER_OUTBOUND_CLICK
COST_PER_THRUPLAY
COST_PER_UNIQUE_ACTION_TYPE
COST_PER_UNIQUE_CLICK
COST_PER_UNIQUE_CONVERSION
COST_PER_UNIQUE_INLINE_LINK_CLICK
COST_PER_UNIQUE_OUTBOUND_CLICK
CPC
CPM
CPP
CREATED_TIME
CTR
DATE_START
DATE_STOP
DDA_COUNTBY_CONVS
DDA_RESULTS
ENGAGEMENT_RATE_RANKING
ESTIMATED_AD_RECALL_RATE
ESTIMATED_AD_RECALL_RATE_LOWER_BOUND
ESTIMATED_AD_RECALL_RATE_UPPER_BOUND
ESTIMATED_AD_RECALLERS
ESTIMATED_AD_RECALLERS_LOWER_BOUND
ESTIMATED_AD_RECALLERS_UPPER_BOUND
FREQUENCY
FULL_VIEW_IMPRESSIONS
FULL_VIEW_REACH
GENDER_TARGETING
IMPRESSIONS
INLINE_LINK_CLICK_CTR
INLINE_LINK_CLICKS
INLINE_POST_ENGAGEMENT
INSTANT_EXPERIENCE_CLICKS_TO_OPEN
INSTANT_EXPERIENCE_CLICKS_TO_START
INSTANT_EXPERIENCE_OUTBOUND_CLICKS
INTERACTIVE_COMPONENT_TAP
LABELS
LOCATION
MOBILE_APP_PURCHASE_ROAS
OBJECTIVE
OPTIMIZATION_GOAL
OUTBOUND_CLICKS
OUTBOUND_CLICKS_CTR
PLACE_PAGE_NAME
PURCHASE_ROAS
QUALIFYING_QUESTION_QUALIFY_ANSWER_RATE
QUALITY_RANKING
QUALITY_SCORE_ECTR
QUALITY_SCORE_ECVR
QUALITY_SCORE_ORGANIC
REACH
SOCIAL_SPEND
SPEND
TOTAL_POSTBACKS
UNIQUE_ACTIONS
UNIQUE_CLICKS
UNIQUE_CONVERSIONS
UNIQUE_CTR
UNIQUE_INLINE_LINK_CLICK_CTR
UNIQUE_INLINE_LINK_CLICKS
UNIQUE_LINK_CLICKS_CTR
UNIQUE_OUTBOUND_CLICKS
UNIQUE_OUTBOUND_CLICKS_CTR
UNIQUE_VIDEO_CONTINUOUS_2_SEC_WATCHED_ACTIONS
UNIQUE_VIDEO_VIEW_15_SEC
UPDATED_TIME
VIDEO_15_SEC_WATCHED_ACTIONS
VIDEO_30_SEC_WATCHED_ACTIONS
VIDEO_AVG_TIME_WATCHED_ACTIONS
VIDEO_CONTINUOUS_2_SEC_WATCHED_ACTIONS
VIDEO_P100_WATCHED_ACTIONS
VIDEO_P25_WATCHED_ACTIONS
VIDEO_P50_WATCHED_ACTIONS
VIDEO_P75_WATCHED_ACTIONS
VIDEO_P95_WATCHED_ACTIONS
VIDEO_PLAY_ACTIONS
VIDEO_PLAY_CURVE_ACTIONS
VIDEO_PLAY_RETENTION_0_TO_15S_ACTIONS
VIDEO_PLAY_RETENTION_20_TO_60S_ACTIONS
VIDEO_PLAY_RETENTION_GRAPH_ACTIONS
VIDEO_THRUPLAY_WATCHED_ACTIONS
VIDEO_TIME_WATCHED_ACTIONS
WEBSITE_CTR
WEBSITE_PURCHASE_ROAS
WISH_BID

endpoint.airbyte.JiraSource.Expand

Option	Description
EXPAND_UNSPECIFIED
RENDERED_FIELDS
TRANSITIONS
CHANGELOG

endpoint.airbyte.MSSQLSource.MSSQLReplicationMethod

Option	Description
MSSQL_REPLICATION_METHOD_UNSPECIFIED
STANDARD
CDC

endpoint.airbyte.S3Source.Jsonl.UnexpectedFieldBehavior

Option	Description
UNEXPECTED_FIELD_BEHAVIOR_UNSPECIFIED
UNEXPECTED_FIELD_BEHAVIOR_IGNORE
UNEXPECTED_FIELD_BEHAVIOR_INFER
UNEXPECTED_FIELD_BEHAVIOR_ERROR

Endpoint

Field	Description
`id`	string
`project_id`	string
`name`	string
`description`	string
`labels`	list
`labels[]`	doublecloud.transfer.v1.Endpoint.LabelsEntry
`settings`	doublecloud.transfer.v1.EndpointSettings

Endpoint.LabelsEntry

Field	Description
`key`	string
`value`	string

EndpointSettings

Field	Description
`settings`	one of: `mysql_source` / `postgres_source` / `kafka_source` / `mongo_source` / `clickhouse_source` / `mysql_target` / `postgres_target` / `clickhouse_target` / `object_storage_target` / `kafka_target` / `mongo_target` / `s3_source` / `aws_cloudtrail_source` / `big_query_source` / `facebook_marketing_source` / `google_ads_source` / `amazon_ads_source` / `instagram_source` / `linkedin_ads_source` / `mssql_source` / `redshift_source` / `snowflake_source` / `object_storage_source` / `datadog_target` / `jira_source` / `hubspot_source` / `coralogix_target` / `bigquery_target` / `metrica_source` / `dynamodb_source` / `kinesis_source`
`mysql_source`	doublecloud.transfer.v1.endpoint.MysqlSource
`postgres_source`	doublecloud.transfer.v1.endpoint.PostgresSource
`kafka_source`	doublecloud.transfer.v1.endpoint.KafkaSource
`mongo_source`	doublecloud.transfer.v1.endpoint.MongoSource
`clickhouse_source`	doublecloud.transfer.v1.endpoint.ClickhouseSource
`mysql_target`	doublecloud.transfer.v1.endpoint.MysqlTarget
`postgres_target`	doublecloud.transfer.v1.endpoint.PostgresTarget
`clickhouse_target`	doublecloud.transfer.v1.endpoint.ClickhouseTarget
`object_storage_target`	doublecloud.transfer.v1.endpoint.ObjectStorageTarget
`kafka_target`	doublecloud.transfer.v1.endpoint.KafkaTarget
`mongo_target`	doublecloud.transfer.v1.endpoint.MongoTarget
`s3_source`	doublecloud.transfer.v1.endpoint.airbyte.S3Source
`aws_cloudtrail_source`	doublecloud.transfer.v1.endpoint.airbyte.AWSCloudTrailSource
`big_query_source`	doublecloud.transfer.v1.endpoint.airbyte.BigQuerySource
`facebook_marketing_source`	doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource
`google_ads_source`	doublecloud.transfer.v1.endpoint.airbyte.GoogleAdsSource
`amazon_ads_source`	doublecloud.transfer.v1.endpoint.airbyte.AmazonAdsSource
`instagram_source`	doublecloud.transfer.v1.endpoint.airbyte.InstagramSource
`linkedin_ads_source`	doublecloud.transfer.v1.endpoint.airbyte.LinkedinAdsSource
`mssql_source`	doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource
`redshift_source`	doublecloud.transfer.v1.endpoint.airbyte.RedshiftSource
`snowflake_source`	doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource
`object_storage_source`	doublecloud.transfer.v1.endpoint.ObjectStorageSource
`datadog_target`	doublecloud.transfer.v1.endpoint.DatadogTarget
`jira_source`	doublecloud.transfer.v1.endpoint.airbyte.JiraSource
`hubspot_source`	doublecloud.transfer.v1.endpoint.airbyte.HubspotSource
`coralogix_target`	doublecloud.transfer.v1.endpoint.CoralogixTarget
`bigquery_target`	doublecloud.transfer.v1.endpoint.BigQueryTarget
`metrica_source`	doublecloud.transfer.v1.endpoint.MetricaSource
`dynamodb_source`	doublecloud.transfer.v1.endpoint.airbyte.DynamodbSource
`kinesis_source`	doublecloud.transfer.v1.endpoint.KinesisSource

CreateEndpointRequest

Field	Description
`project_id`	string
`name`	string
`description`	string
`labels`	list
`labels[]`	doublecloud.transfer.v1.CreateEndpointRequest.LabelsEntry
`settings`	doublecloud.transfer.v1.EndpointSettings

CreateEndpointRequest.LabelsEntry

Field	Description
`key`	string
`value`	string

DeleteEndpointRequest

Field	Description
`endpoint_id`	string

GetEndpointRequest

Field	Description
`endpoint_id`	string

ListEndpointsRequest

Field	Description
`project_id`	string Identifier of the folder containing the endpoints to be listed.
`page`	doublecloud.v1.Paging The maximum number of endpoints to be sent in the response message. If the folder contains more endpoints than `page_size`, `next_page_token` will be included in the response message. Include it into the subsequent `ListEndpointRequest` to fetch the next page. Defaults to `100` if not specified. The maximum allowed value for this field is `500`.

ListEndpointsResponse

Field	Description
`endpoints`	list The list of endpoints. If there are more endpoints in the folder, then `next_page_token` is a non-empty string to be included into the subsequent `ListEndpointsRequest` to fetch the next endpoints page.
`endpoints[]`	doublecloud.transfer.v1.Endpoint
`next_page`	doublecloud.v1.NextPage Opaque value identifying the next endpoints page. This field is empty if there are no more endpoints in the folder. Otherwise, it is non-empty and should be included in the subsequent `ListEndpointsRequest` to fetch the next endpoints page.

UpdateEndpointRequest

Field	Description
`endpoint_id`	string Identifier of the endpoint to be updated.
`name`	string The new endpoint name. Must be unique within the folder.
`description`	string The new description for the endpoint.
`labels`	list
`labels[]`	doublecloud.transfer.v1.UpdateEndpointRequest.LabelsEntry
`settings`	doublecloud.transfer.v1.EndpointSettings The new endpoint name. Must be unique within the folder.

UpdateEndpointRequest.LabelsEntry

Field	Description
`key`	string
`value`	string

GetOperationRequest

Field	Description
`operation_id`	string

AutoSettings

Field

Description

CloudFunctionTransformer

Field	Description
`name`	string
`name_space`	string
`options`	doublecloud.transfer.v1.endpoint.DataTransformationOptions

ColumnsFilter

Field	Description
`include_columns`	list List of columns that will be included to transfer
`include_columns[]`	string
`exclude_columns`	list List of columns that will be excluded to transfer
`exclude_columns[]`	string

DBTTransformer

Field	Description
`git_repository_link`	string
`git_branch`	string
`profile_name`	string
`operation`	doublecloud.transfer.v1.DBTTransformer.Operation

DataObjects

Field	Description
`include_objects`	list
`include_objects[]`	string

DedicatedRuntime

Field	Description
`flavor`	doublecloud.transfer.v1.Flavor
`settings`	doublecloud.transfer.v1.Settings

FilterColumnsTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of the tables to filter using lists of included and excluded tables.
`columns`	doublecloud.transfer.v1.ColumnsFilter List of the columns to transfer to the target tables using lists of included and excluded columns.

FilterRowsTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of included and excluded tables
`filter`	string Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#append-only-sources

IncrementalTable

Field	Description
`table_namespace`	string
`table_name`	string
`cursor_column`	string
`initial_state`	string

JsonParserTransformer

Field	Description
`topic`	string
`config`	doublecloud.transfer.v1.endpoint.GenericParserCommon

ManualSettings

Field	Description
`network_id`	string

MaskFieldTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of included and excluded tables
`columns`	list Specify the name of the column for data masking (a regular expression).
`columns[]`	string
`function`	doublecloud.transfer.v1.MaskFunction Mask function

MaskFunction

Field	Description
`mask_function`	one of: `mask_function_hash` Hash mask function
`mask_function_hash`	doublecloud.transfer.v1.MaskFunctionHash Hash mask function

MaskFunctionHash

Field	Description
`user_defined_salt`	string This string will be used in the HMAC(sha256, salt) function applied to the column data.

NumberToFloatTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter

RawCdcDocGroupTransformer

Field	Description
`keys`	list
`keys[]`	string
`fields`	list
`fields[]`	string

RawDocGroupTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter
`keys`	list
`keys[]`	string
`fields`	list
`fields[]`	string

RegularSnapshot

Field	Description
`mode`	one of: `settings` / `disabled`
`settings`	doublecloud.transfer.v1.RegularSnapshotSettings
`disabled`	doublecloud.transfer.v1.RegularSnapshotDisabled

RegularSnapshotDisabled

Field

Description

RegularSnapshotSettings

Field	Description
`schedule`	doublecloud.transfer.v1.RegularSnapshotScheduleInterval
`tables`	list
`tables[]`	doublecloud.transfer.v1.IncrementalTable
`cron_expression`	string

RenameTable

Field	Description
`original_name`	doublecloud.transfer.v1.Table Specify the current names of the table in the source
`new_name`	doublecloud.transfer.v1.Table Specify the new names for this table in the target

RenameTablesTransformer

Field	Description
`rename_tables`	list List of renaming rules
`rename_tables[]`	doublecloud.transfer.v1.RenameTable

ReplacePrimaryKeyTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of included and excluded tables
`keys`	list List of columns to be used as primary keys
`keys[]`	string

Runtime

Field	Description
`runtime`	one of: `serverless_runtime` / `dedicated_runtime`
`serverless_runtime`	doublecloud.transfer.v1.ServerlessRuntime
`dedicated_runtime`	doublecloud.transfer.v1.DedicatedRuntime

SQLTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter
`query`	string

ServerlessRuntime

Field	Description
`job_count`	int64

Settings

Field	Description
`settings`	one of: `auto_settings` / `manual_settings`
`auto_settings`	doublecloud.transfer.v1.AutoSettings
`manual_settings`	doublecloud.transfer.v1.ManualSettings

SharderTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of included and excluded tables
`columns`	doublecloud.transfer.v1.ColumnsFilter List of included and excluded columns
`shards_count`	int64 Number of shards

SkipEventsTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter

Table

Field	Description
`name_space`	string
`name`	string

TableSplitterTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of included and excluded tables
`columns`	list Specify the columns in the tables to be partitioned.
`columns[]`	string
`splitter`	string Specify the split string to be used for merging components in a new table name.

TablesFilter

Field	Description
`include_tables`	list List of tables that will be included to transfer
`include_tables[]`	string
`exclude_tables`	list List of tables that will be excluded to transfer
`exclude_tables[]`	string

ToStringTransformer

Field	Description
`tables`	doublecloud.transfer.v1.TablesFilter List of included and excluded tables
`columns`	doublecloud.transfer.v1.ColumnsFilter List of included and excluded columns

Transfer

Field	Description
`id`	string
`project_id`	string
`name`	string
`description`	string
`labels`	list
`labels[]`	doublecloud.transfer.v1.Transfer.LabelsEntry
`source`	doublecloud.transfer.v1.Endpoint
`target`	doublecloud.transfer.v1.Endpoint
`runtime`	doublecloud.transfer.v1.Runtime
`status`	doublecloud.transfer.v1.TransferStatus
`type`	doublecloud.transfer.v1.TransferType
`warning`	string
`regular_snapshot`	doublecloud.transfer.v1.RegularSnapshot
`transformation`	doublecloud.transfer.v1.Transformation
`data_objects`	doublecloud.transfer.v1.DataObjects

Transfer.LabelsEntry

Field	Description
`key`	string
`value`	string

Transformation

Field	Description
`transformers`	list Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list.
`transformers[]`	doublecloud.transfer.v1.Transformer

Transformer

Field	Description
`transformer`	one of: `mask_field` / `filter_columns` / `skip_events` / `rename_tables` / `replace_primary_key` / `convert_to_string` / `sharder_transformer` / `sql` / `dbt` / `table_splitter_transformer` / `filter_rows` / `number_to_float_transformer` / `json_mapper_transformer` / `cloud_function_transformer`
`mask_field`	doublecloud.transfer.v1.MaskFieldTransformer
`filter_columns`	doublecloud.transfer.v1.FilterColumnsTransformer
`skip_events`	doublecloud.transfer.v1.SkipEventsTransformer
`rename_tables`	doublecloud.transfer.v1.RenameTablesTransformer
`replace_primary_key`	doublecloud.transfer.v1.ReplacePrimaryKeyTransformer
`convert_to_string`	doublecloud.transfer.v1.ToStringTransformer
`sharder_transformer`	doublecloud.transfer.v1.SharderTransformer
`sql`	doublecloud.transfer.v1.SQLTransformer
`dbt`	doublecloud.transfer.v1.DBTTransformer
`table_splitter_transformer`	doublecloud.transfer.v1.TableSplitterTransformer
`filter_rows`	doublecloud.transfer.v1.FilterRowsTransformer
`number_to_float_transformer`	doublecloud.transfer.v1.NumberToFloatTransformer
`json_mapper_transformer`	doublecloud.transfer.v1.JsonParserTransformer
`cloud_function_transformer`	doublecloud.transfer.v1.CloudFunctionTransformer

ActivateTransferRequest

Field	Description
`transfer_id`	string

CreateTransferRequest

Field	Description
`source_id`	string
`target_id`	string
`description`	string
`project_id`	string
`runtime`	doublecloud.transfer.v1.Runtime
`type`	doublecloud.transfer.v1.TransferType
`name`	string
`labels`	list
`labels[]`	doublecloud.transfer.v1.CreateTransferRequest.LabelsEntry
`regular_snapshot`	doublecloud.transfer.v1.RegularSnapshot
`transformation`	doublecloud.transfer.v1.Transformation
`data_objects`	doublecloud.transfer.v1.DataObjects

CreateTransferRequest.LabelsEntry

Field	Description
`key`	string
`value`	string

DeactivateTransferRequest

Field	Description
`transfer_id`	string

DeleteExporterConnectionInfoMetadata

Field	Description
`transfer_id`	string The ID of the Transfer

DeleteExporterConnectionInfoRequest

Field	Description
`transfer_id`	string The ID of the Transfer

DeleteTransferRequest

Field	Description
`transfer_id`	string

GetMetricsRequest

Field	Description
`transfer_id`	string The ID of Transfer

GetTransferRequest

Field	Description
`transfer_id`	string

ListTransfersRequest

Field	Description
`project_id`	string Identifier of the folder containing the transfers to be listed.
`page`	doublecloud.v1.Paging The maximum number of transfers to be sent in the response message. If the folder contains more transfers than `page_size`, `next_page_token` will be included in the response message. Include it into the subsequent `ListTransfersRequest` to fetch the next page. Defaults to `100` if not specified. The maximum allowed value for this field is `500`.

ListTransfersResponse

Field	Description
`transfers`	list The list of transfers. If there are more transfers in the folder, then `next_page_token` is a non-empty string to be included into the subsequent `ListTransfersRequest` to fetch the next transfers page.
`transfers[]`	doublecloud.transfer.v1.Transfer
`next_page_token`	string Opaque value identifying the next transfers page. This field is empty if there are no more transfers in the folder. Otherwise it is non-empty and should be included in the subsequent `ListTransfersRequest` to fetch the next transfers page.

MetricExporterConnectionInfoMetadata

Field	Description
`connection_string`	string String to use in clients. i.e. url for config
`user`	string User for HTTP basic auth.
`password`	string Password for HTTP basic auth.

MetricExporterConnectionInfoRequest

Field	Description
`transfer_id`	string The ID of the Transfer

TransferMetrics

Field	Description
`metrics`	list The list of the transfer metrics
`metrics[]`	doublecloud.transfer.v1.TransferMetrics.Metric

TransferMetrics.Metric

Field	Description
`help`	string
`type`	string
`value`	string

UpdateTransferRequest

Field	Description
`transfer_id`	string Identifier of the transfer to be updated.
`description`	string The new description for the transfer.
`runtime`	doublecloud.transfer.v1.Runtime
`name`	string The new transfer name. Must be unique within the folder.
`labels`	list
`labels[]`	doublecloud.transfer.v1.UpdateTransferRequest.LabelsEntry
`regular_snapshot`	doublecloud.transfer.v1.RegularSnapshot
`transformation`	doublecloud.transfer.v1.Transformation
`data_objects`	doublecloud.transfer.v1.DataObjects

UpdateTransferRequest.LabelsEntry

Field	Description
`key`	string
`value`	string

endpoint.BigQueryTarget

Field	Description
`project_id`	string
`dataset_id`	string
`credentials_json`	string

endpoint.ClickhouseConnection

Field	Description
`connection`	one of: `connection_options`
`connection_options`	doublecloud.transfer.v1.endpoint.ClickhouseConnectionOptions

endpoint.ClickhouseConnectionOptions

Field	Description
`address`	one of: `on_premise` / `mdb_cluster_id`
`on_premise`	doublecloud.transfer.v1.endpoint.OnPremiseClickhouse
`mdb_cluster_id`	string
`user`	string
`password`	doublecloud.transfer.v1.endpoint.Secret
`database`	string Database

endpoint.ClickhouseInsertOptions

Field	Description
`materialized_views_ignore_errors`	bool

endpoint.ClickhouseMigrationOptions

Field	Description
`add_new_columns`	bool

endpoint.ClickhouseShard

Field	Description
`name`	string
`hosts`	list
`hosts[]`	string

endpoint.ClickhouseSharding

Field	Description
`sharding`	one of: `column_value_hash` / `custom_mapping` / `transfer_id` / `round_robin`
`column_value_hash`	doublecloud.transfer.v1.endpoint.ClickhouseSharding.ColumnValueHash
`custom_mapping`	doublecloud.transfer.v1.endpoint.ClickhouseSharding.ColumnValueMapping
`transfer_id`	google.protobuf.Empty
`round_robin`	google.protobuf.Empty

endpoint.ClickhouseSharding.ColumnValueHash

Field	Description
`column_name`	string

endpoint.ClickhouseSharding.ColumnValueMapping

Field	Description
`column_name`	string
`mapping`	list
`mapping[]`	doublecloud.transfer.v1.endpoint.ClickhouseSharding.ColumnValueMapping.ValueToShard

endpoint.ClickhouseSharding.ColumnValueMapping.ValueToShard

Field	Description
`column_value`	doublecloud.transfer.v1.endpoint.ColumnValue
`shard_name`	string

endpoint.ClickhouseSource

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.ClickhouseConnection
`include_tables`	list While list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.
`include_tables[]`	string
`exclude_tables`	list Exclude list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.
`exclude_tables[]`	string

endpoint.ClickhouseTarget

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.ClickhouseConnection
`alt_names`	list Alternative table names in target
`alt_names[]`	doublecloud.transfer.v1.endpoint.AltName
`migration_options`	doublecloud.transfer.v1.endpoint.ClickhouseMigrationOptions
`cleanup_policy`	doublecloud.transfer.v1.endpoint.ClickhouseCleanupPolicy
`sharding`	doublecloud.transfer.v1.endpoint.ClickhouseSharding
`insert_options`	doublecloud.transfer.v1.endpoint.ClickhouseInsertOptions
`clickhouse_cluster_name`	string

endpoint.OnPremiseClickhouse

Field	Description
`shards`	list
`shards[]`	doublecloud.transfer.v1.endpoint.ClickhouseShard
`http_port`	int64
`native_port`	int64
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode

endpoint.AltName

Field	Description
`from_name`	string Source table name
`to_name`	string Target table name

endpoint.ColSchema

Field	Description
`name`	string
`type`	doublecloud.transfer.v1.endpoint.ColumnType
`key`	bool
`required`	bool
`path`	string

endpoint.ColumnValue

Field	Description
`value`	one of: `string_value`
`string_value`	string

endpoint.DataSchema

Field	Description
`schema`	one of: `json_fields` / `fields`
`json_fields`	string
`fields`	doublecloud.transfer.v1.endpoint.FieldList

endpoint.DataTransformationOptions

Field	Description
`cloud_function`	string Cloud function
`number_of_retries`	int64 Number of retries
`buffer_size`	string Buffer size for function
`buffer_flush_interval`	string Flush interval
`invocation_timeout`	string Invocation timeout
`cloud_function_url`	string
`headers`	list
`headers[]`	doublecloud.transfer.v1.endpoint.HeaderValue

endpoint.FieldList

Field	Description
`fields`	list Column schema
`fields[]`	doublecloud.transfer.v1.endpoint.ColSchema

endpoint.HeaderValue

Field	Description
`key`	string
`value`	string

endpoint.NoAuth

Field

Description

endpoint.Secret

Field	Description
`value`	one of: `raw` Raw secret value
`raw`	string Raw secret value

endpoint.TLSConfig

Field

Description

ca_certificate

string CA certificate

X.509 certificate of the certificate authority which issued the server's certificate, in PEM format. When CA certificate is specified TLS is used to connect to the server.

endpoint.TLSMode

Field	Description
`tls_mode`	one of: `disabled` / `enabled`
`disabled`	google.protobuf.Empty
`enabled`	doublecloud.transfer.v1.endpoint.TLSConfig

endpoint.CoralogixTarget

Field	Description
`token`	string
`domain`	string
`application_name`	string
`column_mapping`	doublecloud.transfer.v1.endpoint.CoralogixTarget.ColumnMapping

endpoint.CoralogixTarget.ColumnMapping

Field	Description
`timestamp`	string
`severity`	string
`message_template`	string
`class_name`	string
`method_name`	string
`thread_id`	string
`category`	string
`subsystem`	string
`host`	string
`known_severities`	list
`known_severities[]`	doublecloud.transfer.v1.endpoint.CoralogixTarget.SeverityMap

endpoint.CoralogixTarget.SeverityMap

Field	Description
`name`	string
`value`	doublecloud.transfer.v1.endpoint.CoralogixSeverity

endpoint.DatadogTarget

Field	Description
`client_api_key`	string
`host`	string
`column_mapping`	doublecloud.transfer.v1.endpoint.DatadogTarget.ColumnMapping

endpoint.DatadogTarget.ColumnMapping

Field	Description
`source`	string
`tags`	list
`tags[]`	string
`host`	string
`message_template`	string
`service`	string

endpoint.DeltaLakeSource

Field	Description
`provider`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Provider
`table`	doublecloud.transfer.v1.endpoint.DeltaResultTable
`settings`	doublecloud.transfer.v1.endpoint.DeltaSettings

endpoint.DeltaResultTable

Field

Description

endpoint.DeltaSettings

Field	Description
`region`	string

endpoint.KafkaAuth

Field	Description
`security`	one of: `sasl` / `no_auth` Authentication with SASL
`sasl`	doublecloud.transfer.v1.endpoint.KafkaSaslSecurity Authentication with SASL
`no_auth`	doublecloud.transfer.v1.endpoint.NoAuth No authentication

endpoint.KafkaConnectionOptions

Field	Description
`connection`	one of: `cluster_id` / `on_premise` Managed Service for Kafka cluster ID
`cluster_id`	string Managed Service for Kafka cluster ID
`on_premise`	doublecloud.transfer.v1.endpoint.OnPremiseKafka Connection options for on-premise Kafka

endpoint.KafkaSaslSecurity

Field	Description
`user`	string User name
`mechanism`	doublecloud.transfer.v1.endpoint.KafkaMechanism SASL mechanism for authentication
`password`	doublecloud.transfer.v1.endpoint.Secret Password for user

endpoint.KafkaSource

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.KafkaConnectionOptions Connection settings
`auth`	doublecloud.transfer.v1.endpoint.KafkaAuth Authentication settings
`topic_name`	string Full source topic name Deprecated in favor of topic names
`parser`	doublecloud.transfer.v1.endpoint.Parser Data parsing rules
`topic_names`	list List of topic names to read
`topic_names[]`	string

endpoint.KafkaTarget

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.KafkaConnectionOptions Connection settings
`auth`	doublecloud.transfer.v1.endpoint.KafkaAuth Authentication settings
`topic_settings`	doublecloud.transfer.v1.endpoint.KafkaTargetTopicSettings Target topic settings
`serializer`	doublecloud.transfer.v1.endpoint.Serializer Data serialization format settings
`compression`	doublecloud.transfer.v1.endpoint.KafkaCompressionEncoding

endpoint.KafkaTargetTopic

Field	Description
`topic_name`	string Topic name
`save_tx_order`	bool Save transactions order Not to split events queue into separate per-table queues.

endpoint.KafkaTargetTopicSettings

Field	Description
`topic_settings`	one of: `topic` / `topic_prefix` Full topic name
`topic`	doublecloud.transfer.v1.endpoint.KafkaTargetTopic Full topic name
`topic_prefix`	string Topic prefix Analogue of the Debezium setting database.server.name. Messages will be sent to topic with name <topic_prefix>..<table_name>.
`topic_config_entries`	list ConfigEntries for newly created topics by transfer
`topic_config_entries[]`	doublecloud.transfer.v1.endpoint.TopicConfigEntry

endpoint.OnPremiseKafka

Field	Description
`broker_urls`	list Kafka broker URLs
`broker_urls[]`	string
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode TLS settings for broker connection. Disabled by default.

endpoint.TopicConfigEntry

Field	Description
`config_name`	string
`config_value`	string

endpoint.KinesisSource

Field	Description
`region`	string
`stream_name`	string
`aws_access_key_id`	string
`aws_secret_access_key`	string
`parser`	doublecloud.transfer.v1.endpoint.Parser Data parsing rules

endpoint.MetricaSource

Field	Description
`counter_ids`	list
`counter_ids[]`	int64
`token`	doublecloud.transfer.v1.endpoint.Secret
`streams`	list
`streams[]`	doublecloud.transfer.v1.endpoint.MetricaStream
`enable_dashboard`	bool

endpoint.MetricaStream

Field	Description
`type`	doublecloud.transfer.v1.endpoint.MetricaStreamType

endpoint.MongoCollection

Field	Description
`database_name`	string
`collection_name`	string

endpoint.MongoConnection

Field	Description
`connection`	one of: `connection_options`
`connection_options`	doublecloud.transfer.v1.endpoint.MongoConnectionOptions

endpoint.MongoConnectionOptions

Field	Description
`address`	one of: `on_premise` / `srv`
`on_premise`	doublecloud.transfer.v1.endpoint.OnPremiseMongo
`srv`	doublecloud.transfer.v1.endpoint.SrvMongo
`user`	string User name
`password`	doublecloud.transfer.v1.endpoint.Secret Password for user
`auth_source`	string Database name associated with the credentials

endpoint.MongoSource

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.MongoConnection
`collections`	list List of collections for replication. Empty list implies replication of all tables on the deployment. Allowed to use * as collection name.
`collections[]`	doublecloud.transfer.v1.endpoint.MongoCollection
`excluded_collections`	list List of forbidden collections for replication. Allowed to use * as collection name for forbid all collections of concrete schema.
`excluded_collections[]`	doublecloud.transfer.v1.endpoint.MongoCollection
`secondary_preferred_mode`	bool Read mode for mongo client

endpoint.MongoTarget

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.MongoConnection
`database`	string Database name
`cleanup_policy`	doublecloud.transfer.v1.endpoint.CleanupPolicy

endpoint.OnPremiseMongo

Field	Description
`hosts`	list
`hosts[]`	string
`port`	int64
`replica_set`	string
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode

endpoint.SrvMongo

Field	Description
`hostname`	string
`replica_set`	string
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode

endpoint.MysqlConnection

Field	Description
`connection`	one of: `on_premise` Connection options for on-premise MySQL
`on_premise`	doublecloud.transfer.v1.endpoint.OnPremiseMysql Connection options for on-premise MySQL

endpoint.MysqlObjectTransferSettings

Field	Description
`view`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Views CREATE VIEW ...
`routine`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Routines CREATE PROCEDURE ... ; CREATE FUNCTION ... ;
`trigger`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Triggers CREATE TRIGGER ...
`tables`	doublecloud.transfer.v1.endpoint.ObjectTransferStage

endpoint.MysqlSource

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.MysqlConnection Database connection settings
`database`	string Database name You can leave it empty, then it will be possible to transfer tables from several databases at the same time from this source.
`user`	string User for database access.
`password`	doublecloud.transfer.v1.endpoint.Secret Password for database access.
`timezone`	string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.
`object_transfer_settings`	doublecloud.transfer.v1.endpoint.MysqlObjectTransferSettings Schema migration Select database objects to be transferred during activation or deactivation.
`include_tables_regex`	list
`include_tables_regex[]`	string
`exclude_tables_regex`	list
`exclude_tables_regex[]`	string
`service_database`	string Database for service tables Default: data source database. Here created technical tables (__tm_keeper, __tm_gtid_keeper).

endpoint.MysqlTarget

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.MysqlConnection Database connection settings
`database`	string Database name Allowed to leave it empty, then the tables will be created in databases with the same names as on the source. If this field is empty, then you must fill below db schema for service table.
`user`	string User for database access.
`password`	doublecloud.transfer.v1.endpoint.Secret Password for database access.
`sql_mode`	string Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.
`skip_constraint_checks`	bool Disable constraints checks Recommend to disable for increase replication speed, but if schema contain cascading operations we don't recommend to disable. This option set FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0.
`timezone`	string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.
`cleanup_policy`	doublecloud.transfer.v1.endpoint.CleanupPolicy Cleanup policy Cleanup policy for activate, reactivate and reupload processes. Default is DISABLED.
`service_database`	string Database schema for service table Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper).
`security_groups`	list Security groups
`security_groups[]`	string

endpoint.OnPremiseMysql

Field	Description
`port`	int64 Database port
`hosts`	list
`hosts[]`	string
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode TLS settings for server connection. Disabled by default.

endpoint.ObjectStorageConnection

Field	Description
`aws_access_key_id`	string
`aws_secret_access_key`	string
`region`	string
`endpoint`	string
`use_ssl`	bool
`verify_ssl_cert`	bool

endpoint.ObjectStorageDataSchema

Field	Description
`schema`	one of: `infer` / `data_schema`
`infer`	google.protobuf.Empty
`data_schema`	doublecloud.transfer.v1.endpoint.DataSchema

endpoint.ObjectStorageEventSource

Field	Description
`source`	one of: `sqs` / `sns` / `pub_sub`
`sqs`	doublecloud.transfer.v1.endpoint.ObjectStorageEventSource.SQS
`sns`	doublecloud.transfer.v1.endpoint.ObjectStorageEventSource.SNS
`pub_sub`	doublecloud.transfer.v1.endpoint.ObjectStorageEventSource.PubSub

endpoint.ObjectStorageEventSource.PubSub

Field

Description

endpoint.ObjectStorageEventSource.SNS

Field

Description

endpoint.ObjectStorageEventSource.SQS

Field	Description
`queue_name`	string
`owner_id`	string
`aws_access_key_id`	string
`aws_secret_access_key`	string
`endpoint`	string
`region`	string
`use_ssl`	bool
`verify_ssl_cert`	bool

endpoint.ObjectStorageProvider

Field	Description
`bucket`	string
`aws_access_key_id`	string
`aws_secret_access_key`	string
`path_prefix`	string
`endpoint`	string
`use_ssl`	bool
`verify_ssl_cert`	bool
`region`	string

endpoint.ObjectStorageReaderFormat

Field	Description
`format`	one of: `csv` / `parquet` / `avro` / `jsonl` / `proto`
`csv`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Csv
`parquet`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Parquet
`avro`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Avro
`jsonl`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Jsonl
`proto`	doublecloud.transfer.v1.endpoint.ProtoParser from queue parsers

endpoint.ObjectStorageReaderFormat.Avro

Field

Description

endpoint.ObjectStorageReaderFormat.Csv

Field	Description
`delimiter`	string
`quote_char`	string
`escape_char`	string
`encoding`	string
`double_quote`	bool
`newlines_in_values`	bool
`block_size`	int64
`advanced_options`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Csv.AdvancedOptions
`additional_options`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Csv.AdditionalReaderOptions

endpoint.ObjectStorageReaderFormat.Csv.AdditionalReaderOptions

Field	Description
`null_values`	list
`null_values[]`	string
`true_values`	list
`true_values[]`	string
`false_values`	list
`false_values[]`	string
`decimal_point`	string
`strings_can_be_null`	bool
`quoted_strings_can_be_null`	bool
`include_columns`	list
`include_columns[]`	string
`include_missing_columns`	bool
`timestamp_parsers`	list
`timestamp_parsers[]`	string

endpoint.ObjectStorageReaderFormat.Csv.AdvancedOptions

Field	Description
`skip_rows`	int64
`skip_rows_after_names`	int64
`autogenerate_column_names`	bool
`column_names`	list
`column_names[]`	string

endpoint.ObjectStorageReaderFormat.Jsonl

Field	Description
`newlines_in_values`	bool
`unexpected_field_behavior`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat.Jsonl.UnexpectedFieldBehavior
`block_size`	int64

endpoint.ObjectStorageReaderFormat.Parquet

Field

Description

endpoint.ObjectStorageResultTable

Field	Description
`table_namespace`	string
`table_name`	string
`add_system_cols`	bool

endpoint.ObjectStorageSerializerConfig

Field	Description
`any_as_string`	bool

endpoint.ObjectStorageSource

Field	Description
`provider`	doublecloud.transfer.v1.endpoint.ObjectStorageProvider
`format`	doublecloud.transfer.v1.endpoint.ObjectStorageReaderFormat
`path_pattern`	string
`result_table`	doublecloud.transfer.v1.endpoint.ObjectStorageResultTable
`result_schema`	doublecloud.transfer.v1.endpoint.ObjectStorageDataSchema
`event_source`	doublecloud.transfer.v1.endpoint.ObjectStorageEventSource
`advanced_settings`	doublecloud.transfer.v1.endpoint.ObjectStorageSourceAdvancedSettings

endpoint.ObjectStorageSourceAdvancedSettings

Field	Description
`unparsed_mode`	doublecloud.transfer.v1.endpoint.ObjectStorageUnparsed

endpoint.ObjectStorageTarget

Field	Description
`bucket`	string
`output_format`	doublecloud.transfer.v1.endpoint.ObjectStorageSerializationFormat
`bucket_layout`	string
`buffer_size`	string
`buffer_interval`	string
`service_account_id`	string
`output_encoding`	doublecloud.transfer.v1.endpoint.ObjectStorageCodec
`connection`	doublecloud.transfer.v1.endpoint.ObjectStorageConnection
`bucket_layout_timezone`	string
`bucket_layout_column`	string
`serializer_config`	doublecloud.transfer.v1.endpoint.ObjectStorageSerializerConfig

endpoint.BasicAuthSR

Field	Description
`user`	string User
`password`	doublecloud.transfer.v1.endpoint.Secret Password for user

endpoint.BlankParser

Field

Description

endpoint.ColumnParams

Field	Description
`name`	string
`required`	bool

endpoint.ColumnParamsList

Field	Description
`col_params`	list
`col_params[]`	doublecloud.transfer.v1.endpoint.ColumnParams

endpoint.ConfluentSchemaRegistryAuth

Field	Description
`confluent_schema_registry_auth`	one of: `no_auth` / `basic`
`no_auth`	doublecloud.transfer.v1.endpoint.NoAuth
`basic`	doublecloud.transfer.v1.endpoint.BasicAuthSR

endpoint.ConfluentSchemaRegistryConnection

Field	Description
`schema_registry_url`	string Schema Registry api url
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode TLS settings for server connection. Disabled by default.
`auth`	doublecloud.transfer.v1.endpoint.ConfluentSchemaRegistryAuth Schema Registry authentication config

endpoint.ConfluentSchemaRegistryParser

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.ConfluentSchemaRegistryConnection

endpoint.DebeziumCDCParser

Field

Description

endpoint.GenericParserCommon

Field	Description
`data_schema`	doublecloud.transfer.v1.endpoint.DataSchema
`null_keys_allowed`	bool Allow null keys, if no - null keys will be putted to unparsed data
`add_rest_column`	bool Will add _rest column for all unknown fields

endpoint.Parser

Field	Description
`parser`	one of: `json_parser` / `debezium_cdc_parser` / `tskv_parser` / `confluent_schema_registry_parser` / `blank_parser` / `raw_table`
`json_parser`	doublecloud.transfer.v1.endpoint.GenericParserCommon
`debezium_cdc_parser`	doublecloud.transfer.v1.endpoint.DebeziumCDCParser
`tskv_parser`	doublecloud.transfer.v1.endpoint.GenericParserCommon
`confluent_schema_registry_parser`	doublecloud.transfer.v1.endpoint.ConfluentSchemaRegistryParser
`blank_parser`	doublecloud.transfer.v1.endpoint.BlankParser
`raw_table`	doublecloud.transfer.v1.endpoint.RawTable

endpoint.ProtoDataSchema

Field	Description
`schema`	one of: `col_params_list`
`col_params_list`	doublecloud.transfer.v1.endpoint.ColumnParamsList

endpoint.ProtoDesc

Field	Description
`desc`	one of: `desc_file`
`desc_file`	bytes

endpoint.ProtoParser

Field	Description
`proto_desc`	doublecloud.transfer.v1.endpoint.ProtoDesc
`msg_package_type`	doublecloud.transfer.v1.endpoint.ProtoMessagePackageType
`msg_name`	string If package type is repeated - name of message containing repeated target field; else - name of target message
`primary_keys`	list
`primary_keys[]`	string
`included_fields`	doublecloud.transfer.v1.endpoint.ProtoDataSchema
`null_keys_allowed`	bool Allow null keys, if no - null keys will be putted to unparsed data

endpoint.RawTable

Field	Description
`value_as_bytes`	bool
`keys_as_bytes`	bool
`add_timestamp`	bool
`add_headers`	bool
`add_key`	bool

endpoint.OnPremisePostgres

Field	Description
`port`	int64 Will be used if the cluster ID is not specified.
`hosts`	list
`hosts[]`	string
`tls_mode`	doublecloud.transfer.v1.endpoint.TLSMode TLS settings for server connection. Disabled by default.

endpoint.PostgresConnection

Field	Description
`connection`	one of: `on_premise` Connection options for on-premise PostgreSQL
`on_premise`	doublecloud.transfer.v1.endpoint.OnPremisePostgres Connection options for on-premise PostgreSQL

endpoint.PostgresObjectTransferSettings

Field	Description
`sequence`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Sequences CREATE SEQUENCE ...
`sequence_owned_by`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Owned sequences CREATE SEQUENCE ... OWNED BY ...
`table`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Tables CREATE TABLE ...
`primary_key`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Primary keys ALTER TABLE ... ADD PRIMARY KEY ...
`fk_constraint`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Foreign keys ALTER TABLE ... ADD FOREIGN KEY ...
`default_values`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Default values ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...
`constraint`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Constraints ALTER TABLE ... ADD CONSTRAINT ...
`index`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Indexes CREATE INDEX ...
`view`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Views CREATE VIEW ...
`function`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Functions CREATE FUNCTION ...
`trigger`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Triggers CREATE TRIGGER ...
`type`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Types CREATE TYPE ...
`rule`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Rules CREATE RULE ...
`collation`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Collations CREATE COLLATION ...
`policy`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Policies CREATE POLICY ...
`cast`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Casts CREATE CAST ...
`materialized_view`	doublecloud.transfer.v1.endpoint.ObjectTransferStage Materialized views CREATE MATERIALIZED VIEW ...
`sequence_set`	doublecloud.transfer.v1.endpoint.ObjectTransferStage

endpoint.PostgresSource

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.PostgresConnection Database connection settings
`database`	string Database name
`user`	string User for database access.
`password`	doublecloud.transfer.v1.endpoint.Secret Password for database access.
`include_tables`	list Included tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.
`include_tables[]`	string
`exclude_tables`	list Excluded tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.
`exclude_tables[]`	string
`slot_byte_lag_limit`	int64 Maximum lag of replication slot (in bytes); after exceeding this limit replication will be aborted.
`service_schema`	string Database schema for service tables (__consumer_keeper, __data_transfer_mole_finder). Default is public
`object_transfer_settings`	doublecloud.transfer.v1.endpoint.PostgresObjectTransferSettings Select database objects to be transferred during activation or deactivation.

endpoint.PostgresTarget

Field	Description
`connection`	doublecloud.transfer.v1.endpoint.PostgresConnection Database connection settings
`database`	string Database name
`user`	string User for database access.
`password`	doublecloud.transfer.v1.endpoint.Secret Password for database access.
`cleanup_policy`	doublecloud.transfer.v1.endpoint.CleanupPolicy Cleanup policy for activate, reactivate and reupload processes. Default is truncate.
`security_groups`	list Security groups
`security_groups[]`	string

endpoint.DebeziumSerializerParameter

Field	Description
`key`	string Name of the serializer parameter
`value`	string Value of the serializer parameter

endpoint.Serializer

Field	Description
`serializer`	one of: `serializer_auto` / `serializer_json` / `serializer_debezium` Select the serialization format automatically
`serializer_auto`	doublecloud.transfer.v1.endpoint.SerializerAuto Select the serialization format automatically
`serializer_json`	doublecloud.transfer.v1.endpoint.SerializerJSON Serialize data in json format
`serializer_debezium`	doublecloud.transfer.v1.endpoint.SerializerDebezium Serialize data in debezium format

endpoint.SerializerAuto

Field

Description

endpoint.SerializerDebezium

Field	Description
`serializer_parameters`	list Settings of sterilization parameters as key-value pairs
`serializer_parameters[]`	doublecloud.transfer.v1.endpoint.DebeziumSerializerParameter

endpoint.SerializerJSON

Field

Description

endpoint.airbyte.AmazonAdsSource

Field	Description
`client_id`	string
`client_secret`	string
`scope`	string
`refresh_token`	string
`start_date`	string
`region`	string
`profiles`	list
`profiles[]`	double

endpoint.airbyte.AWSCloudTrailSource

Field	Description
`aws_key_id`	string
`aws_secret_key`	string
`aws_region_name`	string
`start_date`	string

endpoint.airbyte.BigQuerySource

Field	Description
`project_id`	string
`dataset_id`	string
`credentials_json`	string

endpoint.airbyte.DynamodbSource

Field	Description
`region`	string
`aws_access_key_id`	string
`aws_secret_access_key`	string
`reserved_attribute_names`	list
`reserved_attribute_names[]`	string
`ignore_missing_read_permissions_tables`	bool

endpoint.airbyte.FacebookMarketingSource

Field	Description
`start_date`	string
`end_date`	string
`account_id`	string
`access_token`	string
`include_deleted`	bool
`fetch_thumbnail_images`	bool
`custom_insights`	list
`custom_insights[]`	doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.InsightConfig

endpoint.airbyte.FacebookMarketingSource.InsightConfig

Field	Description
`name`	string
`fields`	list
`fields[]`	doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.Field
`breakdowns`	list
`breakdowns[]`	doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.Breakdown
`action_breakdowns`	list
`action_breakdowns[]`	doublecloud.transfer.v1.endpoint.airbyte.FacebookMarketingSource.ActionBreakdown

endpoint.airbyte.GoogleAdsSource

Field	Description
`customer_id`	string
`start_date`	string
`end_date`	string
`custom_queries`	list
`custom_queries[]`	doublecloud.transfer.v1.endpoint.airbyte.GoogleAdsSource.CustomQuery
`login_customer_id`	string
`conversion_window_days`	double
`credentials`	doublecloud.transfer.v1.endpoint.airbyte.GoogleAdsSource.Credentials

endpoint.airbyte.GoogleAdsSource.Credentials

Field	Description
`developer_token`	string
`client_id`	string
`client_secret`	string
`access_token`	string
`refresh_token`	string

endpoint.airbyte.GoogleAdsSource.CustomQuery

Field	Description
`query`	string
`table_name`	string

endpoint.airbyte.HubspotSource

Field	Description
`start_date`	string
`credentials`	doublecloud.transfer.v1.endpoint.airbyte.HubspotSource.Credentials
`enable_experimental_streams`	bool

endpoint.airbyte.HubspotSource.Credentials

Field	Description
`auth_method`	one of: `private_app`
`private_app`	doublecloud.transfer.v1.endpoint.airbyte.HubspotSource.Credentials.PrivateApp

endpoint.airbyte.HubspotSource.Credentials.PrivateApp

Field	Description
`access_token`	string

endpoint.airbyte.InstagramSource

Field	Description
`start_date`	string
`access_token`	string

endpoint.airbyte.JiraSource

Field	Description
`api_token`	string
`domain`	string
`email`	string
`projects`	list
`projects[]`	string
`start_date`	string
`issues_stream_expand_with`	list
`issues_stream_expand_with[]`	doublecloud.transfer.v1.endpoint.airbyte.JiraSource.Expand
`enable_experimental_streams`	bool

endpoint.airbyte.LinkedinAdsSource

Field	Description
`start_date`	string
`account_ids`	list
`account_ids[]`	int64
`credentials`	doublecloud.transfer.v1.endpoint.airbyte.LinkedinAdsSource.Credentials

endpoint.airbyte.LinkedinAdsSource.Credentials

Field	Description
`credentials`	one of: `oauth` / `access_token`
`oauth`	doublecloud.transfer.v1.endpoint.airbyte.LinkedinAdsSource.Credentials.OAuth
`access_token`	string

endpoint.airbyte.LinkedinAdsSource.Credentials.OAuth

Field	Description
`client_id`	string
`client_secret`	string
`refresh_token`	string

endpoint.airbyte.MSSQLSource

Field	Description
`host`	string
`port`	int64
`database`	string
`username`	string
`password`	string
`replication_method`	doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.MSSQLReplicationMethod
`ssl_method`	doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLConfig

endpoint.airbyte.MSSQLSource.SSLConfig

Field	Description
`ssl_method`	one of: `unencrypted` / `encrypted_trust_server_certificate` / `encrypted_verify_certificate`
`unencrypted`	doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLUnencrypted
`encrypted_trust_server_certificate`	doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLEncryptedTrusted
`encrypted_verify_certificate`	doublecloud.transfer.v1.endpoint.airbyte.MSSQLSource.SSLEncryptedVerifyCert

endpoint.airbyte.MSSQLSource.SSLEncryptedTrusted

Field

Description

endpoint.airbyte.MSSQLSource.SSLEncryptedVerifyCert

Field	Description
`host_name_in_certificate`	string

endpoint.airbyte.MSSQLSource.SSLUnencrypted

Field

Description

endpoint.airbyte.RedshiftSource

Field	Description
`host`	string
`port`	int64
`database`	string
`schemas`	list
`schemas[]`	string
`username`	string
`password`	string

endpoint.airbyte.S3Source

Field	Description
`dataset`	string
`path_pattern`	string
`schema`	string
`format`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Format
`provider`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Provider

endpoint.airbyte.S3Source.Avro

Field

Description

endpoint.airbyte.S3Source.Csv

Field	Description
`delimiter`	string
`quote_char`	string
`escape_char`	string
`encoding`	string
`double_quote`	bool
`newlines_in_values`	bool
`block_size`	int64
`additional_reader_options`	string
`advanced_options`	string

endpoint.airbyte.S3Source.Format

Field	Description
`format`	one of: `csv` / `parquet` / `avro` / `jsonl`
`csv`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Csv
`parquet`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Parquet
`avro`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Avro
`jsonl`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Jsonl

endpoint.airbyte.S3Source.Jsonl

Field	Description
`newlines_in_values`	bool
`unexpected_field_behavior`	doublecloud.transfer.v1.endpoint.airbyte.S3Source.Jsonl.UnexpectedFieldBehavior
`block_size`	int64

endpoint.airbyte.S3Source.Parquet

Field	Description
`buffer_size`	int64
`columns`	list
`columns[]`	string
`batch_size`	int64

endpoint.airbyte.S3Source.Provider

Field	Description
`bucket`	string
`aws_access_key_id`	string
`aws_secret_access_key`	string
`path_prefix`	string
`endpoint`	string
`use_ssl`	bool
`verify_ssl_cert`	bool

endpoint.airbyte.SnowflakeSource

Field	Description
`host`	string
`role`	string
`warehouse`	string
`database`	string
`schema`	string
`jdbc_url_params`	string
`credentials`	doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource.Credentials

endpoint.airbyte.SnowflakeSource.Credentials

Field	Description
`credentials`	one of: `oauth` / `basic_auth`
`oauth`	doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource.Credentials.OAuth
`basic_auth`	doublecloud.transfer.v1.endpoint.airbyte.SnowflakeSource.Credentials.BasicAuth

endpoint.airbyte.SnowflakeSource.Credentials.BasicAuth

Field	Description
`username`	string
`password`	string

endpoint.airbyte.SnowflakeSource.Credentials.OAuth

Field	Description
`client_id`	string
`client_secret`	string
`access_token`	string
`refresh_token`	string

TransferService

Quotas and Limits