New S3 connector in DoubleCloud Transfer
In response to a need to transfer large amounts of data within a reliable timeframe, we have developed a new connector for Object Storage from scratch. Based on our internal tests, this connector provides approximately 8x more throughput in comparison to the Airbyte connector, using the same compute resources.
This improvement was largely achieved by implementing multi-threaded operations for data handling in object storage. Out of the box, we support formats such as CSV, JSON, and Parquet, with Delta format support coming soon.
The new connector also features automatic schema detection and its propagation into ClickHouse, an event-driven approach based on events in AWS SQS, and many other benefits, all achievable with a no-code solution.