DoubleCloud’s 10th product update

Hey everyone, Victor here. Here’s the news I’ve got for this month:

DoubleCloud Terraform providerDoubleCloud Terraform provider

The most requested feature has finally arrived. From now on, you can set up ClickHouse and Apache Kafka clusters, create DoubleCloud Transfer services, manage VPC and networks, or build dashboards and charts on the DoubleCloud platform using the descriptive approach of Terraform. By leveraging Terraform’s infrastructure-as-code capabilities, users can automate, version, and collaborate on infrastructure changes, ensuring consistent setups and reducing manual errors.

Build a full end-to-end analytical pipeline in one unified location and integrate it with your CI/CD processes.The official DoubleCloud provider is available here. If you wish to participate in development or simply provide feedback, you can easily do so in our GitHub repository.

New organizational viewNew organizational view

I’m excited to announce that we have finally launched a new component of the console called Organization. Here both administrators and users can manage their projects, establish org-wide permissions, set up SSO integration, monitor usage, and control billing for their entire organization.

The main motivation behind this was to separate administrative tasks from tasks related to platform services. Now, in the Project UI, you’ll mainly encounter functions related to managing services or resources, while at the organizational level, tasks related to account administration, permissions, managing usage and handling invoices, etc. We anticipate increasing cross-project functionality and decided it would be best to segregate these within the UI.

To access this new section, simply click on the list of projects in the console and select the option 'Manage organization.' Here, you will find a fresh UI and newly added functionalities.

New role and permission modelNew role and permission model

In response to our customers' requests for finer-grained permissions, we have introduced a new role model at DoubleCloud to replace the previous, limited schema of viewer, admin, and owner.

You can now grant permissions at either the organizational or project level, with the ability to inherit permissions. This offers greater granularity for each category of services, such as Clusters, Transfers, Visualization, and so on.

We recognize that our platform is used by a variety of people, from engineers who only wish to see and manage clusters, to business personnel who only need access to specific dashboards and don’t need to interact with other services or functionalities. For a description of the new resource model, click here, and for information about the new permissions, click here.

dbt transformations in Transfer servicedbt transformations in Transfer service

To address the “T” in the ELT abbreviation, we’ve diligently enhanced our esteemed Transfer service with an array of transformations. This now includes the capability to run dbt models after each data snapshot.

Previously, users could execute straightforward, pre-defined transformations, such as renaming columns or altering data types. Moving forward, you’ll also be able to execute SQL transformation scripts on the fly or run dbt models post-data snapshot. For a deeper understanding of these functionalities, please refer to our documentation.

ClickHouse ver 23.8 LTS is available on DoubleCloudClickHouse ver 23.8 LTS is available on DoubleCloud

We’ve launched the 23.8 LTS (long-term support) version on DoubleCloud and designated it as the default version during creation. We strongly urge users to upgrade to this version in the near future. Among the enhancements in this version, one that stands out to me is the improved handling of Parquet files, which have become the predominant format for storing columnar data in files.

Private networks in AWS BYOAPrivate networks in AWS BYOA

We added the option of BYOA mode to work in private networks that limit access to clusters and resources outside of the internet, to make it possible to build more secure deployments using our Bring-Your-Own-Account option.

Quality of life improvementsQuality of life improvements

Here’s a list of some minor but potentially pivotal improvements that simplify workflows and day to day work:

• Over the past month, we’ve dedicated substantial time to fine-tuning the disk configurations for ClickHouse clusters. We’ve adjusted the ratio between IOPS and throughput to better cater to analytical workloads, based on statistics. Additionally, we’ve optimized configurations for the keeper process, among other things. Particularly for workloads handling vast amounts of data, users should encounter fewer “too many parts” issues and might have up to 2-3x increased throughput without any changes from user’s side.

• We added the ability to set offset on linear indicator charts in the Visualization service.

• We’ve also refreshed and adjusted some UI components. Our calendar, for example, now displays names of weekdays, among other subtle UI enhancements.