Autoscaling ClickHouse® cluster storage

Autoscaling lets your ClickHouse® cluster storage to automatically increase when its utilization is nearing capacity. With this, you don't need to constantly monitor available disk space and add storage manually.

Autoscaling also helps you control costs because your cluster only uses the resources it needs. When creating a cluster, you don't need to allocate storage for future potential database growth from the very start. Instead, storage capacity incrementally grows together with the database size, so you only pay for what you use.

How autoscaling works

If autoscaling is enabled, DoubleCloud constantly monitors how much storage is available in the cluster. When disk utilization on any host reaches 95%, it adds storage on all hosts towards the next capacity available in a given resource preset. DoubleCloud sends notification emails before storage increases.

Note

For example, you created a Managed ClickHouse® cluster with 32GB of storage and enabled autoscaling with a maximum limit of 96GB. When 95% of the 32GB of memory is used on a host, the storage automatically scales up to 48GB. When 95% of the 48GB is used, it scales up to 64GB. This repeats until the storage is 96GB.

The interval between storage increases is at least six hours. All storage increases are permanent, and the cost depends on the actual storage size.

Configure autoscaling

You can configure autoscaling when creating a ClickHouse® cluster or enable it later. Autoscaling is available for SSD storage, dedicated Keeper host storage, or both.

Configure autoscaling in the Autoscaling section when creating a ClickHouse® cluster or later on the cluster settings page.

configure autoscaling

By default, the maximum limit is set to the next capacity available in a given resource preset.

To enable autoscaling, specify the max_disk_size values for clickhouse, dedicated_keeper, or both in the ClickHouse cluster resource configuration .

Example resource configuration:

resource "doublecloud_clickhouse_cluster" "example-clickhouse" {
  ...
  resources {
    clickhouse {
      resource_preset_id = "s2-c2-m4"
      disk_size          = 34359738368     # 32 GB in bytes
      max_disk_size      = 103079215104    # 96 GB in bytes
      ...
    }
    dedicated_keeper {
      resource_preset_id = "s2-c2-m4"
      disk_size          = 34359738368     # 32 GB in bytes
      max_disk_size      = 51539607552     # 48 GB in bytes
      ...
    }
  }
  ...
}

max_disk_size must be greater than the base storage size. If it's equal to the base storage size or not specified, autoscaling is disabled.

To enable the cluster SSD storage autoscaling, specify the max_disk_size parameter in ClusterResources.Clickhouse when creating or updating the cluster.

max_disk_size must be greater than the base storage size. If it's equal to the base storage size or not specified, autoscaling is disabled.