Autoscaling Apache Kafka® cluster storage

Autoscaling lets your Apache Kafka® cluster storage to automatically increase when its utilization is nearing capacity. With this, you don't need to constantly monitor available disk space and add storage manually.

Autoscaling also helps you control costs because your cluster only uses the resources it needs. When creating a cluster, you don't need to allocate storage for future potential database growth from the very start. Instead, storage capacity incrementally grows together with the database size, so you only pay for what you use.

How autoscaling works

If autoscaling is enabled, DoubleCloud constantly monitors how much storage is available in the cluster. When disk utilization on any host reaches 95%, it adds storage on all hosts towards the next capacity available in a given resource preset. DoubleCloud sends notification emails before storage increases.

The interval between storage increases is at least six hours. All storage increases are permanent, and the cost depends on the actual storage size.

Configure autoscaling

You can configure autoscaling when creating a Apache Kafka® cluster or enable it later.

resource "doublecloud_kafka_cluster" "example-kafka" {
  ...
  resources {
    kafka {
      resource_preset_id = "s2-c2-m4"
      disk_size          = 34359738368     # 32GB in bytes
      max_disk_size      = 103079215104    # 96GB in bytes
      ...
    }
  }
  ...
}