Modify an Airflow® cluster
Once your Managed Apache Airflow® is created, you can change its settings. Changes can only be applied when the cluster is running.
To change the cluster settings, take the following steps:
-
Go to the Clusters page.
-
Select the Airflow® cluster you want to modify and click Edit at the top right.
-
Modify the cluster settings according to your needs.
Worker node resources-
In Preset, select the amount of CPU, RAM, and SSD storage suitable for your workload.
-
In Min worker nodes and Max worker nodes, specify the numbers of workers for autoscaling. The cluster will automatically adjust the number of workers depending on the load.
-
In Concurrency, specify the limit for how many running task instances a DAG can have. Any task instance on top of this number is queued.
DAG code repository-
In Repository URL, DAG path, and Branch, enter the details of your Git repository with DAGs.
-
If the repository is private, enter the credentials in Username and Password/token. For GitHub, use a personal access token .
-
To make sure the connection details are correct, click Check connection.
Advanced settings-
In Maintenance settings, select whether you want DoubleCloud to perform maintenance at an arbitrary time or by schedule. If you selected By schedule, select the day and time (UTC).
-
In requirements.txt, enter the Python dependencies available from PyPI that you want to install in the cluster. Use the
package_name == versionformat. Learn more -
In Environment variables, add variables you want to use in the cluster.
-
In Service account, select a DoubleCloud service account you want to use to access other DoubleCloud services from the Airflow® cluster. Learn more
-
-
To apply the changes and restart hosts, click Submit.
You can modify an Airflow® cluster by editing the Terraform configuration that you used to create these resources. For a full list of available parameters, refer to the DoubleCloud Airflow® cluster resource schema .
Warning
Some resource parameters can't be updated. For example, you can't change the cloud provider of an existing cluster or decrease the storage volume. If you need to change such settings, recreate the resources.
resource "doublecloud_airflow_cluster" "example-airflow" {
...
resources {
airflow {
max_worker_count = 4 // New maximum number of worker nodes
min_worker_count = 1
...
}
}
...
}