Modify an Airflow® cluster

To change the cluster settings, take the following steps:

1. Go to the Clusters page.

2. Select the Airflow® cluster you want to modify and click Edit at the top right.

3. Modify the cluster settings according to your needs.

   Worker node resources

   • In Preset, select the amount of CPU, RAM, and SSD storage suitable for your workload.

   • In Min worker nodes and Max worker nodes, specify the numbers of workers for autoscaling. The cluster will automatically adjust the number of workers depending on the load.

   • In Concurrency, specify the limit for how many running task instances a DAG can have. Any task instance on top of this number is queued.

   DAG code repository

   • In Repository URL, DAG path, and Branch, enter the details of your Git repository with DAGs.

   • If the repository is private, enter the credentials in Username and Password/token. For GitHub, use a personal access token .

   • To make sure the connection details are correct, click Check connection.

   Advanced settings

   • In Maintenance settings, select whether you want DoubleCloud to perform maintenance at an arbitrary time or by schedule. If you selected By schedule, select the day and time (UTC).

   • In requirements.txt, enter the Python dependencies available from PyPI that you want to install in the cluster. Use the package_name == version format. Learn more

   • In Environment variables, add variables you want to use in the cluster.

   • In Service account, select a DoubleCloud service account you want to use to access other DoubleCloud services from the Airflow® cluster. Learn more

4. To apply the changes and restart hosts, click Submit.