Create an Apache Airflow® cluster

Step 1. Configure resources

1. Go to the Clusters page and click Create cluster at the top right.

2. Select Airflow.

3. Choose a provider and a region. You can create Managed Apache Airflow® clusters on AWS in any of the available regions. By default, DoubleCloud preselects the region nearest to you.

4. In Environment configuration, select the configuration that best fits your needs. It defines the ratio of web servers, schedulers, and workers.

5. Under Worker node resources, select a preset with the amount of CPU, RAM, and SSD storage suitable for your workload.

6. In Min worker nodes and Max worker nodes, specify the lower and upper limits of workers for autoscaling. The cluster will automatically adjust the number of workers depending on the load.

7. In Concurrency, specify the limit for how many running task instances a DAG can have. Any task instance on top of this number is queued.

8. Under Basic settings in Name enter a cluster name, such as airflow-dev.

9. In Version, select the Airflow® version for the cluster. Unless you need a specific version, select the latest one.

10. If your DAGs are stored in a Git repository, configure a connection under DAG Code Repository:

    How to configure a connection

    1. In Repository URL, DAG path, and Branch, enter the details of your Git repository with DAGs.

    2. If the repository is private, enter the credentials in Username and Password/token. For GitHub, use a personal access token .

    3. To make sure the connection details are correct, click Check connection.

Step 2. Configure advanced settings

1. In Maintenance settings, select whether you want DoubleCloud to perform maintenance at an arbitrary time or by schedule. If you selected By schedule, select the day and time (UTC).

2. Under Networking → VPC, select the network where you want to create the cluster.

   If you don’t need to place the cluster in a specific network, leave the preselected default option.

3. (Optional) In Allowlist, configure IP addresses the Airflow® cluster can be accessed from. To do that, click Edit and add or remove IP addresses. You can use both single addresses and CIDR blocks. When you're done, click Save in the dialog.

4. In the Summary block on the right, review the resources to be created and their price.

5. Click Submit.

   Creating a cluster usually takes five to seven minutes depending on the cloud provider and region. When the cluster is ready, its status changes from Creating to Alive.