The Airflow Database – Airflow requires a database to store all the metadata related to the execution history of each task and DAG, DAG schedules, and more. MySqlOperator, PostgresOperator, and other database specific operators for executing SQL commands.Here are some of the most common operators. There a number of operators that ship with Airflow, as well as countless custom ones created by the Airflow community. Operators – Operators are what actually execute scripts, commands, and other operations when a Task is run. Each node in the graph is a Task, and each Task implements an Airflow Operator. Tasks – As noted in the previous section, a DAG is made up of Tasks. In addition to DAGs, there are a number of components and concepts in Airflow that you’ll need to know before you get started. Once Task D is complete, the graph is completed as well.Ī simple DAG with four tasks Airflow Components and Concepts Once both are completed then Task D can start. In this example, Task A must complete before Tasks B and C can start. Without this constraint, a graph could run in an endless loop. Graphs must also be acyclic, meaning that a task cannot point back to a previously completed task. This way, the graph ensures that tasks do not run before all their dependent tasks are completed. This is required to guarantee a path of execution. The graph is directed, meaning it starts with a task or multiple tasks and ends with a specific task or tasks. Not only a graph, but one with some constraints. Directed Acyclic Graphs (DAGs)Īirflow represents the flow and dependencies of tasks in a data pipeline as a graph. This post describes the fundamentals of Airflow, how to install it, and how to build and run your first DAG. Created by Maxime Beauchemin at Airbnb in 2014, it joined the Apache Software Foundation’s Incubator program in March 2016 and has taken off since. Apache Airflow is one of the most popular such platforms, and is open-source as well. Workflow management platforms are what data engineers use to schedule and coordinate the steps in a data pipeline – an activity sometimes referred to as “data orchestration”.

Apache airflow book