Apache Airflow

Apache Airflow is a workflow orchestration platform where pipelines are written as Python code rather than configured through a UI. You define a Directed Acyclic Graph (DAG) that describes which tasks to run, in what order, and with what dependencies. Airflow handles the scheduling, execution, retry logic, and monitoring. The web interface provides a visual representation of each DAG and a history of every run with task-level logs.

The Python-as-code model is both the platform’s greatest strength and the reason it is not for everyone. Any workflow logic expressible in Python — conditionals, dynamic task generation, branching based on upstream results — is straightforward to implement. Hundreds of built-in operators cover connections to AWS, Google Cloud, Azure, Spark, Postgres, HTTP endpoints, and most common data infrastructure. If a pre-built operator does not exist, a PythonOperator runs any arbitrary Python function.

The typical use case is data engineering: extract data from source systems, transform it, and load it to a destination on a schedule. ML teams use Airflow to chain data preparation, feature engineering, training, and evaluation steps with dependency tracking. Anywhere a sequence of tasks has explicit ordering requirements and needs reliable scheduling with retry logic and alerting, Airflow fits.

The operational cost is real. A production-grade Airflow deployment needs a metadata database (PostgreSQL is standard), a message broker (Redis or RabbitMQ) for the distributed executor, a scheduler process, a webserver, and one or more worker nodes. The Docker Compose setup from the official docs handles this, but it runs six or more containers and needs meaningful hardware.

For teams running straightforward cron jobs, Cronicle or n8n is a lower-overhead alternative. Airflow earns its complexity when workflows have non-trivial dependencies, require distributed execution, or need the full observability its UI provides.

Apache Airflow: Pros & Cons