Kestra

Kestra is a workflow orchestration platform where pipelines are written in YAML rather than Python. You describe what to run, when to run it, and what to do when it fails — all in a declarative file that sits in version control alongside your other infrastructure configuration. Kestra reads that YAML, builds the execution graph, and handles scheduling, retries, logging, and alerting.

The YAML approach is the core differentiator from Airflow. In Airflow, a pipeline is a Python file defining a DAG with task objects and operators. In Kestra, the same pipeline is a structured document that any team member can read regardless of their Python fluency. Changing a schedule, adding a notification step, or adjusting a retry policy is a YAML edit rather than a Python code change.

Language agnosticism is a genuine practical advantage. A Kestra workflow can chain a Python script, a shell command, a dbt run, a SQL query against a database, and a call to a REST API — all in the same flow — because each task specifies an execution environment independently. There is no shared Python environment to manage, and adding a task in a different language does not require restructuring the pipeline.

Event-driven triggers go beyond simple cron scheduling. Kestra can start a workflow when a file lands in an S3 bucket, when a message arrives on a Kafka topic, when a webhook fires, or when another workflow completes. This makes it practical for data pipelines that respond to real-world events rather than only running on a fixed schedule.

The plugin ecosystem covers over 600 integrations across cloud providers, databases, messaging systems, and data tools. For ETL pipelines, data engineering workflows, and multi-step automation that spans several systems, Kestra provides the orchestration layer without requiring Airflow’s Python infrastructure.

Kestra: Pros & Cons