Data transformations are only part of the modern data pipeline. Knowing when they should run, what they depend on, and what happens when they fail is just as important as the transformations themselves. As data solutions grow in complexity, orchestration becomes a fundamental part of the architecture rather than simply an operational afterthought.
Execution and orchestration go hand in hand, but they don’t have to be solved by the same platform. In fact, modern data architectures benefit from treating them as separate concerns: one layer is responsible for executing data workloads efficiently, while another coordinates when, how, and under which conditions those workloads should run.
This is precisely where Snowflake and Apache Airflow complement one another. Snowflake provides the scalable compute layer for data processing, while Airflow orchestrates the workflow around it, ensuring each task executes in the right order and under the correct conditions, acting as the bridge between the different components that form the underlying architecture.
Putting the Pattern to Work
One of Hakkoda’s current Snowflake migrations began as a straightforward modernization effort, moving from an on-premises solution to a Snowflake-native one. But with the sheer volume of ingestion processes, transformations, and validation steps involved, the orchestration requirements evolved right alongside them. Every workload became part of a broader picture where execution order, dependencies, and failure handling were essential, not only to replicate the existing process, but, more importantly, to ensure reliable data delivery.
Apache Airflow naturally addresses these requirements. Acting as the orchestration layer, it coordinates Snowflake workloads through directed workflows, ensuring each stage executes only when its prerequisites have been met, while providing monitoring, retry mechanisms, and centralized operational visibility through Airflow’s UI.
With both tools addressing distinct responsibilities, the question becomes: what now?
To Infinity and Beyond: Simplifying Airflow with Astro
Introducing Airflow into the mix naturally raises another consideration: how should the orchestration platform itself be managed? Apache Airflow provides the engine, but its deployment, configuration, and ongoing maintenance can introduce additional operational complexity of their own.
This is where Astro came in. Using Astronomer’s managed Airflow platform gave the team a streamlined approach to testing and deploying Airflow DAGs and environments, significantly reducing the operational overhead of maintaining the orchestration layer. Short-lived deployments, spun up solely to run unit and integration tests against both the DAGs and Snowflake itself, made development considerably simpler.
The team also had the opportunity to leverage Astro AI, Astro’s integrated LLM, to speed up the migration of similar pipelines. By building a working template first, pipelines that shared similar logic could be migrated quickly and consistently. Focusing on strong templates and letting Astro AI replicate that pattern across dozens of similar pipelines proved essential, freeing developers to focus their time on testing rather than repetitive setup work.
Connecting the Pieces
Getting Airflow connected to Snowflake was the first step. RSA key pair authentication, with the public key registered in Snowflake, was the natural choice for a secure connection between environments. Once authentication was in place, the integration itself was surprisingly straightforward. Airflow tasks could execute SQL statements, call stored procedures, and more.
This all happens through DAGs: Python scripts that define a workflow from start to finish. A DAG describes a set of tasks, when they run (on a schedule or triggered by an event), and the order in which they should execute. Each task represents a discrete unit of work, whether that’s a SQL script, a stored procedure, a Snowpark workload, or a simple variable operation.
Airflow’s Snowflake providers do much of the heavy lifting here. These providers offer operators and hooks that make interacting with the platform feel like a natural extension of the workflow itself. A simplified example of a DAG leveraging this is shown below:
The schedule, the tasks, and the task dependencies (load_raw_data runs first, followed by build_sales_mart) are all immediately clear just from reading the DAG. Although deliberately simplified, the same principles apply to much larger workflows.
As pipelines grow, DAGs can branch into parallel execution paths, introduce conditional logic, wait on external systems, or coordinate dozens of Snowflake operations across multiple stages. Despite that added complexity, the workflow remains easy to reason about, because dependencies are explicitly defined rather than implied.
Bringing It All Together
From a delivery perspective, the impact can’t be overstated. By standardizing how pipelines were orchestrated, deployed, and monitored, the team spent less time on operational concerns and more time focused on the migration itself. Reusable patterns emerged naturally, new pipelines became easier to build, and operational visibility improved across the board.
As the project evolved, the orchestration layer scaled right alongside it, not only without introducing unnecessary complexity, but by actively enabling the team to deliver against the Snowflake compute layer more efficiently. The result is a solid foundation for both this migration and the workloads still to come.
Ultimately, integrating Snowflake with Airflow comes down to letting each do what it does best. Snowflake stays focused on executing data workloads efficiently, while Airflow provides the orchestration that turns individual processes into reliable, observable, and scalable data pipelines. More than just a scheduler, Airflow is what tied the solution together.
If your team is navigating a similar Snowflake migration, or looking to bring order to a growing set of pipelines, contact us to talk through how Hakkoda and IBM can help design an orchestration layer built to scale with you.
