Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.
Key Takeaways:
- 00:00 Introduction.
- 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
- 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
- 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
- 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
- 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
- 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
- 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
- 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
- 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
- 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
- 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow
