Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform Engineer at [Headout](headout.com), walks through how her team uses Airflow as the nervous system of their stack: orchestrating dbt with a write-audit-publish pattern, running ML training and inference, and wiring up alerting that points to the exact commit that broke a DAG.
Key Takeaways:
- 00:00 Introduction.
- 01:05 What a data platform engineer does at Headout, and the hub-and-spokes model where analysts and scientists write their own dbt models.
- 04:11 The specific data challenges of a travel platform: fast-changing inventory, real-time fraud analytics, and performance marketing attribution.
- 05:40 Where Airflow sits in the stack, from ingestion to transformation to serving.
- 07:01 The write-audit-publish dbt pattern and why slightly stale data beats wrong data.
- 09:55 Why Headout uses a custom Python operator instead of the dbt provider or Cosmos, reading the dbt manifest to build task groups per model.
- 13:45 ML use cases on Airflow: Feast feature store, model training, inference, and data/feature drift tracking.
- 17:00 Custom Slack failure hooks that stitch together Airflow logs, GitHub commit URLs, and teammate Slack IDs.
- 20:04 A zombie task incident that filled the metadata DB, caused locking issues, and drove the move to Grafana-based monitoring.
- 22:26 Using AI to generate Airflow code, encoding internal patterns as a skill file, and running an AI reviewer bot on every PR.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow
