How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Airflow project.
Key Takeaways:
- 00:00 Introduction.
- 00:48 What Firework does: a video commerce platform bringing short-form shoppable video and livestream experiences directly onto brand websites and apps.
- 02:13 Shawn's team owns the full data and AI platform stack at Firework, from ingestion through BI and applied AI.
- 02:59 Airflow has been the main orchestration layer for batch transformations since the early open source days, and now powers AI workflows like conversational insights, knowledge-base building, and automated eval pipelines.
- 06:45 The AI workflow stack: ingestion of user interaction and content data, transformation into personalized profiles and embeddings, separate eval pipelines dispatched from third-party systems, all on Snowflake with Cortex AI and coordinated by Airflow.
- 08:55 Why Airflow stuck: flexibility, reliability, and clear visibility into dependencies across SQL, Python, and AI tasks.
- 11:27 The deterministic pipeline problem. Airflow assumes predictable input and output. LLM workflows break that assumption, which forces evaluation layers, traceability, regression testing, and feedback loops into the pipeline itself.
- 13:07 Evaluation cost can exceed compute cost. Validating a single output can mean running four or five parallel eval jobs across prompts and configurations, so caching, batching, and selective evaluation become essential.
- 15:58 Building safeguards: alerting and aborting jobs when cost exceeds thresholds, so the platform stays operationally stable while other improvements catch up.
- 18:30 Wishlist: stronger first-class support for AI workflow patterns (evaluation tracking, prompt experimentation, model observability, event-driven AI orchestration) and continued UI/UX improvement in Airflow 3.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #ApacheAirflow
