Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Airflow for critical financial datasets, tiers pipelines by reliability, and orchestrates LLM-enabled workflows with validation layers and agent-checking-agent patterns.
Key Takeaways:
- 00:00 Introduction.
- 01:47 What Wise does and Antonello's role in the central data and analytics engineering team, a hybrid platform-plus-analytics team that owns dbt infrastructure, BI, and Analytics MCPs as a service.
- 05:53 Three principles that guide Airflow pipeline design at Wise: a clean separation between orchestration and computation logic, computational awareness (offloading memory-intensive tasks to EMR or SageMaker), and standardized deployments.
- 07:21 Why Wise treats Airflow as a pure orchestration layer and pushes memory-intensive work to external workers.
- 08:45 Moving to the Python Virtual Environment Operator to standardize Airflow deployments across the org while giving analysts and data scientists per-job Python environments.
- 10:50 The tiering system for pipelines, how it distinguishes highly controlled, well-documented, well-observed pipelines from newer ones, and how requirements from downstream drive tier promotion.
- 17:18 Where LLM-enabled workflows differ from standard pipelines: validation layers for specific use cases, plus observability and evaluation platforms that track model performance across executions.
- 19:08 Using Airflow to orchestrate LLM generation of monthly variance commentary for analysts.
- 21:10 Handling non-idempotent LLM outputs with multi-layer validation against source-of-truth data, and using a second agent (CI/CD style) to validate the first agent's output.
- 23:04 How AI-enabled workflow orchestration differs from batch ETL, and why teams should start small before building fully agentic pipelines.
- 25:25 What Antonello would most like to see from Airflow next: native support for agentic workflows and better local development that mirrors production.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow
