Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python.
Key Takeaways
- 00:00 Introduction.
- 00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers.
- 02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users.
- 03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries.
- 04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management.
- 06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers.
- 08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually solved.
- 09:18 Why sensor polling created scheduler load and worker overload, and how event-driven Assets fixed both.
- 10:15 The lineage view in the Airflow UI that came as a bonus after the Assets migration.
- 10:49 The vision for multi-tenant Airflow inside Skimlinks: replacing cron, Rundeck, and team-local Airflow instances with a shared platform.
- 14:31 Building a custom DAG factory with YAML configuration for onboarding new publishers.
- 17:33 Breaking a single Python class into single-responsibility components for the DataPipe project.
- 19:07 Adding a Pydantic layer so misconfigured YAML fails at DAG parse time instead of run time.
- 20:31 Using AI assistance to guide refactoring decisions and generate tests across the new class structure.
- 22:34 What Julian wants from Airflow next: asset watchers paired with data contracts.
Resources Mentioned
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow
