JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline that validates DAGs before they reach production.
Key Takeaways:
- 00:00 Introduction.
- 00:48 What JLR makes: luxury vehicles under the Range Rover, Jaguar, Defender, and Discovery brands.
- 01:42 Najeeb's team in the Data and AI Office, supporting manufacturing, supply chain, finance, and commerce analytics.
- 03:25 Airflow as the central nervous system of the JLR data stack — the orchestrator that connects every source and downstream system.
- 05:01 How JLR uses Tableau, and the two modes for getting data in: live connection and scheduled extract refresh.
- 06:24 Why scheduled Tableau refreshes go stale: they aren't aware of when the data pipeline actually finished.
- 08:09 First attempt at solving it: Python scripts calling the Tableau REST API directly.
- 08:47 Why the script approach didn't scale across teams — code duplication and version drift.
- 10:00 Trying the community Airflow Tableau provider and hitting the PAT authentication roadblock.
- 12:21 Building a custom provider on top of the community one to keep PAT auth.
- 13:30 Treating CI/CD as a deployment gate for Airflow DAGs at JLR's scale.
- 15:23 What the CI/CD pipeline actually catches: top-level code making external calls, import errors, and Airflow 3 compatibility.
- 17:47 How the gate blocks broken DAGs from reaching production.
- 18:30 What Najeeb wants from Airflow next: native integration testing, better OpenTelemetry support, and built-in lineage.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow
