Our systems generate fantastic amounts of information, but do we have a complete understanding of how we collect, analyze, manage, store, and retrieve possibly petabytes a day? Gal Shpantzer has been doing InfoSec for over 20 years and has managed some huge data engineering projects, and offers a lot of actionable insights in this CISO Tradecraft episode.
Gal's LinkedIn Page - https://www.linkedin.com/in/riskmanagement/
Gal's Twitter Page - https://twitter.com/Shpantzer
Full Transcript - https://docs.google.com/document/d/14RXnsVttvKlRi6VL94BTrItCjOAjgGem/
Chapters
- 00:00 Introduction
- 02:00 How do you Architect Big Data Data Infrastructure
- 03:33 Are you taking a look at Ransomware?
- 06:11 Web Scale Technologies are used mostly in Marketing & Fraud Detection
- 08:11 Data Engineering - The Mindset Shift
- 10:51 The Iron Triangle of Data Engineering
- 13:55 Can I Outsource My Logging Pipeline to a Vendor
- 15:37 Kafka & Flink - Data Engineering in the Pipeline
- 18:12 Streaming Analytics & Kafka
- 22:08 How to Enable Data Science Analytics with Streaming Analytics
- 26:33 Streaming Analytics
- 30:25 Data Engineering - Is there a Security Log
- 32:30 Streaming Analytics is a Weird Thing
- 35:50 How to Get a Handle on a Big Data Pipeline
- 39:11 Data Engineering Hacks for Big Data Analytics
