Shows
Life Is But A StreamLIVE from Current NOLA: Scaling Streaming in the AI EraFrom the heart of the expo hall at Current NOLA, this special episode drops you into the conversations, energy, and the breakthroughs as they happened.Host, Joseph Morais, and co-host, Adi Polak talk with data streaming leaders and community voices to unpack how teams are using Apache Kafka® and Apache Flink® to power low-latency, AI-ready applications—covering patterns from usage-based billing and hybrid operations to cost efficiency and streaming governance. You’ll also hear how shift-left processing and governance make AI-ready data possible at scale. Plus, we break down key launches, including Queues for Kafka, Conflue...
2025-11-201h 18
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovScaling AI in Engineering with Peter Bell | Ep. 7Listen: https://confluent.buzzsprout.com | Today, Adi Polak talks to her guest, Peter Bell (gather.dev), about his career in software engineering leadership, CTO community building, and AI-driven development. Peter’s first job: electronics lab technician at their school (alongside shifts at Tesco). His challenge/theme: working at scale with AI adoption and change management.Check out gather.dev: https://www.gather.dev/SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by...
2025-11-0327 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovHow Viktor Gamov Stays Curious as Tech Rapidly Evolves | Ep. 3Adi Polak interviews her co-host, Viktor Gamov, about his career’s evolution from distributed systems to streaming technology. Viktor’s first job: apple picking. His challenge/theme: staying curious and non-judgmental in the ever-changing landscape of tech.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes. 👍 If you enjoyed this, please leave us a rating...
2025-10-0630 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovBuilding Real-time Systems for Apple, Nike & more ft. Adi Polak | Ep. 1The Confluent Developer Podcast is here! For this first episode, Tim Berglund talks to his co-host, Adi Polak (Confluent), about her career in distributed data systems. Her first job: neighborhood dogwalker. Her challenge/theme: early Hadoop, working at Akamai on data optimization and real-time threat detection for huge global customers like Apple, Nike, Facebook and others, and the power of collaboration. SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subs...
2025-09-2232 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovWe're back! Welcome to the Confluent Developer Podcast.Weekly episodes launching Sept. 22! | Hi, I'm Tim Berglund. It's been about four years since I've been podcasting at Confluent, and "Streaming Audio" has been on hiatus for a little more than two, but I've got great news: we are back! We're back with a new name, a new format, and new hosts. Welcome to the Confluent Developer Podcast, where we talk to software developers of all stripes about some of the most interesting problems they've solved in their career. I'll be joined by my co-hosts, Adi Polak and Viktor Gamov. And hey, you kno...
2025-09-0201 min
In|die ŠtatlNa nádraží Starý Lískovec jsem chodíval v papučíchJaké podniky Maki a Honza řadí do hospod s přidanou hodnotou?
Co je zač divadlo Kapitola? Proč v létě vyrazit na Folkové prázdniny? Co znamená název tohoto pořadu? A proč Honza chodil na nádraží Starý Lískovec v papučích?
Odpovědi a další brněnské tipy najdeš v úvodním díle podcastu In|die Štatl.
In|die Štatl je rozhlasový pořad Radia R věnující se brněnské nezávislé kultuře, podnikům a komunitám.
Sleduj mě i na instagramu!
https...
2023-11-1048 min
GOTO - The Brightest Minds in TechScaling Machine Learning with Spark • Adi Polak & Holden KarauThis interview was recorded for the GOTO Book Club.gotopia.tech/bookclubRead the full transcription of the interview hereAdi Polak - VP of Developer Experience at Treeverse & Contributing to lakeFS OSSHolden Karau - Co-Author of "Kubeflow for Machine Learning" & many more books & Open Source Engineer at NetflixDESCRIPTIONLearn how to build end-to-end scalable machine learning solutions with Apache Spark. With this practical guide, author Adi Polak introduces data and ML practitioners to creative solutions that supersede today's traditional methods. You'll learn a more holistic...
2023-06-3040 min
Let's Talk AI#30 - Scaling Machine Learning Models, Spark, Agile, Software Engineering and KPIs with Adi Polak🎙️ Who is Adi Polak?
Adi Polak is Vice President of Developer Experience at Treeverse, a machine learning and data systems expert with a passion for building a better world through open collaboration. Adi brings her industry knowledge and research experience to educate and help teams design, architect, and build cost-effective data systems and machine learning pipelines with scalability, expertise, and business goals in mind. She is the author of "Machine Learning With Apache Spark," a frequent worldwide presenter, and a proud Databricks ambassador who emphasizes the importance of effective KPIs and collaboration for success. Adi was nominated as one o...
2023-06-2850 minGOTO - The Brightest Minds in TechUnlocking the Power of Real-Time Analytics • Tim Berglund & Adi PolakThis interview was recorded for GOTO Unscripted.gotopia.techRead the full transcription of this interview hereTim Berglund - VP DevRel at StarTree & Author of "Gradle Beyond the Basics"Adi Polak - VP of Developer Experience at Treeverse & Contributing to lakeFS OSSRESOURCESTimtimberglund.comtwitter.com/tlberglundlinkedin.com/in/tlberglundAditwitter.com/AdiPolakinstagram.com/polak.codelinkedin.com/in/polak-adiTools & companiespinot.apache.orgtwitter.com/startreedatalinkedin...
2023-06-0244 min
Partially Redacted: Data, AI, Security, and PrivacyData Access Control with lakeFS’s Adi PolakData access control is becoming increasingly important as more and more sensitive data is being stored and processed by businesses and organizations. In this episode, the VP of Developer Experience at lakeFS, Adi Polak, joins to help define data access control and give examples of sensitive data that requires access control.
Adi also talks about the concept of role-based access control (RBAC), which differs from traditional access control methods and provides several advantages. The steps involved in implementing RBAC are discussed, as well as best practices and challenges. Real-world examples of RBAC implementation and success stories are...
2023-05-0334 minConfluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovGit for Data: Managing Data like Code with lakeFSIs it possible to manage and test data like code? lakeFS is an open-source data version control tool that transforms object storage into Git-like repositories, offering teams a way to use the same workflows for code and data. In this episode, Kris sits down with guest Adi Polak, VP of DevX at Treeverse, to discuss how lakeFS can be used to facilitate better management and testing of data.At its core, lakeFS provides teams with better data management. A theoretical data engineer on a large team runs a script to delete some data, but a bug in...
2023-01-1930 minConfluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovIf Streaming Is the Answer, Why Are We Still Doing Batch?Is real-time data streaming the future, or will batch processing always be with us? Interest in streaming data architecture is booming, but just as many teams are still happily batching away. Batch processing is still simpler to implement than stream processing, and successfully moving from batch to streaming requires a significant change to a team’s habits and processes, as well as a meaningful upfront investment. Some are even running dbt in micro batches to simulate an effect similar to streaming, without having to make the full transition. Will streaming ever fully take over?In this episode, Kr...
2022-11-0943 min
Software Developers Journey#223 Adi Polak continuous learner in the big-data spaceAdi took us on a journey of following learning opportunities. From a machine learning research project in college to learning about distributed systems and big data. We explored how she approached joining Microsoft and creating a scalable network of colleagues while continuing to share her knowledge and learning in the open.Here are the links from the show:https://www.twitter.com/AdiPolakhttps://www.linkedin.com/in/polak-adi/https://linktr.ee/adipolakAdi's Book https://www.oreilly.com/library/view/machine-learning-with/9781098106812/DE Community https://dataengineers-group.slack.com/join/shared_invite/zt-u5dpkqwh-UdlIEDdrYQYrw3jVf2JKmA#/shared-invite...
2022-10-1140 min
NaukowoPochodzenie aureoli, gaworzenie noworodków i poszukiwania muzyki - #039Wraz z wiekiem zmienia się nasz sposób na odkrywanie nowej muzyki i szukanie nowych utworów, a dzięki danym z platform muzycznych możemy analizować jak to się dzieje. Na podstawie zdjęć USG pewnych małpek możemy za to spojrzeć na proces powstawania umiejętności wydawania dźwięków przez ludzi, świeżo po urodzeniu. Zdjęcia, często amatorskie, pozwoliły także na podsumowanie wiedzy o aureolach, nie tych anielskich, lecz atmosferycznych. Nowe badania przedstawiają tez związek między dieta złożoną z ryb a rakiem skóry, pokazują też wadę metody edycji genów CRISPR...
2022-07-2718 min
Monday Morning Data Chat#87 - Challenges of Working With Data Lakes w/ Adi Polak (Treeverse)Data lakes are very common and provide an easy way to store and serve many types of data. But data lakes are not always easy to work with. Adi Polak (Treeverse) joins the show to chat about the challenges of working with data lakes.
#dataengineering #datalakes #dataengineering
---------------------------------
TERNARY DATA
We are Matt and Joe, and we’re "recovering data scientists". Together, we run a data architecture company called Ternary Data. Ternary Data is not your typical data consultancy. Get no-nonsense, no BS data engineering strategy, coaching, and advice. Trusted by gre...
2022-07-121h 03GOTO - The Brightest Minds in TechKubeflow for Machine Learning • Holden Karau & Adi PolakThis interview was recorded for the GOTO Book Club.gotopia.tech/bookclubRead the full transcription of the interview hereHolden Karau - Co-Author of "Kubeflow for Machine Learning" & Open Source Engineer at NetflixAdi Polak - VP of Developer Experience at Treeverse & Contributing to lakeFS OSSDESCRIPTIONMachine Learning has been declared dead several times but that’s far from true. Join Adi Polak, vice president of developer experience at Treeverse, and Holden Karau, open source engineer at Netflix, in their conversation about Ku...
2022-05-2048 min
The Engineering Side of DataCareers and Community with Adi PolakAdi Polak and Bob Haffner host a Twitter Spaces conversation on Careers and Community in Data Engineering.
#data #dataengineering
Connect with Adi
Twitter - @AdiPolak
Connect with Bob
Twitter - @bobhaffner
LinkedIn - linkedin.com/in/bobhaffner
Join the Community - https://lakefs.io/community/
2022-03-1853 min
The Wirehive Podcast: Wirelive.What is Machine Learning?This Episode talks about all things Machine Learning. This episode is hosted by one of Wirehives own, Andy Readman with a special guest from Microsoft, Adi Polak.
Do you want to know more about machine learning? This is the episode for you.
Adi Polak - Cloud Advocate | Microsoft Cloud Advocates
Data Science (grab.com)
The Journey of Deploying Apache Airflow at Grab
Real-time Data Infrastructure at Uber (arxiv.org)
Meet Michelangelo: Uber's Machine Learning Platform
What is Apache Spark? | Microsoft Docs
What is A...
2021-06-1850 min
Cloud GossipData, Data & Data with Adi PolakAdi Polak is going to share her expertise and thoughts on Machine Learning, Big Data, and the future trends in the industry. She’s also gonna talk about how these complex things work, the challenges they present, and some of the buzzwords typical of her industry.Adi’s gonna share with us which tasks she performs on a daily basis, which personality traits are beneficial for her work, and the differences between the different job roles in her field.We’re also are gonna learn more about Adi’s involvement in the community, her approach to teach...
2020-12-0948 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov5 Years of Event Streaming and Counting ft. Gwen Shapira, Ben Stopford, and Michael NollWith the explosion of real-time data, Apache Kafka and event stream processing (ESP) have grown in proliferation, with event streaming technology becoming the de facto technology transforming businesses across numerous verticals. Gwen Shapira (Engineering Leader, Confluent), Ben Stopford (Senior Director, OCTO, Confluent), and Michael Noll (Principal Technologist, Confluent) meet up to talk all about their last five years at Confluent and the changes they’ve seen in event streaming. They discuss what they were doing with Apache Kafka® before they arrived at Confluent, challenges in event streaming challenges that have arisen, and their favorite use cases. They then talk thr...
2020-08-3148 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovChampioning Serverless Eventing at Google Cloud ft. Jay SmithJay Smith helps Google Cloud users modernize their applications with serverless eventing. This helps them focus on their code instead of managing infrastructure, as well as ultra-fast deployments and reduced server costs. On today’s show, he discusses the definition of serverless, serverless eventing, data-driven vs. event-driven architecture, sources and sinks, and hybrid cloud with on-prem components. Finally, Jay shares how he sees application architecture changing in the future and where Apache Kafka® fits in.EPISODE LINKSQuine ProgramsGet Started with QwiklabsKubernetes PodcastsJoin the Confluent Community SlackLearn more with Kafka tutorials, resources, and guides at C...
2020-08-2647 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDisaster Recovery with Multi-Region Clusters in Confluent Platform ft. Anna McDonald and Mitch HendersonMulti-Region Clusters improve high availability in Apache Kafka®, ensure cluster replication across multiple zones, and help with disaster recovery. Making sure users are successful in every area of their Kafka deployment, be it operations or application development for specific use cases, is what Anna McDonald (Team Lead Customer Success Technical Architect) and Mitch Henderson (Principal Customer Success Technical Architect) are passionate about here at Confluent.In this episode, they share common challenges that users often run into with Multi-Region Clusters, uses cases for them, and what to keep in mind when considering replication. Ann...
2020-08-1743 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDeveloper Advocacy (and Kafka Summit) in the Pandemic EraAll Confluent developer advocates...assemble! COVID-19 has changed the face of meetings and events, halting all in-person gatherings and forcing companies to adapt on the fly. In today's episode of Streaming Audio, the developer advocates come together to discuss how their jobs have changed during the worldwide pandemic. Less than a year ago, this group was constantly on the road or in a plane on their way to present something new about Apache Kafka and event streaming, so how has the current climate affected their work? The group talks about Zoom fatigue, online presenting, online conferences/meetups, a...
2020-08-1241 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovApache Kafka 2.6 - Overview of Latest Features, Updates, and KIPsApache Kafka® 2.6 is out! This release includes progress toward removing ZooKeeper dependency, adding client quota APIs to the admin client, and exposing disk read and write metrics, and support for Java 14. In addition, there are improvements to Kafka Connect, such as allowing source connectors to set topic-specific settings for new topics and expanding Connect worker internal topic settings. Kafka 2.6 also augments metrics for Kafka Streams and adds emit-on-change support for Kafka Streams, as well as other updates. EPISODE LINKSWatch the video version of this podcastRead about what's new in Apache Kafka 2.6Join the Con...
2020-08-0610 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovTesting ksqlDB Applications ft. Viktor GamovViktor Gamov (Developer Advocate, Confluent) returns to Streaming Audio to explain the magic of ksqlDB, ideal testing environments for ksqlDB, and the ksqlDB test runner. For those who are just starting to explore the interface, Viktor provides some tips and best practices for what to look out for too. He also talks about the future of ksqlDB, the future of integration testing, and his favorite new feature among recent upgrades.EPISODE LINKSStreaming Audio episodes on ksqlDBWatch #LiveStreams with Viktor Gamov I Don't Always Test My StreamsJoin the Confluent Community SlackLearn more with Kafka tu...
2020-08-0339 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovHow to Measure the Business Value of Confluent Cloud ft. Lyndon HedderlyAs developers, we are good at envisioning the future state of any given system we want to build, but are we as good at telling the business how those changes positively impact the bottom line? Lyndon Hedderly (Team Lead, Business Value Consulting, Confluent) describes his approach to business value, how to justify a new technology that you’re introducing to your company, and tips on adopting new technologies and processes effectively. As Lyndon walks through each part of the business value framework: (1) baseline, (2) target state, (3) quantified benefits, (4) unquantified benefits, and (5) proof points, you’ll learn about...
2020-07-2754 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovModernizing Inventory Management Technology ft. Sina Sojoodi and Rohit KelapureInventory management systems are crucial for reducing real-time inventory data drift, improving customer experience, and minimizing out-of-stock events. Apache Kafka®’s real-time data technology provides seamless inventory tracking at scale, saving billions of dollars in the supply chain, making modernized data architectures more important to retailers now more than ever. In this episode, we’ll discuss how Apache Kafka allows the implementation of stateful event streaming architectures on a cloud-native platform for application and architecture modernization. Sina Sojoodi (Global CTO, Data and Architecture, VMware) and Rohit Kelapure (Principal Advisor, VMware) will discuss data modeling, as well as...
2020-07-2041 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovFault Tolerance and High Availability in Kafka Streams and ksqlDB ft. Matthias J. SaxApache Kafka® Committer and PMC member Matthias J. Sax explains fault tolerance, high-availability stream processing, and how it’s done in Kafka Streams. He discusses the differences between changelogging vs. checkpointing and the complexities checkpointing introduces. From there, Matthias explains what hot standbys are and how they are used in Kafka Streams, why Kafka Streams doesn’t do watermarking, and finally, why Kafka Streams is a library and not infrastructure. EPISODE LINKSAsk Confluent #7: Kafka Consumers and Streams Failover Explained ft. Matthias SaxAsk Confluent #8: Guozhang Wang on Kafka Streams Standby TasksHow to Run Kafka Streams on Kubern...
2020-07-1554 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovBenchmarking Apache Kafka Latency at the 99th Percentile ft. Anna PovznerReal-time stock trades, GPS location, and website click tracking are just a few industries that heavily rely on Apache Kafka®'s real-time messaging and data delivery functions. As such, Kafka's latency is incredibly important.Anna Povzner (Software Engineer, Confluent) gives you the breakdown and everything you need to know when it comes to measuring latency. The five components of latency are produce time, publish time, commit time, catch-up time, and fetch time. When consumer pulling adds to latency, Anna shares some best practices to keep in mind for how to think about partitioning in conjunction wi...
2020-07-0846 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovOpen Source Workflow Automation with Apache Kafka ft. Bernd RueckerWhat started out as a consulting company, Camunda eventually turned into a developer-friendly, open source vendor that now focuses on workflow automation. Bernd Ruecker, a co-founder and the chief technologist at Camunda, talks through the company's journey, how he ended up in open source, and all things automation, including how it differs from business process management and the issue of diagrams. Bernd also dives into dead letter topics in Apache Kafka®, software interacting with software, orchestration tension, and best practices for approaching challenges that pop up along the way. This episode will take you through a t...
2020-06-2943 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovGrowing the Event Streaming Community During COVID-19 ft. Ale MurrayWe've all been affected by COVID-19 in one way or another, resulting in big changes in workplace functionality, productivity, and even our relationships within the Apache Kafka® and Confluent communities as meetings and events have needed to turn virtual. Ale Murray (Global Community Manager, Confluent) shares interesting trends, changes in community metrics, and what we’ve done to adapt as a response. Ale also explains what makes a comprehensive community program and the value of community meetups in light of the pandemic. Despite how much we miss in-person interactions, by digitizing events and focusing on the c...
2020-06-2440 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovFrom Monolith to Microservices with Sam NewmanAuthor Sam Newman catches up with Tim Berglund (Senior Director of Developer Advocacy, Confluent) in the virtual studio on what microservices are, how they work, the drawbacks of microservices, what splitting the monolith looks like, and patterns to look for. The pair talk through Sam's book “Monolith to Microservices” chapter by chapter, looking at key components of microservices in more detail. Sam also walks through database decomposition, integrating with new technology, and performing joins in event streaming architecture. Lastly, Sam shares what he’s excited for in the future, which includes “Monolith to Microservices Volume II.”EPISODE L...
2020-06-1740 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovExploring Event Streaming Use Cases with µKanren ft. Tim BaldridgeTim Baldridge (Senior Software Engineer, Cisco) joins us on Streaming Audio to talk about event streaming, stream processing use cases, and µKanren. First, Tim shares about his work at Cisco related to intaking viruses, the backend, and finding new ways to process data. Later, Tim talks about interesting bank and airline use cases, as well as his time at Walmart, taking a closer look at specific retail use cases and the product that Walmart used to process data streams. If you’re curious about what µKanren is, how it relates to relational programming, the complex math t...
2020-06-0851 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovIntroducing JSON and Protobuf Support ft. David Araujo and Tushar TholeConfluent Platform 5.5 introduces long-awaited JSON Schema and Protobuf support in Confluent Schema Registry and across other platform components. Support for Protobuf and JSON Schema in Schema Registry provides the same assurances of data compatibility and consistency we already had with Avro, while opening up Kafka to more businesses, applications, and use cases that are built upon those data serialization formats. Tushar Thole (Engineering Leader, Confluent) and David Araujo (Product Manager, Confluent) share about these new improvements to Confluent Schema Registry, the differences between Apache Avro™, Protobuf, and JSON Schemas, how to treat optional fields, some of t...
2020-06-0140 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovScaling Apache Kafka in Retail with Microservices ft. Matt Simpson from BodenApache Kafka® is a powerful toolset for microservice architectures. In this podcast, we’ll cover how Boden, an online retail company that specializes in high-end fashion linked to the royal family, used streaming microservices to modernize their business. Matt Simpson (Solutions Architect, Boden) shares a real life use case showing how Kafka has helped Boden digitize their business, transitioning from catalogs to online sales, tracking stock, and identifying buying patterns. Matt also shares about what he's learned through using Kafka as well as the challenges of being a product master. And lastly, what is Matt excited for for...
2020-05-2742 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovConnecting Snowflake and Apache Kafka ft. Isaac KunenIsaac Kunen (Senior Product Manager, Snowflake) and Tim Berglund (Senior Director of Developer Advocacy, Confluent) practice social distancing by meeting up in the virtual studio to discuss all things Apache Kafka® and Kafka Connect at Snowflake. Isaac shares what Snowflake is, what it accomplishes, and his experience with developing connectors. The pair discuss the Snowflake Kafka Connector and some of the unique challenges and adaptations it has had to undergo, as well as the interesting history behind the connector. In addition, Isaac talks about how they’re taking on event streaming at Snowflake by implementing the K...
2020-05-2031 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovAMA with Tim Berglund | Streaming Audio SpecialHappy 100th episode of Streaming Audio! Thank you to everyone who has listened, subscribed, left a review, and mostly, for sharing our passion for event streaming. We can't wait for the next 100! To celebrate, Ben Stopford (Senior Director of the Office of the CTO, Confluent) hosts an AMA (ask me anything) with Tim, covering 62 questions in total—from his career, his time at Confluent, Marvel vs. DC, and what he looks for in a new hire, to how to nail your next conference talk. We hope you enjoy this special 100th episode of Streaming Audio: a podcast abo...
2020-05-1847 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovKubernetes Meets Apache Kafka ft. Kelsey HightowerKelsey Hightower was already an advocate, just like all other developers, long before joining Google officially as a developer advocate and Kubernetes expert. Gaining trust in your product, process, and the way you develop code requires the ability to explain those things well. Kelsey reflects on the journey that brought him to where he is today and how Kubernetes has evolved over the years too, including what makes Kubernetes so successful. But Tim is not the only one with questions. Kelsey asks a few of his own: does Apache Kafka® want to be a database? Does Kafka wa...
2020-05-1342 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovApache Kafka Fundamentals: The Concept of Streams and Tables ft. Michael NollIf you’ve ever wondered what Apache Kafka® is, what it’s used for, or wanted to learn about Kafka architecture and all its components, buckle up! In today’s episode, Michael Noll (Principal Technologist, Confluent) and Tim Berglund (Senior Director of Developer Advocacy, Confluent) discuss a series of fundamental questions: What is Kafka? What is an event? How do we organize and store events? And what is Kafka Streams? Over the course of this episode, Michael covers an in-depth look into Kafka technology and core concepts: the process of reading from a topic, differences between tables and stre...
2020-05-0448 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovIoT Integration and Real-Time Data Correlation with Kafka Connect and Kafka Streams ft. Kai WaehnerThere are two primary industries within the Internet of Things (IoT): industrial IoT (IIoT) and consumer IoT (CIoT), both of which can benefit from the Apache Kafka® ecosystem, including Kafka Streams and Kafka Connect. Kai Waehner, who works in the advanced tech group at Confluent with customers, defining their needs, use cases, and architecture, shares example use cases where he’s seen IoT integration in action. He specifically focuses on Walmart and its real-time customer integration using the Walmart app. Kafka Streams helps fine-tune the Walmart app, optimizing the user experience, offering a seamless omni-channel experience, and contributing to bus...
2020-04-2940 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovConfluent Platform 5.5 | What's New in This Release + UpdatesConfluent Platform 5.5 is out, and Tim Berglund (Senior Director of Developer Advocacy, Confluent) is here to give you the latest updates! The first is improved schema management and Confluent Schema Registry support for Protobuf and JSON, making these components pluggable. The second is better support for languages other than Java within the sphere of librdkafka. And finally, this release includes an upgrade to ksqlDB, which expands its functionality, supports more data types, increases availability for pull queries, and adds a new aggregate function.EPISODE LINKSConfluent Platform 5.5 Release NotesIntroducing Confluent Platform 5.5Watch the video version of...
2020-04-2411 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovMaking Abstract Algebra Count in the World of Event Streaming ft. Sam RitchieDuring his time at Twitter, Sam Ritchie (Staff Research Engineer, Google) led the development of Summingbird, a project that helped Twitter ingest and process massive amounts of data. It relieved some key pain points, saving developers at Twitter from doing work twice, as was a natural consequence of the then-current Lambda Architecture. In this episode, Sam dives teaches us some abstract algebra and explains how it has informed his attempts to make stream processing programs easy to write in a more general way.EPISODE LINKSCheck out SummingbirdJoin the Confluent Community SlackLearn about Kafka at Confluent...
2020-04-2246 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovApache Kafka 2.5 – Overview of Latest Features, Updates, and KIPsApache Kafka® 2.5 is here, and we’ve got some Kafka Improvement Proposals (KIPs) to discuss! Tim Berglund (Senior Director of Developer Advocacy, Confluent) shares improvements and changes to over 10 KIPs all within the realm of Core Kafka, Kafka Connect, and Kafka Streams, including foundational improvements to exactly once semantics, the ability to track a connector’s active topics, and adding a new co-group operator to the Streams DSL.EPISODE LINKSCheck out the Apache Kafka 2.5 release notesRead about what’s new in Apache Kafka 2.5Watch the video version of this podcastJoin the Confluent Community SlackLearn about Kafka a...
2020-04-1610 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovStreaming Data Integration – Where Development Meets Deployment ft. James UrquhartApplications, development, deployment, and theory are all key pieces behind customer experience, event streaming, and improving systems and integration. James Urquhart (Global Field CTO, VMware) is writing a book combining Wardley Mapping and Promise Theory to evaluate the future of event streaming and how it will become a more economic choice for users. James argues that reducing the cost of integration does not deter people from buying but instead encourages creativity to find more uses for integration. He stresses the importance of user experience and how knowing what users are going through helps mend products and workflows, w...
2020-04-1555 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovHow to Run Kafka Streams on Kubernetes ft. Viktor GamovThere’s something about YAML and the word “Docker” that doesn’t quite sit well with Viktor Gamov (Developer Advocate, Confluent). But Kafka Streams on Kubernetes is a phrase that does.Kubernetes is an open source platform that allows teams to deploy, manage, and automate containerized services and workloads. Running Kafka Streams on Kubernetes simplifies operations and gets your environment allocated faster.Viktor describes what that process looks like and how Jib helps build, test, and deploy Kafka Streams applications on Kubernetes for an improved DevOps experience. He also shares about some exciting projects he’s currentl...
2020-04-0641 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovCloud Marketplace Considerations with Dan RosanovaAs the fundamental data abstractions used by developers have changed over time, event streams are now the present and the future. Coming from decades of experience in messaging, Dan Rosanova (Senior Group Product Manager for Confluent Cloud, Confluent) discusses the pros and cons of cloud event streaming services on Google Cloud Platform (GCP), Microsoft Azure, and Confluent Cloud. He also compares major stream processing and messaging services: Cloud Pub/Sub vs. Azure Event Hubs vs. Confluent Cloud, and outlines major differences among them. Also on the table in today’s episode are cloud lock-in, the anxieties around it...
2020-03-3033 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovExplore, Expand, and Extract with 3X Thinking ft. Kent BeckAs a programmer, Kent Beck chats about various topics of broad interest to developers, including some of his books: “Extreme Programming Explained: Embrace Change,” “Test-Driven Development: By Example,” and “Implementation Patterns.” He wrote “Implementation Patterns” to highlight the positive habits a developer should form in order to write accessible code. He also shares about what it’s like to experiment with new ideas and implement them, especially when others doubt what you're trying to achieve. This relates to the concept behind the explore-to-expand transition and a short piece he wrote titled "Idea to Impact." Finally, Tim and Kent talk through t...
2020-03-2554 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovAsk Confluent #17: The “What is Apache Kafka?” Episode ft. Tim BerglundAsk Confluent is back! From questions on Apache Kafka®, data integration, and log aggregation, to potential interview questions that Tim would ask if he were to interview himself, anything goes. If you're already a Kafka expert (or any type of expert), think about becoming a speaker. Gwen and Tim talk through how to submit a proposal and get accepted to conferences. As experienced conference goers, they explain that what makes a successful talk is making sure you present for the attendee instead of making it about yourself. In essence, what can your idea or code do to he...
2020-03-2425 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDomain-Driven Design and Apache Kafka with Paul RaynerDomain-driven design (DDD) is helpful for managing complex processes and rules—especially those between business experts and developers/users—and turning them into models. CEO of Virtual Genius Paul Rayner describes how the vast tooling in DDD enables developers to focus on the coding that really matters and makes systems more collaborative, taking into account three primary considerations: (1) how to get better at collaborating, (2) strategic design and understanding why design really matters, and (3) modeling codes. He also touches on bounded context, microservices, event storming, event sourcing, and the relationship between Apache Kafka® and DDD. EPISODE LINKS
2020-03-1850 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovMachine Learning with TensorFlow and Apache Kafka ft. Chris MattmannTensorFlow is an open source machine learning platform that can be used with Apache Kafka® for deep learning. Chris Mattmann, author of Machine Learning with TensorFlow, introduces us to TensorFlow as a Google technology that teaches computers how to think and make connections like humans do. For example, when there is a signifier that the mind processes, out comes a label to the object in front of you. TensorFlow is Google's version of wrangling various technologies to help group them together and work smoothly as large amounts of data flow through. Chris also breaks down neural networks, ho...
2020-03-1153 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDistributed Systems Engineering with Apache Kafka ft. Gwen ShapiraAs an engineering leader managing a team, Gwen Shapira talks through the steps she took to get to Confluent and how she got started working with Apache Kafka®. She shares about what it's like being on the Project Management Committee (PMC) for the Apache Software Foundation as well as some of the responsibilities involved, such as choosing Kafka Improvement Proposals (KIPs), monitoring releases, and making contributions to the community. For Gwen, part of finding Kafka was her willingness to take risks, learn all types of code bases, and leave companies for a new technology that showed promise an...
2020-03-0448 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovTowards Successful Apache Kafka Implementations ft. Jakub KorabWhether it's stream processing, real-time data analytics, to adding business value, Professional Services helps customers thrive within their chosen software or products and ultimately be successful as a digital enterprise. As a solutions architect and member of the Professional Services Team at Confluent, Jakub Korab discusses what Professional Services actually is and how it relates to customer success. It all centers around what customers want to do, and you’ll hear about trends, Apache Kafka® use cases, and real-life examples of Professional Services in action within various industries over the last year.EPISODE LINKSUnderstanding Message Bro...
2020-02-2655 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovKnative 101: Kubernetes and Serverless Explained with Jacques ChesterWhat is Knative and how does it simplify Kubernetes-related processes through seamless extension? Jacques Chester (Software Engineer, VMware) is publishing a book called “Knative in Action” that walks through the problems Knative is trying to solve. You don’t need to be an expert to fully understand Knative, so start getting hands on and see what you can do with it! You also don't need to be an expert on Kubernetes to read the book, but some experience with the tool can help you get it working with your software more quickly. This episode will help you understand the relati...
2020-02-1947 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovPaving a Data Highway with Kafka Connect ft. Liz BennettThe Stitch Fix team benefits from a centralized data integration platform at scale using Apache Kafka and Kafka Connect. Liz Bennett (Software Engineer, Confluent) got to play a key role building their real-time data streaming infrastructure. Liz explains how she implemented Apache Kafka® at Stitch Fix, her previous employer, where she successfully introduced Kafka first through a Kafka hackathon and then by pitching it to the management team. Her first piece of advice? Give it a cool name like The Data Highway. As part of the process, she prepared a detailed document proposing a Kafka roadmap, which ev...
2020-02-1246 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDistributed Systems Engineering with Apache Kafka ft. Jun RaoJun Rao (Co-founder, Confluent) explains what relational databases and distributed databases are, how they work, and major differences between the two. He also delves into important lessons he’s learned along the way through the transition from the relational world to the distributed world. To be successful at a place like Confluent, he outlines three fundamental traits that a distributed systems engineer must possess, emphasizing the importance of curiosity and knowledge, care in code development, and being open-minded and collaborative. You may even find that sometimes, the people with the best answers to your problems aren't even at...
2020-02-0554 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovHow to Write a Successful Conference Abstract | Streaming Audio SpecialLearn how to write an abstract for conference submissions and call for papers with tips from Tim Berglund, chair of the Kafka Summit Program Committee. Whether you're giving a talk for the very first time or you consider yourself to be an experienced speaker, these guidelines will help you craft a strong story that stands out from the others.EPISODE LINKSJoin #summit-office-hours on the Confluent Community SlackSign up to speak at a meetupWatch the video version of this podcastGet 30% off Kafka Summit London registration with the code KSL20AudioSEASON 2
2020-02-0407 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovStreaming Call of Duty at Activision with Apache Kafka ft. Yaroslav TkachenkoCall of Duty: Modern Warfare is the most played Call of Duty multiplayer of this console generation with over $1 billion in sales and almost 300 million multiplayer matches. Behind the scenes, Yaroslav Tkachenko (Software Engineer and Architect, Activision) gets to be on the team behind it all, architecting, designing, and implementing their next-generation event streaming platform, including a large-scale, near-real-time streaming data pipeline using Kafka Streams and Kafka Connect.Learn about how his team ingests huge amounts of data, what the backend of their massive distributed system looks like, and the automated services involved for collecting data from...
2020-01-2746 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovConfluent Platform 5.4 | What's New in This Release + UpdatesA quick summary of new features, updates, and improvements in Confluent Platform 5.4, including Role-Based Access Control (RBAC), Structured Audit Logs, Multi-Region Clusters, Confluent Control Center enhancements, Schema Validation, and the preview for Tiered Storage.This release also includes pull queries and embedded connectors in preview as part of KSQL.EPISODE LINKSConfluent Platform 5.4 Release Notes Introducing Confluent Platform 5.4Download Confluent Platform 5.4Watch the video version of this podcastJoin us in Confluent Community SlackGet 30% off Kafka Summit London registration with the code KSL20AudioSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor G...
2020-01-2214 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovMaking Apache Kafka Connectors for the Cloud ft. Magesh NandakumarFrom previously focusing on Confluent Schema Registry to now making connectors for Confluent Cloud, Magesh Nandakumar (Software Engineer, Confluent) discusses what connectors do, how they simplify data integrations, and how they enable sophisticated customer use cases. With connectors built for Confluent Cloud on Google Cloud Platform (GCP), Microsoft Azure, and Amazon Web Services (AWS), this helps users implement Apache Kafka® within their existing systems in an easy way. There’s a lot that Magesh is looking forward to when the world of connectors and the world of cloud collide.EPISODE LINKSWhy Kafka Connect? ft. Robin Mof...
2020-01-1325 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovLocation Data and Geofencing with Apache Kafka ft. Guido SchmutzOne way to put Apache Kafka into action is through geofencing and tracking the location data of objects, barges, and cars in real time. Guido Schmutz (Principal Consultant, Trivadis) shares about one such use case involving a German steel company and the development project he worked on for them, which he featured in a talk at Berlin Buzzwords. EPISODE LINKSLocation Analytics – Real-Time Geofencing Using Kafka (Video) Location Analytics – Real-Time Geofencing Using Kafka (Slides) Join the Confluent Community SlackGet 30% off Kafka Summit London registration with the code KSL20AudioSEASON 2 Hosted...
2020-01-0848 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovMulti-Cloud Monitoring and Observability with the Metrics API ft. Dustin CoteThe role of monitoring hosted services is evolving, but the ability to let go of the details to get what you are paying for with SaaS has always been there. Dustin Cote (Product Manager for Observability, Confluent Cloud) talks about Apache Kafka® made serverless and how beyond just the brokers, Confluent Cloud focuses on fitting into customer systems rather than building monitoring silos. When it comes to monitoring, logging, tracing, and alerting, Dustin defines what they all mean and how they operate in a database before diving into the requirements needed in order for a properly observable cl...
2019-12-3042 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovApache Kafka and Apache Druid – The Perfect Pair ft. Rachel PedreschiAs the head of global field engineering and community at Imply, Rachel Pedreschi is passionate about engaging both externally with customers and internally with departments all across the board, from sales to engineering. Rachel’s involvement in the open source community focuses primarily on Apache Druid, a real-time, high-performance datastore that provides fast, sub-second analytics and complements another powerful open source project as well: Apache Kafka®. Together, Kafka and Druid provide real-time event streaming and high-performance streaming analytics with powerful visualizations.EPISODE LINKSHow To Use Kafka and Druid to Tame Your Router DataETL and Event Str...
2019-12-2350 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovApache Kafka 2.4 – Overview of Latest Features, Updates, and KIPsApache Kafka 2.4 includes new Kafka Core developments and improvements to Kafka Streams and Kafka Connect, including MirrorMaker 2.0, RocksDB metrics, and more.EPISODE LINKSRead about what's new in Apache Kafka 2.4Check out the Apache Kafka 2.4 release notesWatch the video version of this podcastSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you listen to podcasts. ▶️ Subscribe on YouTube, and hit the 🔔 to catch new...
2019-12-1615 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovCloud-Native Patterns with Cornelia DavisDeveloping cloud-based applications requires unique patterns and practices that make them suitable for modern cloud platforms. Host Tim Berglund catches up with Cornelia Davis, author of Cloud-Native Patterns and VP of Technology at Pivotal, on what cloud-native patterns are, the example code she created, her latest book, and how she wrote the book for the customers she interacts with on a daily basis. EPISODE LINKSGet 40% off Cloud Native Patterns with the code podcon19Join the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Ti...
2019-12-1653 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovAsk Confluent #16: ksqlDB EditionVinoth Chandar has led various infrastructure projects at Uber and is one of the main drivers behind the ksqlDB project. In this episode hosted by Gwen Shapira (Engineering Manager, Cloud-Native Apache Kafka®), Vinoth and Gwen discuss what ksqlDB is, the kinds of applications that you can build with it, vulnerabilities, and various ksqlDB use cases. They also talk about what's currently the best version of Apache Kafka version for performance improvements that don’t cause breaking changes to existing Kafka configuration and functionality. EPISODE LINKSRead about ksqlDB on the blogLearn more about ksqlDBksqlDB Demo | The Even...
2019-12-1230 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovMachine Learning with Kafka Streams, Kafka Connect, and ksqlDB ft. Kai WaehnerIn this episode, Kai Waehner (Senior Systems Engineer, Confluent) defines machine learning in depth, describes the architecture of his dream machine learning pipeline, shares about its relevance to Apache Kafka®, Kafka Connect, ksqlDB, and the related ecosystem, and discusses the importance of security and fraud detection. He also covers Kafka use cases, including an example of how Kafka Streams and TensorFlow provide predictive analytics for connected cars.EPISODE LINKSHow to Build and Deploy Scalable Machine Learning in Production with Apache KafkaLearn about Apache KafkaLearn about Kafka ConnectLearn about ksqlDB, the successor to KSQLJoin the Confluent C...
2019-12-0438 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovReal-Time Payments with Clojure and Apache Kafka ft. Bobby CalderwoodStreamlining banking technology to help smaller banks and credit unions thrive among financial giants is top of mind for Bobby Calderwood (Founder, Evident Systems), who started out in programming, transitioned to banking, and recently launched Evident Real-Time Payments. Payments leverages Confluent Cloud to help banks of all sizes transform to real-time banking services from traditionally batch-oriented, bankers’ hours operational mode. This is achieved through Apache Kafka® and the Kafka Streams and Kafka Connect APIs with Clojure using functional programming paradigms like transducers. Bobby also shares about his efforts to help financial services companies build their next...
2019-11-2758 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovAnnouncing ksqlDB ft. Jay KrepsJay Kreps (original Co-creator of Apache Kafka and CEO, Confluent) introduces ksqlDB, an event streaming database. As the successor to KSQL, ksqlDB seeks to unify the multiple systems involved in stream processing into a single, easy-to-use solution for building event streaming applications.ksqlDB offers support for running connectors in an embedded mode, in addition to support for both push and pull queries. Push queries allow you to subscribe to changing query results as new events occur, while pull queries allow you to look up a particular value at a single point in time. To use a ride-sharing...
2019-11-2026 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovInstalling Apache Kafka with Ansible ft. Viktor Gamov and Justin Manchester“It’s one thing to get a distributed system up and running. It’s another thing to get a distributed system up and running well.” Ansible keeps your Apache Kafka® deployment, management, and installation consistent, and it enables you to implement best practices that make it easy to get started. Justin Manchester (Platform DevOps Engineer, Confluent) and Viktor Gamov (Developer Advocate, Confluent) discuss the problems that Ansible is trying to solve, enabling collaboration and optimizing all components for top performance.EPISODE LINKSLearn more about AnsibleFollow Viktor Gamov on TwitterFollow Justin Manchester on TwitterThe Easiest Way to Instal...
2019-11-1846 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovSecuring the Cloud with VPC Peering ft. Daniel LaMotteEverything is moving to the cloud, which makes it increasingly important to secure your cloud infrastructure and minimize the threat of potential attackers. With a virtual private cloud (VPC)—your own private network in the cloud that you can launch your own instances into—this can be done with VPC Peering, connecting VPCs together to create a path between them to keep your data safe and accessible to you alone. Although typically performed in a single cloud provider, it is possible to do in more than one—think of it as your cloud routerDaniel LaMotte (Site Reliab...
2019-11-1331 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovETL and Event Streaming Explained ft. Stewart BrysonMigrating from traditional ETL tools to an event streaming platform is a process that Stewart Bryson (CEO and founder, Red Pill Analytics), is no stranger to. In this episode, he dispels misconceptions around what “streaming ETL” means, and explains why event streaming and event-driven architectures compel us to rethink old approaches:Not all data is corporate data anymoreNot all data is relational data anymoreThe cost of storing data is now negligibleSupporting modern, distributed event streaming platforms, and the shift of focus from on-premises to the cloud introduces new use cases that focus primarily on building new syst...
2019-11-0649 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovThe Pro’s Guide to Fully Managed Apache Kafka Services ft. Ricardo FerreiraSeveral definitions of a fully managed Apache Kafka® service have floated around, but Ricardo Ferreira (Developer Advocate, Confluent) breaks down what it truly means and why every developer should care. Addressing a handful of questions around Apache Kafka®, Confluent Cloud, hosted solutions, and how they all work, Ricardo describes the benefits of using a fully managed service as a means of simplifying the lives of developers and letting them get back to building—which is why they started out as developers in the first place! EPISODE LINKSThe Rise of Managed Services for Apache KafkaExcerpt from The B...
2019-11-0456 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovKafka Screams: The Scariest JIRAs and How To Survive Them ft. Anna McDonaldIn today's spooktacular episode of Streaming Audio, Anna McDonald (Technical Account Manager, Confluent) discusses six of the scariest Apache Kafka® JIRAs. Starting with KAFKA-6431: Lock Contention in Purgatory, Anna breaks down what purgatory is and how it’s not something to fear or avoid. Next, she dives into KAFKA-8522: Tombstones Can Survive Forever, where she explains tombstones, compacted topics, null values, and log compaction. Not to mention there’s KAFKA-6880: Zombie Replicas Must Be Fenced, which sounds like the spookiest of them all. KAFKA-8233, which focuses on the new TestTopology mummy (wrapper) class, provides one option...
2019-10-3046 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovData Integration with Apache Kafka and AttunityFrom change data capture (CDC) to business development, connecting Apache Kafka® environments, and customer success stories, Graham Hainbach discusses the possibilities of data integration with Kafka and Attunity using Replicate, Compose, and Enterprise Manager. He also shares real-life examples of how Attunity best leverages Kafka in their systems.EPISODE LINKSApache Kafka Transaction Data Streaming for DummiesJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed M...
2019-10-2843 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDistributed Systems Engineering with Apache Kafka ft. Colin McCabeColin McCabe shares about what it’s like being a distributed systems engineer on the Core Kafka team at Confluent, where he has worked previously, and how that led to his interest in Apache Kafka®. As an active member of the Apache open source community, he describes that the community is a place that both welcomes newcomers and fosters different ideas that help make the product the best that it can be for everyone.Being a distributed systems engineer versus a full stack engineer comes with its own unique challenges. Colin offers some advice for those interested in...
2019-10-2345 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovApache Kafka on Kubernetes, Microsoft Azure, and ZooKeeper with Lena HallLena Hall joins Tim Berglund in the studio to talk about Apache Kafka®, the various ways to run Kafka on Microsoft Azure, Kafka on Kubernetes (K8s), and some exciting events that are happening in the Kafka world. Lena shares about serving double duty as both a senior software engineer and senior cloud developer advocate for Azure Engineering, including her unique roles and responsibilities, and how she balances engineering with advocacy. From writing tech articles to her experience with fuzzing and presence on YouTube, Lena is a strong community supporter and believes in the importance of staying ro...
2019-10-1646 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovImproving Fairness Through Connection Throttling in the Cloud with KIP-402 ft. Gwen ShapiraThe focus of KIP-402 is to improve fairness in how Apache Kafka® processes connections and how network threads pick up requests and new data. Gwen Shapira (Engineering Manager for Cloud-Native Kafka, Confluent) outlines the details of this KIP and her team’s efforts to make user-facing Kafka improvements. Halfway through the episode, Gwen shares how to send metadata and produce client messages. EPISODE LINKSKIP-402: Improve fairness in SocksetServer processorsJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov ...
2019-10-0948 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovData Modeling for Apache Kafka – Streams, Topics & More with Dani TraphagenHelping users be successful when it comes to using Apache Kafka® is a large part of Dani Traphagen’s role as a senior systems engineer at Confluent. Whether she’s advising companies on implementing parts of Kafka or rebuilding their systems entirely from the ground up, Dani is passionate about event-driven architecture and the way streaming data provides real-time insights on business activity. She explains the concept of a stream, topic, key, and stream-table duality, and how each of these pieces relate to one another. When it comes to data modeling, Dani covers importance business requirements, including the ne...
2019-10-0740 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovMySQL, Cassandra, BigQuery, and Streaming Analytics with Joy GaoJoy Gao chats with Tim Berglund about all things related to streaming ETL—how it works, its benefits, and the implementation and operational challenges involved. She describes the streaming ETL architecture at WePay from MySQL/Cassandra to BigQuery using Apache Kafka®, Kafka Connect, and Debezium. EPISODE LINKSCassandra Source Connector DocumentationStreaming Databases in Real Time with MySQL, Debezium, and KafkaStreaming Cassandra at WePayChange Data Capture with Debezium ft. Gunnar MorlingJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.SEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov P...
2019-10-0243 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovScaling Apache Kafka with Todd PalinoTodd Palino, a senior SRE at LinkedIn, talks about the start of Apache Kafka® at LinkedIn, what learning to use Kafka was like, how Kafka has changed, and what he and others in the community hope for in the future of Kafka. If you’re curious about life as an SRE, Todd shares the details on that too, and goes into how Kafka is used at LinkedIn, as well as several wins and challenges over the years with the product. EPISODE LINKSKafka: The Definitive Guide by Neha Narkhede, Gwen Shapira & Todd PalinoURP? Excuse You! The Thre...
2019-09-2546 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovUnderstand What’s Flying Above You with Kafka Streams ft. Neil BuesingNeil Buesing (Director of Real-Time Data, Object Partners) discusses what a day in his life looks like and how Kafka Streams helps analyze flight data.EPISODE LINKSUsing Location Data to Showcase Keys, Windows, and Joins in Kafka Streams DSL and KSQLKafka: The Definitive Guide by Neha Narkhede, Gwen Shapira & Todd PalinoRead the Confluent blogJoin the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Sub...
2019-09-2313 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovKIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason GustafsonTim Berglund sits down with Colin McCabe and Jason Gustafson to talk about KIP-500. The pair, who work on the Kafka Core Engineering Team, discuss the history of Kafka, the creation of KIP-500, and what it will do for the community as a whole. They break down ZooKeeper's role in Kafka, the implications of removing ZooKeeper dependency, replacing it with a self-managed metadata quorum, and how they've been combatting security, stability, and compatibility issues. With pending improvements towards scalability and inter-broker communication, and now that KIP-500 has been adopted within the community—there's a lot covered in this episode th...
2019-09-1843 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovShould You Run Apache Kafka on Kubernetes? ft. Balthazar RouberolWhen it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. Kubernetes provides all the building blocks that are needed, and a lot of thought is required to truly create an enterprise-grade Apache Kafka® platform that can be used in production. But before running Kafka on Kubernetes, there are some factors to consider. What are the maturing stages of Kubernetes adoption? How did Datadog experience these stages? Balthazar Rouberol shares what to think about before hopping on Kubernetes hype train.EPISODE LINKSKafka-Kit: Tools f...
2019-09-1629 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovJay Kreps on the Last 10 Years of Apache Kafka and Event StreamingAs Confluent turns five years old, special guest Jay Kreps (Co-founder and CEO, Confluent) brings us back to his early development days of coding Apache Kafka® over a Christmas holiday while working at LinkedIn. Kafka has become a breakthrough open source distributed streaming platform based on an abstraction of the distributed commit log, and his involvement in the project eventually led him to start Confluent with Jun Rao and Neha Narkhede. In this episode, Jay shares about all the highs and lows along the way, including some of his favorite customer success stories with companies like Lyft an...
2019-09-1248 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovConnecting to Apache Kafka with Neo4jWhat’s a graph? How does Cypher work? In today's episode of Streaming Audio, Tim Berglund sits down with Michael Hunger (Lead of Neo4j Labs) and David Allen (Partner Solution Architect, Neo4j) to discuss Neo4j basics and get the scoop on major features introduced in Neo4j 3.4 and 3.5. Among these are geospatial and temporal types, but there’s also more to come in 4.0: a multi-database feature, fine-grained security, and reactive drivers/Spring Data Neo4j RX. In addition to sharing a little bit about the history of the integration and features in relation to Apach...
2019-09-0954 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovAsk Confluent #15: Attack of the Zombie ControllerGwen Shapira (Core Kafka Software Engineer, Confluent) sits down to answer the questions you've had about event streaming, Apache Kafka®, Confluent, and everything in between. This includes creating tables in nested JSON topics, how to balance ordering, latency and reliability, building event-based systems, and how to navigate the tricky endOffsets API. She talks about the hardships of fencing Zombie requests, some of the talks given at previous Kafka Summits, and an important question from Ask Confluent #3. EPISODE LINKSKIP-91: Provide Intuitive User Timeouts in The ProducerKIP-79: ListOffsetRequest/ListOffsetResponse v1 and add timestamp search methods to the ne...
2019-09-0422 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovHelping Healthcare with Apache Kafka and KSQL ft. Ramesh SringeriIn today’s episode of Streaming Audio, Tim Berglund sits down with Senior Applications Developer of Mobile Solutions Ramesh Sringeri to discuss Apache Kafka®—specifically two Kafka use cases that Children’s Healthcare of Atlanta is working on.First, they discuss achieving near-real-time streams of data to support meaningful intracranial pressure prediction and managing intracranial pressure (ICP) in a timely manner to help the care team achieve better outcomes with traumatic brain injuries.Children’s Healthcare of Atlanta is in the process of building machine learning models for predicting ICP values 30 and 60 minutes in the future. T...
2019-08-2852 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovContributing to Open Source with the Kafka Connect MongoDB Sink ft. Hans-Peter GrahslSink and source connectors are important for getting data in and out of Apache Kafka®. Tim Berglund invites Hans-Peter Grahsl (Technical Trainer and Software Engineer, Netconomy Software & Consulting GmbH) to share about his involvement in the Apache Kafka project, spanning from several conference contributions all the way to his open source community sink connector for MongoDB, now part of the official MongoDB Kafka connector code base. Join us in this episode to learn what it’s like to be the only maintainer of a side project that’s been deployed into production by several companies!EPISOD...
2019-08-2150 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovTeaching Apache Kafka Online with Stéphane MaarekStreaming Audio welcomes Stéphane Maarek (CEO, Datacumulus) on the podcast to discuss how he got started hosting online Apache Kafka® tutorials and teaching on Udemy, the challenges he faces as an instructor, his approach to answering hard questions, and the projects he is currently working on.EPISODE LINKSKSQL Training for Hands-On LearningJoin the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Co...
2019-08-1942 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovConnecting Apache Cassandra to Apache Kafka with Jeff Carpenter from DataStaxWhenever you see an Apache Cassandra™ in the wild, you probably also see an Apache Kafka®️. In this episode, Tim Berglund (Senior Director of Developer Experience, Confluent) and Jeff Carpenter (Director of Developer Advocacy, DataStax) discuss the best way to get those systems talking using the DataStax Apache Kafka Connector and build a real-time data pipeline. EPISODE LINKSAbout the DataStax Apache Kafka ConnectorDataStax Academy: DataStax Apache Kafka Connector CourseJoin the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Galla...
2019-08-1247 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovTransparent GDPR Encryption with David JacotThe General Data Protection Regulation (GDPR) has challenged many enterprises to rethink how they deal with customer data. Viktor Gamov chats with David Jacot about a unique approach to inter-broker traffic encryption that he implemented for his customer’s sidecar pattern use case.EPISODE LINKSLearn about IstioLearn about EnvoyLearn about LinkerdHandling GDPR with Apache Kafka®: How to Comply Without Freaking Out? Join the Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coas...
2019-08-0816 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovConfluent Platform 5.3 | What's New in This ReleaseA quick summary of the most important features in Confluent Platform 5.3. We discuss improved Kubernetes and Ansible support, improvements to Confluent Control Center that give you better insight into the data in your cluster, and an important new set of security features—Role-Based Access Control—aimed at making complex deployments more secure.EPISODE LINKSRead the docsRead the blogWatch the video version of this podcast (featuring an actual stream)Download Confluent Platform 5.3Join us in Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by N...
2019-07-3113 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovHow to Convert Python Batch Jobs into Kafka Streams Applications with Rishi DhanarajZenreach is a company that makes tools to help retailers use digital marketing more effectively. If that sounds like a problem that only marketing people would be interested in, that’s because you don’t know what they do! There are all kinds of fascinating technology problems to solve by utilizing event streaming platforms to process data at volume. Rishi Dhanaraj, our guest today, worked at Zenreach as an intern, and took on a big pile of Python batch jobs, turning them into some really interesting Kafka Streams code. Listen in as he walks us through how he did it.
2019-07-2931 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovAsk Confluent #14: In Control of Kafka with Dan NorwoodIs Apache Kafka® actually a database? Can you install Confluent Control Center on Google Cloud Platform (GCP)? All this, plus some tips from Dan Norwood, the first user of Kafka Streams.EPISODE LINKSControl Center Docker imageControl Center Docker configurationComplete Streams exampleWatch the video version of this podcastJoin us in Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed Music by Coastal Kites Artwork by Phil Vo 🎧 Subscribe to Confluent Developer wherever you list...
2019-07-2223 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovKafka in Action with Dylan ScottAuthor Dylan Scott tells all about his upcoming Manning title Kafka in Action, which shares how Apache Kafka® can be used by beginners who are just starting out their own projects and dispels common Hadoop-related myths, as Kafka has grown to become a powerful event streaming platform beyond big data ecosystems alone. To get 40% off Manning products, use the following code: podcon19EPISODE LINKSJoin us in Confluent Community SlackSEASON 2 Hosted by Tim Berglund, Adi Polak and Viktor Gamov Produced and Edited by Noelle Gallagher, Peter F...
2019-07-1538 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovChange Data Capture with Debezium ft. Gunnar MorlingFriends don’t let friends do dual writes! Gunnar Morling (Software Engineer, Red Hat) joins us on the podcast to share a little bit about what Debezium is, how it works, and which databases it supports. In addition to covering the various use cases and benefits from change data capture (CDC) in the context of microservices—touching on the outbox pattern in particular, Gunnar walks us through the advantages of log-based CDC as implemented through Debezium over polling-based approaches, why you’d want to avoid dual writes to multiple resources, and engaging with members from the community to work...
2019-07-1049 min
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor GamovDistributed Systems Engineering with Apache Kafka ft. Jason GustafsonEver wonder what it’s like to be a distributed systems engineer at Confluent? Core Kafka Engineer Jason Gustafson dives into the challenges of working on distributed systems, particularly when it comes to a unique system like Apache Kafka®. He also discusses ways in which Confluent is working with the community to solve active problems and what it takes to be a distributed systems engineer.As always, Confluent is looking for engineers who are interested in distributed systems, and you don’t have to have 10 years of experience to do it!EPISODE LINKSKIP-392: Allow...
2019-07-0245 min