Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Your Streaming Data Platform
This discussion is about how Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud in 2014. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann describes the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob also talks about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.