This talk is about the features of FoundationDB and the Record Layer that help build CloudKit, Apple’s cloud storage system for structured data. Scott Gray, a software engineer at Apple focuses on three key areas where FoundationDB and the Record Layer have unlocked large benefits for CloudKit. First, FoundationDB’s arbitrary multi-key ACID transactions have allowed implement advanced secondary indexing at scale, including our recently-developed transactional full-text search system.
In this talk Shir Bromberg a Big Data team leader at Yotpo,discusses their open-source dockers for running Spark on Nomad servers. She highlights the following;
* The issues they had running spark on managed clusters and the solutions developed.
* How to build a spark docker.
* What to achieve by using Spark on Nomad.
Nielsen Marketing Cloud needs to ingest billions of events per day into their big data stores for their real time analytics. Etti Gur the Senior Big Data developer and Itai Yaffe Tech Lead, Big Data group discuss how they significantly optimized Spark-based in-flight analytics daily pipeline, reducing its total execution time from over 20 hours down to 2 hours, resulting in a huge cost reduction.
This talk is about some Flink use cases and basic requirements of stream processing, and how Flink fills the gaps and stands out with some of its unique core building blocks, like pipelined execution, native event time support, state support, and fault tolerance.
There is also a highlight of how Flink is going beyond stream processing into areas like unified data processing, enterprise intergration, AI/machine learning especially online ML, and serverless computation, and how Flink fits with its distinct value.
Running Kafka on Kubernetes is becoming more and more popular. Frank Pientka, Principal Software Architect, Materna Information & Communications SE introduces a setup, used components and recommendations from an own project with Kafka on Kubernetes.He shares the lessons learned from this still evolving field.
See how Lyft and Datatonic are using Apache Flink, Apache beam and python in stream processing, machine learning and analytics.