Filter (clear filters)





Overview for cloud

Using FoundationDB and the FDB Record Layer to Build CloudKit - Scott Gray, Apple

This talk is about the features of FoundationDB and the Record Layer that help build CloudKit, Apple’s cloud storage system for structured data. Scott Gray, a software engineer at Apple focuses on three key areas where FoundationDB and the Record Layer have unlocked large benefits for CloudKit. First, FoundationDB’s arbitrary multi-key ACID transactions have allowed implement advanced secondary indexing at scale, including our recently-developed transactional full-text search system. 


The Benefits of Running Spark on your own Docker

In this talk Shir Bromberg a Big Data team leader at Yotpo,discusses their open-source dockers for running Spark on Nomad servers. She highlights the following; 
* The issues they  had running spark on managed clusters and the solutions developed.
* How to build a spark docker.
* What to achieve by using Spark on Nomad.


Optimizing Spark-based data pipelines - are you up for it?

Nielsen Marketing Cloud needs to ingest billions of events per day into their big data stores for their real time analytics. Etti Gur  the Senior Big Data developer and Itai Yaffe Tech Lead, Big Data group discuss how they significantly optimized Spark-based in-flight analytics daily pipeline, reducing its total execution time from over 20 hours down to 2 hours, resulting in a huge cost reduction.

Apache Flink 101 - the rise of stream processing and beyond

This talk is about some Flink use cases and basic requirements of stream processing, and how Flink fills the gaps and stands out with some of its unique core building blocks, like pipelined execution, native event time support, state support, and fault tolerance.

There is also a highlight of how Flink is going beyond stream processing into areas like unified data processing, enterprise intergration, AI/machine learning  especially online ML, and serverless computation, and how Flink fits with its distinct value.


The Need for Speed – Data Streaming in the Cloud with Kafka®

Running Kafka on Kubernetes is becoming more and more popular. Frank Pientka, Principal Software Architect, Materna Information & Communications SE introduces a setup, used components and recommendations from an own project with Kafka on Kubernetes.He shares the lessons learned from this still evolving field. 


Apache Beam meetup 7 at Datatonic: Beam at Lyft + datalake using Beam + schemas

See how Lyft and Datatonic are using Apache Flink, Apache beam and python in stream processing, machine learning and analytics.