Uber relies on Big Data and ML to make business critical decisions such as pricing, trip ETA, etc. Today, those workloads such as Hive and Spark are running on YARN. To save millions of dollars by efficient use of cluster resources, Uber is planning to use Kubernetes to co-locate BigData/ML and micro-service workloads.
This talk will covers the following:
- Learnings of running large-scale BigData/ML on Kubernetes with Peloton
- Colocation of mixed workloads
- Federation across zones
- Feature and API parity with YARN
Uber started using docker containers at scale in 2015, and has gone through a few generations of cluster management and service discovery technologies. In early 2019, Uber started working on migration from Mesos to Kubernetes to support secure service mesh and machine learning workloads.
This talk highlights the following:
- Overview of Uber Compute Infra
- API server benchmark and tweaks
- Custom controller and scheduler logic
- CRI: resource, health check, logging, isolation
- SPIRE and service discovery setup at Uber
Mingmin Chen from Uber discusses how they leverage Apache Kafka at Uber for delivering high perfomance at scale.
Omkar Joshi a senoir software engineer at Uber discusses a new Spark ingestion system known as Marmaray. This new system has been designed to ingest billions of Kafka messages at intervals of 30 minutes.
Learn how Uber Data Org is using Kafka in disaster discovery.
Learn about how Uber is using Big Data for analytics.