Uber relies on Big Data and ML to make business critical decisions such as pricing, trip ETA, etc. Today, those workloads such as Hive and Spark are running on YARN. To save millions of dollars by efficient use of cluster resources, Uber is planning to use Kubernetes to co-locate BigData/ML and micro-service workloads.
This talk will covers the following:
- Learnings of running large-scale BigData/ML on Kubernetes with Peloton
- Colocation of mixed workloads
- Federation across zones
- Feature and API parity with YARN
This talk is about how Lyft has solved the complexity of automating cluster upgrades and how that is incorporated into the design for - k8srotator - a Kubernetes custom controller.
This talk is about how Cruise has been using Kubernetes. It highlights the motivations, story, and results of migrating to multitenant Kubernetes, along with some hard-earned Pro Tips from the trenches. You’ll also learn about the open source tooling they built around Spinnaker, Vault, Google Cloud, and Istio in order to integrate with our multitenant Kubernetes.
Uber started using docker containers at scale in 2015, and has gone through a few generations of cluster management and service discovery technologies. In early 2019, Uber started working on migration from Mesos to Kubernetes to support secure service mesh and machine learning workloads.
This talk highlights the following:
- Overview of Uber Compute Infra
- API server benchmark and tweaks
- Custom controller and scheduler logic
- CRI: resource, health check, logging, isolation
- SPIRE and service discovery setup at Uber
As Lyft migrated its applications to Kubernetes, assumptions baked into the networking layer were tested. This talk discusses how Lyft used Envoy’s xDS protocol to design their own flexible service mesh and handle new challenges from a multi-cluster architecture such as:
- Routing across multiple Kubernetes clusters
- Handling Deployments
- Rapid scale-in and scale-out
- Service Discovery
- Active/Passive Health Checking
- Readiness in the service mesh
This talk will also go over changes that were made in the Envoy codebase to make this work.
This talk highlights the the evolution of Redis support in Envoy. Initially Envoy redis proxy only supported sharding to clusters of independent Redis nodes. Recent developments have enabled support for the open source Redis Cluster protocol as well as some unique features such as multicluster routing, flexible load balancing options, and traffic shadowing.
As the usage of Redis expanded different usage patterns emerged, requiring different availability, durability and consistency trade-offs. Henry Yang from Lyft and Mitch Sulaski from Workday discuss how the Envoy redis proxy was extended to support these new requirements in large scale environment(10+ Millions rps) at Lyft and Workday.