Radek Maciaszek presents his learnings from the migration of machine learning and big data processing pipelines to Apache Airflow.
He highlights how they use Airflow to power their company big data infrastructure where they analyze hundreds of terabytes of data. Examples will cover the building of the ETL pipeline and use of Airflow to manage the machine learning Spark pipeline workflow.
The talk covers the basic Airflow concepts and show real-life examples of how to define your own workflows in the Python code. It finishes with more advanced topics related to Apache Airflow, such as adding custom task operators, sensors and plugins as well as best practices and both the pros and cons of this tool.
Uber relies on Big Data and ML to make business critical decisions such as pricing, trip ETA, etc. Today, those workloads such as Hive and Spark are running on YARN. To save millions of dollars by efficient use of cluster resources, Uber is planning to use Kubernetes to co-locate BigData/ML and micro-service workloads.
This talk will covers the following:
- Learnings of running large-scale BigData/ML on Kubernetes with Peloton
- Colocation of mixed workloads
- Federation across zones
- Feature and API parity with YARN
Verizon IT we manages multiple multi-tenant Kubernetes clusters across on-prem and multiple clouds hosting hundreds of applications. Containers, Kubernetes, and cloud-native are central pillars: both for application modernization strategy, and for our north star architecture. This discussiion is about evolving the developer experience in this space, despite the security constraints, leveraging open source tooling such as Skaffold, Harbor, Kaniko, and Jib.
Uber started using docker containers at scale in 2015, and has gone through a few generations of cluster management and service discovery technologies. In early 2019, Uber started working on migration from Mesos to Kubernetes to support secure service mesh and machine learning workloads.
This talk highlights the following:
- Overview of Uber Compute Infra
- API server benchmark and tweaks
- Custom controller and scheduler logic
- CRI: resource, health check, logging, isolation
- SPIRE and service discovery setup at Uber
In this talk Anoop Koloth and Hanzhang Wang from eBay present how they managed to build a monitoring system and leveraged data generated from envoy clusters:
(1) Processing billions of hits served from different platforms from worldwide in real-time.
(2) Key Performance Indicators from Envoy ecosystem.
(3) Effective ML solution for proactive monitoring diversified eBay systems.
(4) Graph-based modeling and algorithms to deal with system complexity.
(5) Symbiosis and enhancement with existing SRE solution.
MakeMyTrip is India’s #1 online travel platform having more than 70% of the traffic from mobile apps embarked on a journey to revolutionize its customer experience by building a scalable, personalized, machine learning based platform which powers onboarding, in-funnel and post-funnel engagement flows, such as ranking, dynamic pricing, persuasions, cross-sell and propensity models.