Ebay has built a GraphDB cloud service called NuGraph, which is based on JanusGraph. FoundationDB is chosen as the JanusGraph’s storage plugin, because of its high-performance and distributed transaction support. Jun Li and Hieu Nguyen from eBay present the GraphDB architecture, and focus on how they deploy and manage FoundationDB in Kubernetes, how they improve JansGraph query performance in a cross-data center environment, how they bulk load the graph into FoundationDB with its transactional support, and how they secure the 3-tier cloud service with limited security support from FoundationDB.
In this talk Anoop Koloth and Hanzhang Wang from eBay present how they managed to build a monitoring system and leveraged data generated from envoy clusters:
(1) Processing billions of hits served from different platforms from worldwide in real-time.
(2) Key Performance Indicators from Envoy ecosystem.
(3) Effective ML solution for proactive monitoring diversified eBay systems.
(4) Graph-based modeling and algorithms to deal with system complexity.
(5) Symbiosis and enhancement with existing SRE solution.
Learn how eBay is migrating its 30 PB MPP database to Apache Spark. Currently, 15000+ ETL jobs have been running on a 1000+ nodes Spark cluster each day, processing PB scale data and these numbers are increasing quickly. Optimization is critical during the migration, because the cluster resource is usually very stressful, well-optimized system can hold more jobs in the limited resource. Yuming Wang and Yucai Yu both from eBay talk about the top performance challenges eBay encountered and how these problems were addressed.
eBay has been using Analytical DBMS (ADBMS) data warehouse solution for over a decade, there are millions of batch queries running every day against 6000+ key DW tables, which contains over 22PB data (compressed) and still keeps booming every year. Based upon that, data services and products enables eBay business decisions and site features, so it has to be always available and accurate.
Lipeng Zhu and Edward Zhang from Ebay discuss how eBay has been working on migrating ADBMS batch workload to Spark.
Learn how eBay moved their ETL computation from conventional RDBMS environment over to Spark. This was a journey which led to an implementation of a 1000+ node Spark Cluster running 10,000+ ETL jobs daily, all done in a span of less than 6 months, by a team with limited Spark experience.
Ajinkya Kale of eBay discusses their use of Neo4j as a backend to the AI technology in eBay's virtual shopping assistant: eBay ShopBot. The team discusses how they used Neo4j as a probabilistic graph model to drive conversations based upon their Knowledge Graph. They also touch upon the key learnings for deployment and scalability in Google Cloud Platform, and touch upon the application oriented learnings of using Neo4j for a year in production.