Blog Post
3 min read
3 Approaches to Intelligent, Proactive Monitoring & Anomaly Detection
Capping off a busy and productive week at Strata + Hadoop World where Ira Cohen, our Chief Data Scientist, led a presentation on the need for anomaly detection on mobile apps, we were pleased to be part of the San Jose Meetup: Finding the needle in the haystack.
Nearly 100 people (standing room only!) came to hear about anomaly detection for proactive monitoring from PayPal, Uber and Anodot, hosted by PayPal in their Town Hall building.
PayPal’s Proactive Monitoring Transformation
The evening started out with a presentation from Bryant Chan, PayPal’s Director of Engineering, who leads the monitoring and logging team. Bryant described the problem of inadequate monitoring that many organizations face today and went on to explain how PayPal is transitioning their monitoring to be more proactive and intelligent. It was interesting to learn how PayPal is blurring the borders of traditional logging and monitoring to create one unified platform that enables them to reach the highest standards of availability at scale. Bryant shared their current architecture which leverages open source technologies including Kafka, Druid, OpenTSDB and Elastic to handle the huge volume of transactions. He also talked about the need for smarter solutions that enable fast detection and - most importantly - faster root cause analysis, and what they are doing in the anomaly detection space to achieve this.
Uber’s Challenge: Make Transportation as Reliable as Running Water
Next up was Franziska Bell, Data Science Manager at Uber, who discussed how her company developed its own in-house anomaly detection solution. Three years ago, Uber realized they needed an anomaly detection solution to realize Uber’s mission to make “transportation as reliable as running water.” Since then, the company has worked to develop a solution to detect anomalies for their more than 500 million metrics. Fran described the requirements of an anomaly detection system and what they have achieved so far, which was very impressive. She noted that the solution is a work in progress because with so many metrics (increasing in double digits % on a monthly basis), it’s a huge problem to tackle. Currently in process are improvements in different models for what is considered “normal” and correlating alerts for quick investigation.
Anodot Presents Autonomous Analytics
Finally, Ira Cohen presented “Autonomous Monitoring,” the central concept behind the Anodot solution, enabling organizations to perform any type of analytics (i.e. past, real-time and predictive) on practically any data with minimal configuration.
Below is Ira’s full presentation which goes through an example of a successful mobile application that suddenly sees a steep increase in the number of uninstalls and explains how Anodot’s real-time anomaly detection solution helps uncover exactly what happened to cause this behavior so that the development team can fix it quickly.
After the sessions, everyone was invited to mingle and speak with the presenters. Everyone had the opportunity to ask questions, exchange business cards and network. Special thanks to Uber and PayPal for joining us for this informative and successful event! We look forward to meeting you at our next event.
Read more
