Real Time Anomaly Detection: Solving Problems, Seizing Opportunities
The Business Case for Real Time Anomaly Detection
In the first post ‘What is anomaly detection?’ of our three-part series on why anomaly detection is a business essential, we discussed what anomalies are, how they are detected and why detecting them is so vital for data-driven businesses. In this second post, we’ll discuss the reasons why real time anomaly detection is crucial.
The Necessity of Real-Time Anomaly Detection
In our previous post, we gave an example of a software update causing online sales from Asia to plummet. Obviously an anomaly in online sales volume for any specific region or device type needs to be detected immediately, and the same is true for other anomalies. This is because many real-life business anomalies require immediate action. That bad software update is causing you to lose a lot of money every second. And since discovering the problem is the first step in resolving it, eliminating the delay between when the problem occurs and when the problem is detected immediately brings you one crucial step closer to rolling back that update and restoring revenue flow from Asia.
This is also true for anomalies which aren’t problems to be solved, but opportunities to be seized. For example, an unusual uptick in mobile app installations from a specific geographical area may be due to a successful social media marketing campaign that has gone viral in that region. Given the short lifespan of such surges, your business has a limited time window in which to capitalize on this popularity and turn all those shares, likes and tweets into sales.
Real-time anomaly detection is advantageous even when the detected anomalies include ones which don’t require an immediate response. This is because you can always choose to postpone action on an instant alert, but you can never react in real-time to a delayed alert. In other words, real-time anomaly detection is always advantageous over delayed detection.
But let’s think about it – what kind of anomaly of detection systems are able to provide this type of real-time notification? For only one or a few KPIs, a human monitoring a dashboard may work. This manual approach, however is not scalable to thousands or millions of metrics while maintaining real-time responsiveness. Beyond the mere number of metrics in many businesses, is the complexity of each individual metric: different metrics have different patterns (or no patterns at all) and different amounts of variability in the values of the sampled data. In addition, the metrics themselves are often changing, often exhibiting different patterns as the data exhibits a new “normal.”
Manual vs. automated anomaly detection
If manual anomaly detection is inadequate, then automated anomaly detection must be used to achieve real-time anomaly detection at large scale, and it must be sophisticated enough to handle all the complexity described above at the scale of millions of data points or more, updating every second.
The machine learning algorithms that power Anodot’s automated anomaly detection system utilize the latest in AI research to meet this task. Our patented machine learning algorithms fall under the “online” category. This means that each data point in the sequence is processed only once and then never considered again. Online machine learning applications have the added benefit of scalability to the massive amount of metrics businesses keep track of.
As each data point is processed, the online machine learning algorithms work in a way similar to the human brain in the jogger example of the previous post:
- A model which fits the data is created.
- This model, in turn, is used to predict the value of the next data point.
- If the next data point differs significantly from what the model predicted, that data point is flagged as a potential anomaly.
- Anodot’s machine learning algorithms use each new data point to intelligently update the model.
AI anomaly detection in the real world
The power of this application of AI to spot anomalies and the opportunities they present far faster than humans could, has already been used to great scientific success. An AI system developed by NASA’s Jet Propulsion Laboratory was able to detect and command an orbital satellite to image a rare volcanic event in Ethiopia – before volcanologists even asked NASA for that satellite to take images of the eruption.
When working with thousands or millions of metrics, real-time decision making requires online machine learning algorithms. Whether it’s saving your business money or gleaning scientific insights from a brief volcanic eruption, real-time anomaly detection has enormous potential for catching the important deviations in the data.
In the third post, we’ll dive a little deeper into the anomaly detection techniques which power Anodot’s software.