• Brought to you by
IBM - SM360

Predictive insights: The secret sauce

June 13, 2014 by Nathan Cullen

“How does SmartCloud Analytics – Predictive Insights predict a potential anomaly?”

This is the question my teammates get asked almost every day. I work as part of the IBM SmartCloud Analytics – Predictive Insights development team, and we recognize that this is a reasonable question to ask given the bold nature of the word predict. The simple answer to this question is “mathematics”; however, there are some parts of the Predictive Insights logic that can be explained without the need for a doctorate in statistical analysis.

I wrote this blog post along with Ian Manning, lead developer on SmartCloud Analytics – Predictive Insights, in an attempt to answer this question as fully as possible without completely giving away our “secret sauce.”

Behavioral learning

“Analytics is the discovery and communication of meaningful patterns in data.” Predictive Insights discovers anomalies by analyzing your data in the following ways:

  • Identifying meaningful patterns or “features” in the data: This includes features like variance and standard deviation, as you would expect any analytics product to include, but also a list of other features that we have identified as appropriate for the IT Operations Analytics (ITOA) space.
  • Using a set of analytic algorithms: Predictive Insights uses algorithms developed by IBM Watson to take the set of defined features, learn what is normal and tell you when that normal behavior changes.
  • Employing heuristics: These heuristics leverage what we have learned in ITOA both to help detect anomalies and to stop anomalies from becoming alarms (not alarming on spikes, for example, which are common in ITOA).
  • Analyzing relationships between key performance indicators (KPIs): Predictive Insights can discover anomalies on individual metrics and on groups of metrics. Predictive Insights learns the mathematically discovered relationships between multiple metric instances and can identify anomalies within the data based on changes within these relationships.

Relationships are the key to understanding your data

The following image illustrates how Predictive Insights can identify an anomaly event by observing the relationship between two metric instances.


The key is that Predictive Insights has learned over time the relationship between these two metric instances and understands that their divergence is a cause for concern. Predictive Insights analytics identifies the anomaly before the metric (web response time) emerges from the baseline.

The image shows that by observing the relationship, and not just the baseline, you can gain an early warning, giving you the time to act before the situation negatively affects the business.

When an anomaly becomes a problem

image2Predictive Insights does not raise an alarm for all anomalies it discovers. An illustration that was mentioned at the IBM Pulse event earlier this year compared the dynamic baselines Predictive Insights generates to golf course fairways: just because your ball has landed outside the fairway does not mean it has ended up in a water hazard; likewise, just because your ball has landed inside the fairway does not mean it is not lodged in a fairway bunker.

After Predictive Insights has identified an anomaly within your data, it applies another layer of logic, or heuristics, to establish confidence that the anomaly really is a problem. The anomaly is tracked over time, and if the anomaly is seen by the analytics as persistent, Predictive Insights becomes more confident of a problem within your data. Only when Predictive Insights is fully confident that the anomaly really is a problem will it escalate that anomaly to the user as an alarm.

Predictive Insights is like the groan from the gallery: if you don’t hear anything, you know your ball is safe.

If you have any further questions on Predictive Insights, please contact me on Twitter @NathanCullenIRL, add a comment to this blog post or use the SmartCloud Analytics – Predictive Insights forum.


  • Dmitri Dodor

    Good info thanks

    • https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000002984&ps=25 Nathan Cullen

      Thanks Dmitri, and we have just launched version @NathanCullenIRL