As usage-based pricing models have continued to increase over the past decade, particularly for technology companies, there has been a major shift in budget planning and resource allocation. Since CFOs can no longer predict or approve each and every expense before they’re incurred, variations in usage costs can often make or break business profitability.
Two of the most common usage-based costs come from online advertising and IT-related cloud costs. If not monitored correctly, one wrong configuration can lead to serious runaway expenses.
To solve this challenge, companies developed the role of Financial Operations, or FinOps, which draws on the input of developers, operations, and finance, and the use of software solutions, in order to monitor and manage expenses.
In this guide, we’ll discuss what FinOps is, how AI and machine learning is used to automate this task and obtain more accurate insights, and cover how FinOps is being used for cloud-cost monitoring.
What is FinOps?
Just as DevOps merges developers and operations, the FinOps department bridges the gap between finance and operations. Since usage-based costs are often incurred in real time, FinOps also uses developers and technology to monitor these expenses and detect anomalies before they became serious incidents.
We mentioned that FinOps can be applied to any usage-based expense. Here, we’ll focus on its application for one the largest variable expenses that companies incur today: cloud costs.
One reason that cloud costs are so well-suited for change is that a significant portion of the expense is often wasted each year. In fact, a study by DevOps.com found that between idle resources and over-provisioning, cloud-related expenses wasted over $14.1 billion in 2019.
The Three Phases of Adopting FinOps
As you may know, two major challenges in the adoption of FinOps in an organization are talent and technology.
In terms of talent, this emerging field requires financial professionals to have an understanding of diverse fields such as data analytics, AI, and machine learning. This requires not only an investment in upskilling employees, but also requires a cultural shift of diverse departments working together. In particular, the FinOps Foundation describes three iterative phases to adopt this new discipline and optimize variable expenses in real-time, these include:
- Inform: The first phase of adoption involves empowering the organization with “visibility, allocation, benchmarking, budgeting and forecasting.” As mentioned, the on-demand nature of usage-based billing means that the accuracy and visibility associated expenses is crucial for intelligent decision making.
- Optimize: Once organizations and teams have the necessary visibility into their expenses, the next step is optimization. In the case of cloud costs, optimization can come from more accurate forecasting, which allows you to reserve instances for cost-efficiently.
- Operate: The final phase involves continuously monitoring and evaluating how variable expenses are tracking against business objectives. Since these expenses often involve tracking thousands of metrics simultaneously, this is where technology such as AI and machine learning comes into the picture for real-time monitoring.
Here is an overview of the team structures that enable FinOps adoption within an organization:
AI & Machine Learning for FinOps
As mentioned, the fact that variable expenses such as cloud costs and advertising expenses are incurred in real time, having the right monitoring technology in place is a necessity to prevent runaway expenses.
One of the main issues with traditional monitoring technologies, however, is that they are largely reactive in nature. In particular, they typically rely on static alert thresholds, dashboards, and reports. Similarly, the tools offered by cloud or advertising providers often have at least a 24 hour delay in the availability of data.
As discussed in our guide to business monitoring, business metrics are not well suited for static monitoring described above for three main reasons:
- Business metrics can’t be evaluated in absolute terms due to the fact that they derive significance from a unique context.
- Compared to machine data in which there is a known relationship between machines, business metrics have an unknown topology. In other words, the relationships and correlations between metrics are simply too dynamic to be known.
- Finally, business metrics often have an irregular sampling where it may be minutes or hours between data, for example between a purchase or a click.
To deal with these unique challenges, AI and machine learning have proven to be well suited for real time monitoring. The inherent complexity of business metrics are handled in the following ways:
- 100% of the data can be monitored in real time with the use of a technique called unsupervised learning
- Each individual metric’s normal behavior is learned autonomously
- Each metric is correlated with the others and paired with a deep root cause analysis
As shown in the example below, the technological advantages of machine learning result in a faster time-to-detection and time-to-resolution. In other words, AI-based monitoring offers FinOps teams the right technology to shift from a reactive to a proactive approach.
Use Case: Machine Learning for FinOps Monitoring
As mentioned, cloud cost monitoring is a good place to start with FinOps since small mistakes can often lead to large and unnecessary expenses. For one company, the average hourly bucket size of an AWS service spiked 120 percent over the course of a week. In this case, the spike was caused by a developer accidentally releasing a script that continuously created new buckets.
This increase in average S3 bucket size was detected by Anodot’s machine learning algorithms detected the spike within the first hour. Unfortunately, Anodot’s alert went unnoticed and the issue persisted for more than seven days and caused $10k+ in avoidable expenses.
In addition to cloud costs, FinOps monitoring proves useful in tracking cost anomalies across the business. Check out the graph below showing two spikes related to a Google Maps API. In this example, these spikes cost the user $1,685 and $285, respectively.
If this company was not using Anodot, the data delay could have easily gone undetected, costing the company thousands in revenue. From this graph, you can see, however, that these two anomalies were each detected and resolved within an hour.
Summary: FinOps Monitoring
The rise of the FinOps field has made it necessary for companies to completely rethink how departments share data and work together to control runaway expenses.
The two main investments required for adoption include talent and technology. On the talent side, financial professionals need to beef up their data science skills. On the technology side, new techniques for real-time monitoring, such as AI and machine learning, are being used as a competitive advantage.
As discussed, AI-based monitoring for usage-based expenses can not only track 100% of the data in real time, but by correlating these metrics with a deep root cause analysis FinOps teams have the information they need to detect and resolve incidents when it matters most.