Mastering Cloud Cost Analysis with Metabase:
Leveraging Open-Source Business Intelligence for Effective Cloud Expense Management
Every cloud provider supplies distinct services to manage cloud expenses, such as AWS Cost Management, Azure Cost Management, or GCP Cloud Billing. At the moment, these tools offer a range of robust features like comprehensive cost breakdowns, forecasting, budgeting, anomaly identification, and even cost-cutting suggestions. However, can they fulfill all your requirements?
We think not. The following are circumstances where tools provided by cloud service providers may fall short:
- Monitoring the expenses of multi-cloud or hybrid-cloud applications from a central location.
- Accessing historical data – most tools from cloud service providers only provide a one-year history.
- Circumventing tool limitations – for example, in Azure, costs can only be displayed for each subscription separately. If you’re operating under a pay-as-you-go billing model, you cannot consolidate and display costs for all your subscriptions. This can be burdensome for customers who use subscriptions for resource grouping.
- Carrying out more complex cost analyses like displaying trend lines or comparing the current month to previous ones.
With the increasing attention to the FinOps trend and focus on cloud cost optimisation, it’s becoming evident that cloud service providers’ cost analysis tools are sometimes lacking.
The Fundamental Solution: Excel
For organisations beginning their journey into cloud cost analysis, Excel often seems like the obvious solution. However, while it may be a suitable solution for some scenarios, it comes with its own set of challenges:
🚧 Limited scalability: Excel is not built to handle large datasets or intricate calculations. As cloud computing environments expand, the data volume and complexity of calculations required for cost analysis may quickly outstrip Excel’s capabilities, resulting in errors and data inconsistencies.
🛠️ Manual data entry: Excel necessitates manual data input, which can be both time-consuming and error-prone. Guaranteeing accuracy and consistency across multiple sheets becomes progressively difficult when dealing with large quantities of data.
👥 Lack of automation: Excel does not support automated tasks, such as retrieving data from cloud cost management tools, scheduling data updates, or generating reports. This leads to a higher likelihood of errors and delays in data analysis.
🧍♂️ Limited collaboration: Only one user can edit an Excel file at a time, which poses challenges for collaborative work on cloud cost analysis projects. Version control can also become problematic, leading to confusion and errors when multiple users make changes to the same file.
🔐Security risks: When dealing with sensitive data, it’s vital to exercise caution when sharing or copying Excel files, as this can lead to data breaches. Unauthorised access or accidental data leaks can have severe consequences.
Metabase is an open-source business intelligence and data analytics tool that enables users to conveniently visualise, analyse, and share data. It offers an intuitive interface that assists non-technical users in quickly creating and running queries, as well as crafting custom dashboards and reports.
With Metabase, you can establish a dedicated dashboard for different stakeholders at varying abstraction levels. For instance, a high-level view can present monthly data and cost trends for the CFO, while a low-level view can display specific resources managed by an engineering team, thereby promoting cost optimisation.
Using this BI tool, cost data export can be automated, reducing error risk and the need for manual data export and processing. Fetching data from various sources is no longer a problem. Report generation can also be automated. Issues such as scalability and version control are also addressed. Lastly, the security is significantly more robust than in Excel.
Even users with limited technical skills can use Metabase. It doesn’t take an expert to create a dashboard like the one below.
Despite all these benefits, Metabase has one significant limitation – it does not directly integrate with the cost services of cloud providers.
Accessing Cloud Cost Data
To harness the capabilities of Metabase, you need to export the cost data to SQL or another database type supported by Metabase.
The diagram below provides an overview of the basic DataPipeline we use at Exlabs for handling Azure costs data. For other clouds, you would need to use equivalent services.
Here’s a brief rundown of the main components:
- Azure Cost Management 🧮 – This service exports cost data to a CSV file. Each subscription’s data must be shipped separately. You can automate this process by selecting daily, weekly, or monthly export types. Historical data can also be exported, but only for the last three months! The generated CSV files are delivered to an Azure Storage Account.
- The next step involves Azure Data Factory💾 – a service that processes CSV files and places the data in the database. This processing includes renaming a few columns, converting variables, and ensuring the cost data won’t be duplicated. We strongly recommend using Triggers to automatically execute data processing after a new CSV file is uploaded to the Storage Account.
- A database📂 is the final step of the pipeline. Any engine supported by Metabase can be used. To minimize costs, you may want to use a Serverless deployment.
- The last piece of the puzzle is Metabase👁️🗨️ itself. The simplest option is to use Metabase’s cloud offering, which starts at $85 right now. However, as Metabase is open-source, you’re free to host it on your own VMs. Once Metabase is smoothly connected to the database, you’re ready to start posing FinOps questions and creating dashboards and reports.
Metabase is a robust tool that can assist you in understanding your cloud costs. Now it’s your turn to harness its power!
This article outlines the approach we employ at Exlabs but glosses over the technical details that may be challenging. Check out our webinar “Serverless🚀Right fit for Your Next Project?💰 A Cost Evaluation” for more details.
Here’s what we’ll explore together:
- 💰 The ins and outs of serverless and how it differs from just AWS Lambda
- 🧮 Analyzing whether serverless is always a cost-effective choice
- 📊 Understanding the components of serverless infrastructure costs
- 🔍 Comparing serverless maintenance costs to VMs
- 🏆 Sharing serverless success stories, and lessons from its failures
- 🚧 Guiding you to evaluate if serverless is a cost-effective choice for your use-case
If you find yourself stuck, don’t hesitate to get in touch. We’re eager to assist you with the implementation. Conversations around FinOps are also always welcome.