Does Your Company Have an AI or A1 Strategy?

A politician was recently mocked for talking about the societal impact of “A1”, like the steak sauce, when it appeared she was referring to AI.  But for companies developing an AI strategy, there is real risk of developing an A1 strategy, one that depends on following the AI trend, but ends up costing a lot due to vague definitions of what AI is.  

AI tools and technologies are advancing quickly, often faster than the typical decision making pace of many companies.  To stay nimble, companies might allow AI or ML experts to develop strategy, often with limited input from Finance, FinOps, Procurement, or even parts of the Engineering team.  But ultimately this can prove as costly as having FinOps provide cost input after architectural decisions have been made, rather than before.  A lot of time and effort can be wasted to make what’s already been built more cost effective, rather than designing efficiency up front.

Alternatively, the CTO and possibly CEO can pull everyone together to come up with a strategy “to embrace AI”, without really defining what that means.  This can send every department back out across the company to start labeling every software purchase as “AI”, which can allow process automation to get tangled up with model training or customer usage analytics.   But there are ways to prevent this, and to ensure you don’t fall into these traps.

FinOps Role in Strategy and Establishing Metrics

To prevent wasteful spending and tagging it as “AI” to justify the outlay, and to avoid getting too bogged down in slow cross-functional decision making, the first step is to build linkages between the AI/ML engineering team and FinOps.   And FinOps has an important role to play in developing the operating metrics by which the price/performance of AI investments can begin to be evaluated.  For example, when purchasing vector databases, a foundational part of building LLMs, the number of vector dimensions queried and stored are often usage-based metrics and should have some kind of projection before purchase.  While vendors will often assist with this, the reason for doing it internally isn’t to have a perfectly accurate forecast, but rather to build an understanding of what factors drive the number of vector dimension queries, because this will make optimization much easier after the first bills start to arrive.

The next link that needs to be established is between FinOps and Finance.    It is up to FinOps to translate what are sometimes detailed consumption metrics into a language Finance is clear on, and for which it can budget.  Finance might not get into the nuances of how embeddings use semantic representation, but it’s up to FinOps to segregate what’s signal and what’s noise in terms of driving cost.  For example, vectorized search queries can be a significant cost driver when building out RAG (Retrieval Augment Generation) systems, while CPU might be less of a concern.  Additionally, providing a sense of what’s fixed and what’s variable is also important.  

The third organizational link that needs to be created is between FinOps and technology procurement to ensure alignment on metrics, particularly unit costs.  If procurement is negotiating storage costs when the real need is to manage query costs, the misalignment could cause unnecessary financial problems, one that could persist until the next contract negotiation.  

Staying Focused on Just a Few Metrics in Essential to Achieving Results

Another challenge for business teams is that along with other areas of software spend, many AI SaaS vendors primarily offer usage-based products.  This requires not just FinOps understanding of billing metrics, but communicating those across impacted teams, and maintaining tight priorities over which consumption metrics to be optimized.  Importantly, this does not mean building yet another dashboard with all kinds of donut charts, tree maps, and impressive uses of color palettes, but rather keeping teams focused on 1-3 metrics so that they remain centered on top cost priorities.  You can share these via e-mail, write them on a piece of paper, put them in your Zoom background, it really doesn’t matter.  What matters is that they are well understood, well communicated, and that action is being taken based off of them.

Once FinOps has worked with Engineering, Procurement, and Finance to establish how costs will be measured, how they will be optimized, and how they will factor into vendor negotiations, groups can then develop a cross-functional connection rooted in metrics, measures, and priorities, and assess risks and areas of focus with more thought than if the CTO calls everyone together, with a vague mandate to “embrace AI”, without defining what that really means.  

Strategy also needs to be segregated into pieces.  Having an AI strategy is often too vague, not unlike having an Internet strategy 25 years ago.  It typically makes more sense to break down the AI strategy into components, and then consider if those components have any impact on each other.  For example if Accounting is using AI to process monthly close faster, does that impact the developer teams using RAG to augment an LLM?

Once a strategy is defined and segmented, FinOps can set up a simple cost metric dashboard by which projects can be tracked, and broken into categories.   While this will vary by company, it could look something like this:

Hardware/Cloud Infrastructure:

Sample Vendors : NVIDIA, AWS, GCP, Azure, NVIDIA DGX Cloud

Sample Metrics: Cost per hour, Cost per FP16 Teraflop

RAG: 

Sample Vendors Haystack, Pinecone, AWS Bedrock, Elastic Enterprise Search, Google Cloud Vertex AI

Sample Metric: Cost per token per minute

LLMs:  

Sample Vendors: OpenAI, Anthropic, Google/Gemini, Meta/Llama, Cohere, xAI/Groc

Sample Metrics: latency, failed LLM tasks

Vector Databases:  

Sample Vendors: Weaviate, Cloudflare Vectorize, Pinecone, Milvus, Qdrant, LanceDB

Sample Metrics: Cost per queried vector dimension, cost per stored vector dimension

Code Development: 

Codeium, Code Whisperor, GitHub, CoPilot, Cursor AI

Sample Metric: Time saved on code completion, bug fixes, documentation

Code Deployment:

Note: Most AI code deployment tools are the same as traditional CI/CD providers

Sample Vendors: CircleCI, Jenkins, GitLab, BitBucket

Feature Management: 

Sample Vendors: Hospworks, Continual, Tecton

Sample Metrics: Cost per feature value

Data Warehouses/Data Lakes:

Sample Vendors: Databricks, Snowflake

Sample Metrics: Cost per DBU, cost per table, cost per query

End-to-End Build Tools:

AWS Sagemaker, Google Vertex AI

Sample Metrics: Cost per training run, data prep time

API Access to Foundation Models for Model Building:

OpenAI API, AWS Bedrock, Anthropic Claude API, Azure OpenAI Service

Sample Metrics: Invocation latency, cost per input token, cost per output token

By breaking AI into pieces, companies stand a much better chance of focusing on the components of AI that can truly have an impact on the business, and avoid the trap of following the AI hype with vague promises and statements.  FinOps can play an essential role here, both creating the metrics by which costs will be measured, and importantly, communicating which should be the top 1-3 priorities for cost reduction.