Transcript 0:00 Welcome, everyone, to another episode of [pop] the FinOps Weekly Podcast. 0:03 And today, we are going to talk about AI and how it differs from FinOps', uh, usual work and how to control, uh, their cost, techniques that you can use, how to feed the models correctly to your workloads, and, you know, how to train your people here, uh, into AI cost management. 0:22 And for helping me with that, we have Rafaela. Rafaela, how are you? I'm doing well. How are you, Victor? I'm doing great. I'm doing great. So we met in, in FinOps X. 0:30 It was wonderful to finally meet you in, in person, and you know, definitely wanted to, to chat with you, uh, on FinOps here in the, in the podcast. So thanks for, for taking the time today. 0:40 And, you know, wanted to, to talk about AI. So let's, let's go, uh, into it. 0:46 So I've thinking and I've talked with a lot of people about the difference between the usual, uh, issue with, you know, cloud cost that FinOps, uh, try to help, and now all the issues that token... 0:58 that, uh, you know, the tokenomics practice, the AI cost practice has. So can you tell us a bit more about the difference? Because AI is impacting now 1:07 everyone in the company, and, you know, cloud is only used by, uh, the IT department. So can you tell us a bit more about the, those differences that you're finding? So traditionally, I... 1:16 So my, in my current role, I am focusing on cloud cost only. 1:20 So traditionally speaking, for FinOps practice within cloud, we only had to educate the engineers and the leadership on how to optimize costs and control costs, right? But nowadays in AI, 1:33 everyone's using it in the company, everyone's encouraged to use it. So it w- it's not just technical people that we are talking about, but the HR, the marketing team, or admin people that everyone's using. 1:44 So there's that gap of speaking their language, uh, talking to, to non-technical people as well as technical... usual technical users. Yeah. No, that's true. 1:56 That's one of the things that we got from, uh, FinOps X that, you know, a lot of, like... I think the numbers were, like, 80 to 90% of the applications developed by AI are out of the IT, uh, uh, department. 2:10 So that means, like, you know, uh, 9X, uh, the workload and, and definitely it's a problem that now is full company. 2:17 So, you know, what, what are the challenges that you are finding and, and how you think, you know, the FinOps practice can help govern those, those costs now, now generated by, by AI? I think right now when... 2:32 Let's go back to the term tokenomics. Hopefully, that term- Yeah... [laughs] remains relevant by the time this podcast comes out, 'cause who knows what term we're gonna be using next. But- Yeah... 2:41 um, if I'm talking to somebody who's non-technical, even technical people nowadays, when I'm talking about, "Hey, you're using X number of tokens," that really doesn't mean anything, right? 2:50 So we have to kind of transl- again, translate into language that people can understand in the context of their own work. So let's say you're cr- creating a customer support ticket from the s- support team. 3:04 We have to say, "This... It, it costs X amount of money for X amount of tokens that you're using," instead of saying, "It's costing 5,000 tokens." What does that mean, right? 3:14 So it's a- having the c- giving the right context to the metrics that we'll be using for AI. 3:22 And the other issue is that there are all these related costs that, uh, to AI that, uh, that's not just strictly gen AI costs that you might be seeing as a line item in your billing. 3:34 Um, that's hard to track because of tagging issues. But that's... I think tagging issues is something that we all have been struggling with since the beginning of FinOps', uh, practice- Yeah... has been, uh, flow, right? 3:47 It's not something that's new. But there are many hidden cost items that people are not m-might, might not be familiar with right now. 3:54 And I'm still, you know, figuring out myself within my company and, you know, on my own as well. So there are a lot of uncertainties that people face with, uh, that comes with AI cost. Yeah. No, that's, that's true. 4:06 There is... There are also, I think, the, the undeterministic things, uh, related to AI. 4:13 So, you know, you never know the output that it's going to be, how much tokens are going to be consumed in certain things unless you, you know, you constrain it a lot. 4:22 Um, so there is a lot of variables that are, let's say, more out of our control. Rather than in cloud, you can be more predictable, let's say, unless you have spikes and everything. Um, but it's less predictable. 4:33 And, you know, one, one of the major things, uh, that I'm seeing is the governance issue, um, because now everyone has access. 4:40 So now, before in, in FinOps, let's say, so we had the issue in governance because only a certain set of people will use it, um, and we need to govern that, and that was a problem already. 4:51 But if you have a full, uh, company-wide technology enablement like A- AI, now the cost governance is a whole company government, uh, like a whole com- company government issue. 5:04 So, you know, why do you think this cannot stay only in FinOps, and how do you, can control all the cost governance across, across the company? I think it goes into... 5:16 It goes back to the knowledge of technology that people might be having, right? Again, there is a different levels of technical knowledge that w- all the AI users will be having. And 5:28 again, without guardrails, as you mentioned, it can go quite [laughs] devastating. It can have a quite devastating effect on the cost because people might be uploading, let's say, like, 10,000-page PDF, uh, to, uh, 5:44 gen AI that they're not supposed to, 'cause that's going to drive a crazy amount of cost. 5:48 So I think that having some kind of guardrails and training for, to people to know how to use AI correctly, and also from security perspective as well. 5:57 It's not just a cost issue that, you know, people should be knowing what to put into AI, what not to put into AI, and what kind of-... questions to ask and how to ask those questions, that's very important. 6:08 And right now, I think, uh, and I'm not saying this about in, in my particular company or, like, anyone in general, but I know that some people are, you know, prone to using shadow AI as well. 6:20 That's not [laughs] approved, explicitly approved in their company, um, but they still upload it, uh, upload whatever information they want to into their own accounts in AI. Yeah. 6:31 So I think that's the, um, one of the many it- risks that people are facing. Yeah. And, you know, but that's also, like, a security risk and, like, a lia- liability risk. 6:43 We already have seen, uh, that, uh, popping up in several ways. We just have a, a recent, like, issue with, uh, OpenAI and Hugging Face weird thing- Yeah... that happened. 6:54 [laughs] The AI, yeah, the AI acting upon itself, um, which is crazy. [laughs] Yeah. So that's also, like, another... I, I was discussing that before our call with another friend. It's like the, 7:08 sometimes the value is not in the, in the value generation, but in the risk prevention. So, like, you, for example, let's say a fraud, uh, detection system. 7:21 Fraud detection system is making you not to lose money, so it's making you, let's say, uh, you know, value, let's say value, value loss avoidance or however you wanna call it, because you are going to have, you may be a, you may have certain set of frauds or certain set of security attacks and everything. 7:37 If you allow all of them, let's say you lose whatever, but if you are able to, to generate a system that prevents that, so you avoid having these security issues, you are adding value to your company, like in, 7:51 you know, value, let's say, instead of, uh, value creation is not, is avoiding value loss, let's say. Uh, so that's also an interesting thing that we need to, uh, consider as well because we need to be... 8:04 No, we are not always generating. We always, sometimes we need to, you know, uh, prevent losing value. [laughs] Yeah. I think it goes back to the concept of, like, ROI and unit metrics- Mm-hmm... 8:16 um, unit costs, because you wanna look at how much AI tokens or whatever consumption, AI cost that you're consuming versus how much value you're creating out of it, right? 8:27 The, it goes back to the whole concept of FinOps. 8:28 It's not about saving the most amount of money, but how much value you're getting out of per dollar that you're spending, or per euro [laughs] per eu- for your case, or per token that we'll be using, right? Mm-hmm. 8:39 So it's not, um... 8:41 I remember, like, I was talking to a friend about their AI usage tracking within their company, and they have, like, a leaderboard, uh, showing which develop we're using, was using the most amount of tokens or u- using the AI the most, but they weren't really tracking what it was being used for- Yeah... 8:59 which makes absolutely no sense. But right now, I think I've, I think a lot of companies are doing it. 9:03 They wanna see how much AI usage is being done within the company, how many people are using it to do whatever they want, whatever tasks they wanna get completed. But we have to go back to value creation, right? 9:14 Like, what kind of, like, h- what exactly are you doing with the AI that's saving us money, that's, uh, preventing certain things from happening, right? So that's the, um, 9:24 metrics that we need to be looking at, rather than just looking at, uh, how much time we're cutting off, how much, uh, dollar we're saving from, by using AI instead of having another person do it. Yeah. No, that's true. 9:37 I think that'll, that'll be solved once a significant bill or a bill shock, uh, happens that puts everyone in, in the right place. Yeah. Like, the bill doesn't lie. [laughs] So, uh, it's, it's going to be fun. 9:51 And, and one of the things with the, with the bills and, you know, uh, that we suffer also in cloud was the, the volatility of prices, or like, let's say, the less predictability of the pricing. 10:01 You know, we, coming from the, the data center or the on-prem perspective is, like, you buy this stack of or rack of servers that cost- Mm-hmm... that amount of money, and you amortize over time. 10:12 Then cloud was more or less predictable because you are consuming and you are creating and, and reducing workload. But here is, like, everyone is spending, like, every time. 10:22 Uh, like, uh, any command or anything, uh, introducing to cloud or whatever AI provider is a cost, uh, decision. So how are you seeing, like, the volatility of the, of the AI, you know, cost and, and ecosystem? Yeah. 10:38 The g- I think it's more akin to serverless. Um, unless you have certain guardrails in place, serverless can drive up the cost quite significantly, right? Mm-hmm. 10:49 And that's the, that's, you know, one of the things that people don't expect so much 'cause they think, "Oh, serverless option is going to be cheaper," which is not true unless you have, you know, everything configured correctly. 11:00 So ins- uh, unlike, you know, as you mentioned, unless un- um, unlike traditional cloud 11:06 infrastructure, where you can have more predictable usage in an hour, um, and AI is more driven by user interaction styles, the autonomous agent loops, you know? 11:18 So that's the kind of guardrails that you can have in place. 11:21 How many, uh, connect, API connections that can be made, for example, uh, how many loops you can try before it breaks, um, and also controlling the payload size. So, you know- Mm-hmm... controlling the context. 11:34 And, um, having those guardrails in place and also training the users how to use it properly ahead of time would prevent that kind of volatility from, um, going out of hand. Yeah. No, that's true. 11:47 Like, uh, everything reduces to, to guardrails and setting, you know, policies in place that, that makes everything controlled. 11:54 And so you mentioned, like, uh, several topics that are related to AI, like techniques about, you know, the prompt, uh, the context, and the, the guardrails. 12:02 On your side, like, what's your- I'm going to say your favorite to, to tweak, but also the ones that is providing you the most results. What are you, you know, looking, uh, for more? 12:13 Like, it's the context or it's the prompts or it's creating guardrails. What's the one that you tend to work the most? 12:20 So I used to laugh at people, [chuckles] like a few years ago when people used to use the term prompt engineering, because for me personally, I thought that was, uh, kind of like overkill. 12:30 I thought people knew how to do it. But I've seen some prompts created by other people and realized that not everyone knows how to use it properly, right? So- Mm-hmm... again, the whole... 12:42 It, it depends on the how the AI cost is triggered. Um, some companies, they have it triggered by license base, and for now it's kind of unlimited usage. 12:52 But eventually what I think what all the AI companies are going to do is going to translate- uh, transfer it into, um, token usage or more limited usage based on, um, how many prompts or how many contexts or whatever they can have, right? 13:08 So I think it needs to start from prompt engineering. You have to re- really know like how to ask the questions and in a more concise 13:17 manner so that way y- you're giving it enough information for it to process, but also leaving it enough room to, for it to think on its own, rather than kind of feeding you the answer that you wanna hear from it. 13:32 You have to be able to ask the right amount of... give it the right amount of context, right amount of words or prompt, and have the flexibility built into your, um, answers. Does that make sense? [chuckles] Yeah. 13:47 No, no. I think- Yeah... it's finding that right balance between instructions and, and flexibility and being able to- Right... provide. Because if it's, you know, depends also on the answer that you want. 13:59 If you want it very straightforward and, you know, same format repeatable, that's one thing and maybe AI is... 14:06 I would say maybe AI, you need a lot of pre-processing before or post-processing after AI and to give it to the, like the right structure and everything. Right. 14:15 But also if you want a more free, let's say documents, images, et cetera, you want a more creative thing, then that- Right... hmm, 14:23 really, uh, needs a more flexible thing because otherwise it's going to be like the same answer. And you ask the another question, it's like the same answer. And you're like... For example, that happens a lot. 14:33 I'm not going to, uh, cite, but if you, if you put like Gemini, I think it happened to me like a lot with Gemini Pro, uh, l- before one of the, one of the models just would like... 14:44 It was so structured, like if you give it super structured answers, it will give you like the same answer over and over and over again. 14:50 It's like at a given point that the context window, uh, I think it was like, you know, super compressed and that's also another point. [bell dings] Like the, 14:59 the context and, and the, the window itself, like if you have long chats where, you know, uh, the model starts to compress the... 15:08 now they compress the, the context, you end up with the latest ones, like the latest messages and everything, and that causes a, a case. 15:16 I don't know if you've seen like these issues with the, with the context window and, you know, these long- Right... uh, situations. So yeah, it's... 15:24 You don't want the conversations to go too long because then it'll be running everything each time you ask a new prompt, right? 15:30 So you want a set of instructions which is, I guess, it, it can work as both as guardrail and also just kind of general guideline, right? 15:39 Um, for example, like the instruction that I'll be using as a FinOps person, I would say like act as a cloud FinOps expert, and then that's how it would, um, construct its answers all the time based on my instructions. 15:52 And always also you can put up like more security, um, related instructions based on that. 15:58 Say, "Do not access this X, Y, and Z," and then AI would not be, um, running that prompt every time, but it would just know, um, consistently, so you don't have to y- type it each time that you ask a question. 16:13 And by doing s- by setting instructions, because it's a repeated, uh, process that you h- always have to ask, you're saving the context, uh, um, you're, you're, you know, condensing the context window and, you know, saving tokens by having less words to repeat each time. 16:28 And also like the other thing that I've been do- using, um, lately that I really enjoy is using skills. Uh, there are a lot of skills that people are, have been cr- creating. You know, you mentioned Gemini. 16:38 The Gemini has their own, um, repository of skills. There are task specific playbooks that you can be using, um, but instead of have... 16:48 But unlike instructions, you don't have to load it yourself, but you kind of trigger AI to pull from it, so that way it's not always sitting on your instructions, which can again blow it up and kind of limit your answer. 17:02 Um, but it can always refer to it, um, acting as an- another like a expert source that you can pull from each time that you are asking the relevant questions to AI. Yeah. 17:15 And that's also something that people need to be careful about is like depending on... I think it depends on the model and the system that you're using. Uh, the skills and the MCPs can generate a lot of 17:27 context like or overhead to the model when, when it's trying to answer, and especially I think it's MCPs because there is a lot of, you know, words and things that could trigger and it could think like, "Yeah, I think that'll be..." 17:40 And it can, uh, you know, uh, consume a lot of the context and the, the reasoning of the model. So you need to be careful about like, uh, that also relates to, you know, the context and the specialization. 17:51 It's like you need to put it in constrained things and in guardrails that, okay, you have this set of skills, you do this, and you are specialized in that and you don't have like, I don't know, 100 skills enabled and your context is-... 18:05 all skills and you cannot introduce anything because it's not going to happen. It's, you know, we try to make specialized people, uh, and in this case, specialized agents, uh, for, for making it, uh, work. And 18:19 other things that, that we've seen in AI is the, the caching system. 18:23 So caching is very, uh, interesting in, you know, in databases, like usual, uh, things, but in, in prompts it's kind of the same idea but different because it's less predictable. 18:36 How are you seeing the, the prompt caching, uh, working for you, or how are you seeing, or how you are you taking advantage of, of the prompt caching? Yeah. That's, um... 18:48 So I think right now that's something that we are working on as a company- Mm... to figure out for es- especially for each user. 18:54 So, you know, as I mentioned, going back to the beginning of the recording that we were saying that it's now expanded to the entire company, so it's not just technical people that I'm speaking to. So it's kind of, I... 19:06 right now, like in the process of gathering information on what's most, like, commonly asked information or what's something that it should be, um, maintaining, um, so we can cache the information rather than, um, uh, cache the, like, static repeated, uh, information from the prompt rather than having to send same information over and over again. 19:28 And that's something, um, that, you know, each team or, like, you know, each user has to figure out, and we... Like, right now my process is to provide some kind of guardrails, 'cause a lot of people don't even... 19:41 Like, they're not even aware of this whole idea of cach- caching and, you know, prompt caching or context caching. And there are different options to do it, um, but 19:51 yeah, right now we're still [laughs] in the process of figuring it out. Yeah. It's not, it's not... I think it's not as easy as, uh, in the database world, let's say. Uh, even though it's... 20:01 I always think of it like a double-edged sword, uh, because you're gonna have a lot of advantages if done right, but if done wrong, uh, caching can be a, a pain, uh, to, to solve. 20:12 And, and other thing is y- that, that I've... Just in the, I think it was in the past podcast, was the selecting the right model, uh, like fitting the request to the right model. How are you seeing, like, people... 20:27 Because I don't know, maybe people are using Opus 4.8 to request, uh, what's the hour or what time is it. Um, so how are you seeing that? What are your advice on, you know, on the right sizing of models? 20:39 Let's call it that way. Yes. So yeah, right now the approach that I'm pushing for is a T-shirt size approach, um, that you've probably heard from, you know, FinOps practice. 20:49 So what it is, is that we have, like, small, medium, large size, depending on the usage. So small is probably, like, you know, when you're thinking about lightweight usage, let's say composing an email, 21:00 uh, or creating, um, meeting summary, et cetera. 21:05 That should be using the lightest model possible because, again, like, different models have different token con- consumption, and you don't wanna overblow, like, Opus, for example, using, using it for your email consumption. 21:16 I don't think it... Like, for email usage, I personally don't think you should be using agentic AI [laughs] to begin with. Uh, but some people do it. So that's the... where T-shirt size approach comes in. 21:27 So you have small for lightweight usage and large or, like, medium or large for more heavyweight usage, meaning something more advanced, more technical. 21:36 So let's say for developing a certain, like, non-proda- environment or coding something new that's more, a lot more complicated or, like, a lot more mathematical, et cetera. 21:46 So, um, people would not have access, the same access to different types of models. It would have... 21:52 It, it would depend on the department, def- de- depend on the apps they're working on, depending on the projects, et cetera. That's, that's true. 22:01 We need to avoid, like, you know, setting default Opus or setting default Sonnet, and then every, uh, for everything we use Sonnet instead of, you know, right size and, and what's the task, and based on the task we can, we can set it right. 22:13 So we need to be a little bit more conscious about it. Especially when it's your own money, you, you see it more and you are more careful about it, but when it's company money, it's, it's different. Um, 22:25 and wanted to, to also dive into the training because, um, 22:30 on FinOps there was a lot of effort into the, you know, educating the other departments, educating engineers, finance, and everyone involved to, uh, to be aware of costs, uh, in the, in the cloud side. 22:44 So, and before, like, before in the, I think, 2025, mostly all the big companies were about, you know, AI, um, you know, um, scaling and AI training for the company, but less, like, a skill and less about AI consumption, uh, culture. 23:04 So how are you seeing the differences and, you know, uh, now that it involves more non-technical crowd that in, in FinOps where it was targeting- Yeah... IT? Yeah. I think, like, you know, 23:16 like, I think as late as, like, last year, it was a lot more free usage encouraged, uh, meaning- Yeah... we didn't have the concept of tokenomics. [laughs] So it was kind of run whatever you want in AI prompt. 23:28 But right now it's, you know, a lot of companies, uh, AI providers, they're starting to curb the usage, um, by assigning token limits. Um, so again, it goes back to educating people from prompt engineering. 23:41 So teaching people how to... like, what kind of questions to ask and how to ask them, I think that's important. 23:47 Teaching them about the different concepts of prompts, context, instructions, skills, that's important as well. And also standardizing certain things for u- their usage. So when I tell people, "Okay, build a new... 23:59 You know, build, um, AI skills or, as a template for your team or, you know, figure out context, um, or prompt caching on your own," they would not exactly know what to do. 24:11 So that I think that's w- where the role of FinOps and AI and tokenomics comes in. 24:16 We give them- We give, uh, non-technical teams, uh, prebuilt guidelines or even prebuilt templates for, like, AI skills that are approved. 24:25 So rather than ha- having them figure out, you know, long prompts from, on their own from scratch, we can give them guardra- guidel- guidelines to work from so they don't have to start something that's really daunting, but they know exactly what we're asking for and, um, have them trained that way so they are optimizing their usage rather than us having to tell them each time, "Hey, this is how you should be writing this prompt." 24:53 Yeah. I think it's, it goes all about, you know, uh, enablement and, you know, call it, like- Yeah... shift left or whatever you wanna call it. 25:02 Uh, like, you know, putting the, the workload into a pre, you know, a proactive approach, or you try to people to use it efficiently from scratch and not going back after the, you know, the spend was made, like, "Hey, you need to do this, and you need to do, you need to do that." 25:19 And on, on AI training, like, what are your tips or what do you think, let's say, what topics are the, let's say, the easiest to... 25:29 that you see non-technical people, for example, learn that could help companies, you know, reduce their spend? 25:37 I think right now I'm seeing a lot of, um, companies, a lot of AI leaders offering free, uh, courses on their website. I know Claude is doing it, OpenAI, Gemini. 25:48 I think the basic foundational course, everyone should be taking it. If you wanna use AI, I think that should be for everyone. 25:55 And from there, I think it would be really helpful if they have some kind of, like, AI champion, um, form in, within their own team, their co- um, their department, so that way that person is more, gets, maybe pursues, like, a further education on it. 26:14 But again, it c- it can... You know, there is so much information everywhere. Like, you can watch it on YouTube, right [laughs] to learn- Yeah... more basic, uh, foundational AI knowledge. 26:23 And it sh- again, it should go, I think it should really start from prompt engineering. I used to laugh at it, but a lot of people don't know how to do it, uh, don't know how to properly ask the right questions. 26:35 I think it can start from prompt engineering, and then understand different terminology that we're using when it comes to AI costs. So AI, uh, not just AI cost, but, like, AI structure, right? Gen AI structure. 26:45 So learning about prompt, caching, um, context, instructions, et cetera. So I think that's where they can start, and then once they go advanced, get advanced skills, and they can go anywhere [laughs] I guess right now. 27:00 Yeah. It's... Who knows what other materials they will have, right? No, that's for sure, and we're seeing, like, marketing people doing PR request in GitHub. 27:07 So, uh, there is a lot of, you know, a lot things going on and a lot of things changing, and there is a, I think there is a learning curve already for, especially for non-technical people to understand what's going on. 27:19 Yes. Uh, but I think once we see, you know, the f- what we are already seeing, like, the first restriction on token usage, now everyone will try to be more efficient. 27:29 I think probably, I estimate, like, 2027 is going to be more the year where everyone is going to be controlling AI costs and see what's going to happen. 27:40 [laughs] Yeah, I think right now that's the direction that my role is taking. 27:45 Uh, it's, you know, some of it is token based and some of it is license based, and right now I'm just trying to make sure there are all the guardrail, right, right, guardrails in place to prevent costs from going crazy once the token usage limitation kicks in, right? 28:01 So, um, I think for technical people, that's the kind of, um... 28:07 As FinOps people, actually, we can, what we can do to pr- prevent that is even now, like, you can set the monthly team usage for AI, and then eventually once it becomes token usage, you can translate into maybe showback, right? 28:20 So show monthly team usage in simple dollar figures rather than just say, "Hey, this team used 10,000 tokens." Like, what does that mean? Nobody understands- Yeah... outside of FinOps team. 28:29 So you wanna set it up as simple dollar value so you can see who's using how much, uh, in terms of dollar value in AI usage. Oh, that's, that's a pretty good advice. 28:40 Um, so yeah, I think with that, we, we've already covered, like, a lot of value. So thanks, uh, Rafaela, for, for being here today, and I think you have provided a lot of wonderful insights to, to the people. 28:52 So thanks for taking the time today. Thank you so much. I mean, w- hopefully this information will stay relevant [laughs] by the time the podcast gets released because who knows? 29:02 The latest, uh, information from AI is, you know, is rapidly changing, so. [laughs] Yeah. 29:08 Yeah, we hope so, and we hope that in the, in the time that we take to edit and publish the, you know, it's not a radical change in the, in the ecosystem. I hope summer- Yeah... 29:17 you know, gives us a break and nobody does anything crazy even though the OpenAI thing was in July. So we never know. Right. Um. But, I mean, I don't know. 29:25 Maybe another AI will hack into another one and change the whole environment, [laughs] so. Uh, let's hope we have a, a quiet summer, and then maybe September. Peaceful summer. [laughs] Yeah. 29:34 And then in September we'll have all the, all the issues going on. Sure, the thriller. [laughs] Um, so yeah. Thanks. Thanks for taking the time, and see you in the next episode. Bye-bye. Thank you. Bye. 29:44 [upbeat music]