Transcript 0:00 [upbeat jingle] Welcome, everyone, to another episode of FinOps Weekly Podcast, and today we have a very special guest. 0:09 We are going to dive into how AI is distracting people from controlling our cloud cost, and I don't think we can have a better expert, uh, [chuckles] about cloud than David Linthicum. David, how are you? 0:21 I'm doing great, man. It's great to be on your podcast. Long time coming. I hope I get invited back. Yeah, for sure. For sure. You definitely should be here more. And, you know, I wanted to, to people... 0:31 If people don't know you, like, can you tell us a bit about your, your background and why you think this topic that we are going to talk today is, is relevant? Yeah. 0:40 I'm obviously an enterprise, big enterprise technology geek. 0:43 Been in the business, uh, 35 years, and actually was doing AI development back in 1985 when I got out of college, but spent most of my time as a CTO for different companies. 0:52 In my last, uh, gig before I retired, I was the chief cloud strategy officer at Deloitte, uh, there for a happy eight years, and now I've been doing the influencer thing, have a couple of YouTube channels, David Linthicum Is Not AI, and the Cloud Computing Insider, which is what you're gonna be on. 1:06 And so I'm looking forward to getting that episode out. Um, I do a lot of writing and thinking about cloud computing, AI, how enterprise, pri- uh, enterprises look at it. 1:15 If anything that, uh, I'm known for is being kind of the contrarian out there, take people to task in terms of what's working and what's not, and really kind of ask the tough questions to the vendors that I think many people aren't willing to do, and I think that's something that, uh, needed to be said in the industry. 1:30 It's the role I play. I'm, I'm sure I'm leaving millions of dollars on the table [chuckles] by asking these questions, but that's the ro- that's what I love to do. I love, uh, taking people to task. 1:39 I love taking p- uh, topics apart and figuring out what really is working. 1:44 That's, that's a very, you know, a tough work, but it's, uh, I think it's a very rewarding one to ask the, you know, the tough questions, the, the things that people want to hear and they never get an answer for. 1:55 And it's, it's a very well-known, uh, job in the industry. 1:59 And, you know, I will leave all the links in the, in the description about you, but can you tell us, you know, the, the main topic of today is, like, you know, we have seen this, let's say, bubble or this growth of AI, um, and then we come from the FinOps perspective. 2:14 So FinOps was built upon cloud cost, and we are seeing that the conversation is totally focused on AI, and people are kind of getting distracted out of cloud cost. So are you feeling the same? 2:27 Are you seeing that trend in the, in the industry? 2:30 Yeah, and the r- and one of the things that drives me nuts, I don't think there's a demarcation line between public cloud providers, private cloud u- usage, things like that, and the enterprise systems that have been there for 35 years. 2:42 So I get frustrated when people try to isolate everything associated with FinOps with just cloud-based FinOps when I think FinOps is systemic. 2:49 So most of the time, enterprises do not have good cost controls pl- in place and cost governance, and they first get introduced to it because the cloud, and it's a consumable product. 2:58 They don't wanna overspend in the market, and so they put these cost governance and policies around utilization of the cloud services, and they don't understand the interdependence between those services and the services that exist in the enterprise. 3:09 The old DB2 system, you know, the COBOL system was written 30 years ago, things like that. 3:14 All of those things probably work together to increase your bill, and so you have to understand systemically what's going on, and that's what drives me nuts. Now, with the AI thing, it's a bit more of, of more. 3:25 In other words, as people are standing up these AI systems, they're not thinking about FinOps. 3:28 They're not thinking about cost controls, yet these things, when I'm pricing these things out, they're 10 to 20 times more expensive than a traditional version of the same system. 3:37 So in other words, if you build an inventory control system that leverages an LLM, for example, it's gonna be 10, 10 to 20 times the cost of a traditional inventory control system that may run on AWS, you know, S3, for example, uh, a- and, uh, EC2, for example. 3:52 And so that drives me nuts because it's all interrelated, and if you're moving into AI, the cost controls are even more important. 4:00 And so I'm getting a call a week now from people who are like, "You know, I just got my, uh, AI bill from AWS, and it's $100,000, and I thought it was gonna be 10,000." I said, "W- and that's a true story, by the way. 4:11 Well, what, what's going on? Do you set limits? Did you put policies in place? Uh, did you s- set spending restrictions? You'd be able to measure, you know, how are you utilizing the remote LLMs?" 4:21 All these sorts of things that kinda come in, uh, come into play, and the reality is they haven't thought about it yet. 4:26 So everybody's pushing these AI systems out, and typically they're more pilot projects now and experiments than anything else. 4:33 They're not considering how these things are gonna be paid for, how they're gonna monitor cost, and how they're gonna optimize these systems. 4:40 In many cases, people are building these huge AI systems that are way overbuilt. They shouldn't be using GPUs. CPUs are gonna be just fine. You know, on the cloud providers, you're able to get it both. 4:51 Uh, and they're not necessarily thinking about how to optimize these sorts of things, and now that we have this data center and evidently this capacity shortage coming up, we're gonna have to get really good at optimizing these architectures so they're able to burn the least amount of money and provide the most amount of business value. 5:06 You know, who'd have thought that was a good idea? So guess what? 5:08 If you do that, you gotta be really good at FinOps, and you gotta have to understand how FinOps lays into that infrastructure, lays into that architecture, and how everything is really complex, very heterogeneous, and interde- interdependent, and this is something that we're awful at now. 5:22 I think it's the worst thing in the industry that no one is bothering to figure out this FinOps story. They're not watching your podcast or reading your newsletter, which I recommend that they do today. 5:33 Get smart on this stuff, and it's just blowing everything up. Yeah. No, indeed. 5:38 We, we are seeing, um, you know, um, mispractices, and I think it's, after all, it's going to be a good side effect for, for FinOps, for people to understand how the framework works, how to apply to their business, how it, how, as you mentioned, how general it is for, in general, for IT, um, spend and how it can be applied to, to a lot of things. 6:00 And one, uh, thing that we have seen is that- People weren't d- doing that for, for cloud. They, they weren't doing it... 6:09 They may be doing something for IT integrated into a more general, you know, cost attribution system. 6:16 But now with AI, especially with the, the difficulty to predict its, its spend, um, it m- got to a point where, where it's almost mandatory and, you know, are you seeing 6:28 more, like, more questions on how complex it is to measure or to predict how much it's going to, uh, cost your AI next month? You're seeing that from, from your clients? Forecasting is the biggest question I get. 6:42 And in other words, they're trying to do it after the fact. In other words, they're getting the big bills and suddenly they, "Well, I guess what? I need to do some forecasting." 6:48 We just saw the same, you know, issue with Uber. They were saying, "Hey, everybody out, go out and burn some tokens." 6:53 And everybody burned some tokens, and suddenly their budget blew up and now they don't have any money to burn any more tokens. [laughs] And AI is obviously gonna be a big part of their business. 7:01 So understanding the complexities of it and how everything is interrelated is the toughest thing to do, 'cause people are willing to put a meter on top of, uh, you know, on top of Bedrock and other cloud-based services that are running AI and things like that, and it's fairly easy to do. 7:17 The thing is, Bedrock doesn't act on its own. It's communicating with your DB2 system, it's communicating with your data warehouse that you bui- built 15 years ago. It's communicating with your mobile computing platform. 7:27 All these very heterogeneous things, things that may live and not live in the cloud, uh, even live in different clouds. 7:32 So you may have them in Amazon, Microsoft and Google, you know, in different, uh, heterogeneous environments. People haven't figured out the interdependencies and how to make that work. 7:40 So they're very much into dealing with FinOps as a island of automation, but that's not the way systems work. 7:46 Most of the AI, uh, architectures that I do, you know, they have 15, 20 different systems, and they're on five to 10 different platforms. 7:53 And so they all charge you different money for different, uh, different resources at different times, and how you do it, things like that. 7:59 Many of them are gonna be owned resources, they're gonna be running on a managed service provider, which means you're still charged depreciation on those things, and you have to figure out how much you're paying for those things ongoing, as well as the humans to keep it up and running. 8:10 So you get into these very, um, complex cost models that scares people. And so I find they wanna do incremental tactical cost understanding and optimization on the particular platforms by different people, by the way. 8:23 They're not talking one to another. 8:25 So the person who's doing cost optimization on AWS is not to look, doing cost optimization on the mainframes, even though these things are coupled together, and they're trying to figure out how to make this happen. 8:35 So most of the times people will kick the can down the road, but the complexity is too much, too fast, and that really is what's going on right now is this stuff's too expensive. 8:43 You know, people could ignore cloud FinOps. 8:45 You know, cloud was typically three to four times the amount of money if it's, if it's misoptimized, and that wasn't really gonna kill your budget, and they just threw money at it. 8:53 Now this stuff is 10 to 20 times expensive, and now it's, they're in a big panic trying to figure, "Okay, how do I make this work? These things are all interdependent. 9:01 It's not big one siloed applic- uh, siloed server that's running my AI. It's 20, 30 servers on different resources. They're charging me different money. Help us, Dave. How can FinOps make this work?" 9:12 Yeah, no, I, I think what you said is, is fairly true about the, the complexity with and how, you know, you need to measure all the different things. 9:21 That why, you know, ideas such as, as the focus specification made it, you know, uh, through to the communities because you need that way to unify everything because each provider reports in a different thing. 9:32 You have cost going through for, you know, the on-prem, the private clouds, the public clouds, everything, and now you have to ingest AI cost and to measure everything in, in one picture. 9:43 And what you've said is very interesting because I wanted to ask you a question about how important you think is the role of observability systems and you, in being able to solve these cost problems. 9:56 Because after all, we've seen billing data, and billing data is, in general, slow and tends to, you know, be reported, and some people report it one way, the other, depending on the provider. 10:07 But obse- everyone, uh, you know, every system has some observability, uh, in place. So how are you seeing the role of observability both on cloud and on AI? 10:16 Because on AI you can really leverage, uh, observability to have more information about what y- your costs are doing. So what do you th- think about that? 10:26 It's critical 'cause ultimately you need observability to deal with the operational practice. Things are gonna break. You gotta figure out how to fix them, you figure out real cause and all that kind of stuff. 10:33 Observability tools can be very handy in not only telling you when something breaks, but when things are likely to break, when you're actually moving into more of a crisis situation, all those kind of sums keeping you out of trouble in terms of outages. 10:46 Cost observability needs to be thought in the same sort of lines. In many cases, they can be the same tool, in many cases they're not. 10:51 The ability to gather metrics from 1,000, sometimes 10,000 data points within the architecture that you're running within your enterprise, within your cloud environments, your on-prem systems, managed service providers, mobile, whatever you're doing, and the ability to use it as some sort of information portal to figure out where you've been, where you're going, what you're doing, and, you know, how your system's doing in terms of, uh, value that's coming back to the business. 11:17 That, that needs to be the outcome of this, by the way. They are the, basically the metric that we get. It's not how much money we're spending. I'm not as care, uh, care about that as much, but I'm worry... 11:26 I'm concerned about the operational viability of the system and the s- ability for the system to bring value back. I'm cool with spending $10 million on an AI system, um, if I'm able to make 100, 100 million bucks. 11:36 But I'm not cool with spending, uh, $10 million on a system in which I'm seeing more often than not, where it's only making $1 million because we're over, we're over, um, understanding or basically n- not... 11:49 uh, putting policies around the cost, uh, you know, the cost growth within these systems. And so observability needs to be taken as a, as a concept into cost observability. 11:59 So I always tell people this, we need to have, uh, we need to have data points in every system that's running, what they're doing, how they're being billed. That's the big thing. 12:06 You know, in many cases they go, "Well, we're just, we're being billed 100, you know, $100 a day on average." That doesn't tell me anything. What are you doing? 12:13 What, you know, what kind of storage mix and processing mix is in there? Are you leveraging GPUs or CPUs for making that happen? Is there an overage for leveraging CPUs? 12:22 And by the way, can we get into some optimization of it, which I find you can get normally knock 50% off the bill if we do things like buy ahead instant reserves, which cloud providers will do. 12:32 They'll sell you instances ahead of time, and you have to do some predicting and make sure you're aligning this stuff when you do it. 12:38 But the thing is, I've seen organizations save many millions of dollars a month in making these things happen. 12:43 So the only way to do this is you know everything that's going on in your mix as it happens, and you're able to make, uh, intelligent adjustments around it. 12:53 You're able to use observability to enforce policies and cost governance systems. And if you're unwilling to do it, you're, you know, trying to get these monthly uploads, which everybody can... 13:02 They get their bill and they go, "Look, this is my observability." That's not observability. That's just you getting to an outcome where you have no understanding in terms of where those charges came from. 13:11 And if you're unable to do that, then you need to reconsider all of your financial and, you, you know, all of your cost systems in there and making sure you're connected to the right stuff, 'cause right now you're not. 13:22 Yeah, definitely. 13:23 And you, you need to go back to the, to the beginning of the, of the phenoms, which is, you know, the attribution and the visibility to know where things are coming from and, you know, what's causing what, what cost, and being able to, okay, I have now the picture, I can do... 13:37 see it, uh, almost in real time. Now let's move to optimize it. 13:42 And, you know, one, one thing that I wanted to, to catch is, you know, we've talked a lot about forecasting, and we've talked about how AI is affecting everything and how cloud is affecting everything, so I want to ask you about your forecast. 13:54 So what do you see coming in the next six to 12 months happening on AI and cloud cost, uh, with all the changes that we have seen? 14:05 I think that, uh, one of the things we're gonna see at re:Invent in a month, and I'm gonna be there and I think you're gonna be there, too, um, you know, ultimately the ability to have the tools so we can finally forecast this stuff. 14:15 'Cause I hear this from the vendors all the time, "You can't forecast utilization of AI. It's gonna do what it does." And you, you truly... I understand that. I build AI systems. They have dynamic behavior. 14:25 They're gonna change behavior or take on different personas as they need to. 14:28 An agentic system may move from 9 agents that are running to 900 agents that are running, you know, depending on what you're doing, and they're gonna make their own decisions interdependent one to another. 14:36 Well, there's ways in which you can forecast that. We understand what they're gonna do. I can pre-guess how the agents are gonna behave. 14:42 Believe it or not, you can do that, and you can pre-guess how the monolithic AI systems are gonna behave. Believe it or not, you can do that. And you know already the cost metrics. 14:50 I mean, w- you, when we were talking on the other podcast, you had a great, you know, observation, the ability to kind of use points of consumption as the ability to kind of figure out how many resources you're using. 15:00 And based on the, um, projection in terms of what's gonna be going on, whether it's inventory control system, logistic management system, things like that are leveraging AI, we know the behavior of these things and we can predict behavior. 15:12 And people are like, "No, you can't, Dave. They're AI systems. They're gonna have their own intelligence." I get it. However, w- for the most part, we can predict what they're gonna be doing. 15:20 We can model what these, uh, what these systems are gonna be doing. 15:23 And if you don't believe me, you can actually go to those systems and, and any LLMs and other things, you can give it the patterns in terms of what you're doing and say, "What is the range of cost utilization over the next couple of months?" 15:34 And they'll be able to tell you that, and then be able to put the policies and processes in place. So forecasting is a big cop-out right now with the AI, world of AI. They don't think you can do it. 15:44 The reality is if you don't, you're not able to do it, you're not able to limit spending, and you're not able to keep your costs under control, which means you're not gonna bring value back to the business, which means the AI's gonna fail, just like the 95% of the AI projects that we saw failing via a study from MIT last year. 16:00 Yeah, that's, that's true. And, you know, we, we may see the, the raising of the forgotten... 16:06 I, I'm going to call, call it the forgotten brother of AI, which is machine learning, that is fairly, a fairly useful, uh, practice to forecast and to, you know, analyze data patterns better than, than probably the LLMs that took all the, all the fame. 16:21 So we'll probably see, uh, you know, those raising up based on, you know, observability, information, and data, which is one of the major, I think, side consequences of the use of AI is the amount of data that it's generating and how we can leverage more traditional things to, to go. 16:39 So I think that's, that's amazing and, you know, for people that m- may not know you, what, where do you think they should find you and what do you recommend to check from you, David? 16:50 Yeah, I mean, Cloud Computing Insider YouTube channel. It's one, I had it out there for about three years now. It's at about 400,000 s- uh, 400,000 subscribers. 16:58 And then Dave Linthicum Is Not AI, where I'm doing the AI s- AI expert, AI skeptic, which is truly what I am, and the ability to kind of take a less observed look at what this technology means and how you bring it to your enterprise. 17:09 Also, you can find me on X and LinkedIn, the normal places, or you hang out as well. Yeah, for sure. And we'll leave all the links in the description below. 17:18 But definitely check, uh, the analysis from, from David on, on YouTube, because they are very, very insightful and, you know, uh, follow the, the truth on the reality of what we are seeing. 17:29 So David, it's a pleasure to, to have you here today, and thanks for taking the time and sharing with us. Thank, thanks for having me. See you in the next episode. Bye-bye. [outro music]