Transcript 0:00 Estimating is silly Mm-hmm It's a waste of time, it's a waste of money, and so on and so forth. Estimation, no. Fire the person who estimates. 0:07 [laughs] I want to have as many dev environments as I have people in my company Mm-hmm I want to have thousands, but I want them to be active and to be running only when they're used. 0:20 The easiest way to justify anything is to say, "If we invest this amount of money, then we will get this amount of money back." Right? Right. That's the easiest way to justify it. Yeah. But it's not really about FinOps. 0:32 Y- you don't want to scale because it's cheaper. Cheaper, being cheaper is just a side effect. You want to scale because you will have no downtime releases, you will have better performance, and so on and so forth. 0:46 You will get many other technical benefits, and also human benefits, many, many benefits, and all those benefits will at the same time result in, in lower cost. [laughs] Oh, we were very successful. 0:59 We managed to survive Black Fri- Friday because we were scaling up, and then a week later kind of, "But did we scale down?" Thanks everyone for being here today. 1:08 Um, today we have a very special guest into our FinOps podcast, uh, Viktor Farcic, which is someone that I've been following, I think, for two or three years, and then I happened to knew that, uh, that we have met up in several years before. 1:24 Uh, so it was a, an honor for, for me to know that he was available for, um, for the podcast, and to talk him today into a topic that I think it's, um, really, really hot topic based on what we saw in, in FinOps, uh, X conference, that is all related to, um, DevOps costs, the Kubernetes, the resource optimizations. 1:47 And so if you have had issues with Kubernetes costs, or you hate your DevOps because he's doing a lot of, or she is doing a lot of, uh, costs in your cloud bill, uh, you should stay and know what, what Viktor is, is going to bring us. 2:01 So Viktor, thanks, thanks for being here today. Thank you for having me. So, and of course, uh, we always are honored to have our, uh, co-host, Damian. Damian Monafo, how are you doing, man? I'm doing great. 2:14 First of all, it's very good to see Viktor here with us. I remember back in the days in the Sp- [laughs] Spain club, first Spain, Spain club summit, where Viktor joined us, um, to, um, uh, to that special event. 2:30 I think it was great. We know, I know Viktor since then, and it's good to have you here with us and see you again. Thank you for joining. Yeah. It's amazing being back. Yeah. 2:42 So, so Viktor, uh, even though you are a well-known, uh, person in the, in the community, especially in the DevOps, but since this podcast is also oriented to more FinOps that are more specialized, some people may not know you. 2:56 Um, so yeah, go ahead and, and tell us who you are and, you know, why are you, uh, here today. [laughs] Oh, who I am? I'm Viktor. I just play with random things and- [laughs]... 3:09 you know, I, I don't even know what I do anymore. [laughs] It changes from one week to another. It's, it's impossible- You do-... for me to describe what I do- Yeah... to be honest. Yeah. You do a lot of tools. 3:18 And even if I would describe what you do, then what I do, then my boss might find out what I do and then start demanding things. [laughs] Yeah, yeah, yeah, yeah. 3:28 It will be, it will prob- it's not probably beneficial, so we'll, we'll keep it secret then. [laughs] No, but it will be all right. But, uh, and why I'm here, I don't know. You called me. You tell me why I'm here. 3:41 [laughs] No, you're, you're here because, because we have seen that there is a, a topic in, especially in the FinOps community, where two, uh, I don't know, roles or battles, uh, lands, which is like the DevOps against the, the costs, uh, in the cloud. 3:58 [laughs] So there is, there is a, a, a hot topic there because I, for example, I am, I'm a cloud DevOps engineer, even though I do a lot of, uh, FinOps stuff, and that's why I, I, I don't know, I check your, your things. 4:11 So we've seen that, uh, the DevOps sometimes are more focusing to the reliability and the scalability and the performance of the, of the systems that we are seeing, and now there is a, a consideration of the costs, uh- But-... 4:25 that we are seeing in FinOps, so what are your thoughts on this? Scalability and cost are the same thing, aren't they? Why so? Because so let me backtrack a second, right? 4:35 I believe that the, the, the m- big part of FinOps is- Uh-huh... really resource management. Okay. I think that it's all about how can we make sure that we, that our resources are not using more than they need, 4:53 but also not less than what they need. That they're j- just using just the right amount of resources, right? Mm-hmm. Uh, so if our application uses more or less the exact amount of memory or CPU than it needs, then more- 5:10 Mm-hmm... that's how we're reducing costs, right? If our clusters are scaling up and down, uh, to, to pr- to provide just enough, uh, compute to what we need and not more- Mm-hmm... that's how we reduce costs, right? 5:27 Uh, so in my head, and I might be wrong because FinOps is not my main area of interest, it's all about, uh, the resource management, right? Mm-hmm. And, and autoscaling is a big part of that, right? Otherwise- Mm. 5:45 You know, if I cannot scale... So let's say a simple scenario, right? Mm-hmm. If I cannot scale my application up and down, up and down to meet the, the changes of the traffic and the demand- Mm... 5:59 what am I going to do if I'm not doing that? I'm going to probably set my applications and my servers to use the maximum capacityUh, the or, the capacity that is equivalent to the maximum traffic I will have. Mm-hmm. 6:16 Yeah. So let's say that right now in the morning I have 100 users and afternoon I have 1,000 users. Mm-hmm. If I don't scale automatically, then my capacity will be hard-coded to 1,000 users. Yeah, for sure. 6:29 And I will be losing money when- whenever I have less than that. Mm-hmm. Yeah. 6:33 But do you think, uh, so coming from, from the DevOps front, the practices that I am seeing is that do you think that DevOps tend to probably overestimate, uh, the provisioning or, or not care that much about probably the base provisioning? 6:48 And then even though they set up the auto-scaling, which I do, the base provisioning is normally way over-provisioning, especially in Kubernetes. What do you... What are your thoughts on that? So, um, 7:00 base provisioning is- What I mean is like when you deploy the workload and it's more or less stable, uh, do you think that, uh, initial estimate could be overpro- tends to be overprovisioning or do you think it's, it tends to be like, well, uh, say- Nobody should ever estimate. 7:16 Uh, estimating is silly. Mm-hmm. It's a waste of time, it's a waste of money and so on and so forth. Mm-hmm. You don't estimate. Yeah. At least mem- you maybe estimate first day and that's it. Yeah. Right? 7:28 After that you look at data, you don't estimate, right? Yeah, for sure. That, that's just silly. That's how we were doing things 30 years ago, right? I mean, nobody knows. 7:37 So if, if I now develop a new application- Mm-hmm... I honestly have no idea how much memory and CPU that application needs. Mm-hmm. 7:44 I have no bloody idea and I will challenge anybody who tells me that they know in advance how much- [laughs]... how many resources they need. Whomever says that, that, that person is lying or is, or is delusional. 7:55 Nobody knows and that's fine because we have now mechanisms to find out. Mm-hmm. Yeah, for sure. We can actually even go further. There are... 8:05 Actually we, th- there are certain things that we can do that, uh, will actually help us never need to find out, right? Mm-hmm. We can... If you're using Kubernetes, right? You have... 8:17 And I'm not going even to the fancy solutions and expensive ones. Uh, vertical pod autoscaler is part of every Kubernetes distribution. Mm-hmm. 8:26 You can use it in a, in a observe only mode to find out how much memory and CPU your application is using, right? It's not the best solution, but it's a good start. So estimation, no. Fire the person who estimated. 8:42 [laughs] Okay. That's, that's quite a statement, but do you think that from, from that point when, uh, the, the application is deployed and when you, you, you know, you do your... 8:54 You deploy, you put your requests, I don't know, like by a random value, uh, do you think that there is the, the co- the concept or the habit to do the right sizing after that or do you think that that's a practice that is still to be implemented in the- I don't... 9:10 ... uh, life cycle? Most of us have bad habits. Most of us don't know- [laughs]... what we're doing, of course. There is a bad habit in everything. You, you, you don't need to talk about FinOps. 9:18 Anything you can mention and if you, if you talk about and if you end that sentence with a question, is there a bad habit? Of course there is. Yeah. We should just- Yeah... uh, try to remove our bad habits, right? Yeah. 9:33 Okay. That's, that's part of the, the role. I ha- I heard one, uh, uh, the other day I was talking to an architect and he say, "I start with the smallest, uh, you know, resource and then start scaling about." 9:46 I'm not sure if that is the, the right approach, but I, I thought it was interesting, an interesting one. You know, start small and, and grow a, a bit by, a bit by bit. What do you think? 9:59 Yeah, I mean, I, I think that that's how we were doing before when we had, when we... So you know there were three phases in software industry, right? Mm-hmm. 10:10 One phase is that I don't see what's going on so I'm making pure random guesses and we'll see what happens, right? Yeah. 10:18 And then we s- then we learned l- uh, still a long time ago, okay, actually I can observe what's going on, right? I can have something as simple... I'm going to stick with Kubernetes now, but that applies to everything. 10:31 Yeah. 10:31 I can have something as simple as Prometheus where I'm collecting memory and CPU and I can just go there once a week, once a day, once a month, a year, whatever the frequency is, go and see what, how were resources used over time, what so not and adjust. 10:50 Mm-hmm. And then now we don't even need to do that. We can instruct literally machines and say, "You know what? Go there yourself. 11:00 Find out what are the averages, uh, what are the rates and so on and so forth over a period of time and scale up and scale down," right? So there is no need for that architect to exist even for that particular task. 11:15 You just define your applications in a way, way that they themselves can ask query databases with data- Mm-hmm... 11:25 and find out what, what is it that they were doing and how they were doing and so on and so forth and adjust automatically. Yeah. Interesting. But that, that kind of like, uh, let me... 11:39 I understand the, the point that you are, okay, fine if we have in production, you know, the, the, you need to have flexibility and the, and, and in order it's not only to scale but also makes sense, right? 11:51 To make sure that it both scale up and down and it's flexible. Uh, but do you really need- Hopefully left and right rather than up and down- Yeah... but yes. Right. You're right. Yes. Uh, yeah. 12:05 That's a, a, a best practice. But, uh, [laughs] but tell me something, um-How, how, how will you separate the environment? 12:15 Because now we are seeing, like, this approach, you have a multi-s- account strategy, and in which in dev you wouldn't need so much scaling, only for testing times, right? 12:25 And then, and then I guess that the question that Victor asked before comes more into, into play. How, how do, how do you plan it or how, how do you do this kind of... Do, do you even work in a multi-account strategy? 12:40 But, you know, in a dev the logic is still the same, right? Mm-hmm. So if I have a system that goes to some, to goes to some database, something that has metrics- Yeah... 12:52 and asks, okay, so what, what was the usage, this or that, and adjust accordingly by scaling up and down or left and right or both, right? 13:01 So then if you have in dev, uh, if you have that in dev, uh, your memory CPU usage will most likely be much smaller, right? Right. And it will find much smaller numbers. Mm-hmm. 13:13 And it will scale down to smaller number of replicas, right? So is, it, it, the logic is still the same, right? 13:20 The, the question is only whether you wanna bother or no, but if you already set it up for production, uh, to be like that, then it's basically no additional effort to have the same scaling capabilities and, uh, 13:33 whatever else you have, uh, in other environments. Uh, however, in, in dev environments, the real cost is not really the, you know, how much memory CPU you're consuming and 13:48 through that how big of your nodes are and so on and so forth. The, the, the biggest bottleneck in dev for many organizations is that they're n- they're using permanent environments. Hmm. Yeah. 14:02 What that means is that you're spinning up some development environment for the whole team, for the company, for individuals, whatever it is, and that something is running all the time. Yep. Right? 14:17 And that produces two very potentially big issues. One is that you're wasting money like crazy, right? Uh, that doesn't matter how, whether your memory consumption is correct or no, you're still running compute 14:32 while you're sleeping. Yep. And unless you're very, very rich, that's just silly. And at the same time, because those things are expensive, you are not only wasting too much money but never having enough, right? Mm-hmm. 14:48 That's why usually organizations have one or two dev environments, and I, and to me, that's unacceptable. I no-- I don't want to have two dev environments. 14:57 I want to have as many dev environments as I have people in my company. Mm-hmm. I want to have thousands, but I want them to be active and to be running only when they're used. Mm-hmm. 15:12 So at the same time I'm going to reduce cost but also enable people to be effective because big part, and, and I feel that this is probably the, the area that is often forgotten in dev, in FinOps, is that the biggest cost are people. 15:29 Let's start with that. Yeah. That's what costs the most, right? And whenever somebody needs to wait for something, that's the cost. Mm-hmm. Right? 15:39 Oh, I need to wait because Joe needs to finish something over there before I can start working on that something. That's cost. Mm-hmm. That is often overlooked because, hey, we actually measure the tangible things. 15:51 We measure how much service costs and how much networking costs and what so not. We often forget that we are wasting people's time, right? 15:59 So I want at the same time to make people more productive, but als- by, by giving them everything that they need, that there is no bottleneck in what we do, 16:10 but at the same time use the absolute minimum amount of resources. And to accomplish that, the solution is simply to use ephemeral environments or ephemeral servers or ephemeral applications or whatever else. 16:25 Everything ephemeral, right? Production is permanent. Mm-hmm. I have permanent clusters that run production, and the nodes in those clusters are going up and down, scaling down, scaling up. Mm-hmm. 16:36 Everything else, "Hey, I'm going to start working on this." Excellent. I get the environment in a few seconds, maybe a minute. I do, do whatever I'm doing. 16:45 The moment the, the, the system detects that I haven't been working on it for whatever period of time, let's say five minutes, it goes down. Yep. I totally agree with that. Like, we are seeing... 16:58 So we have, uh, I, I have seen two approaches. One is, like, the permanent dev and the permanent Q&A, like, um, shared environment when everyone gets there and all the stuff is deployed there. 17:10 And then we are, uh, seeing another approach that is the ephemeral. 17:13 Like, every developer has its own, uh, ephemeral, uh, cluster, or in this case, if it's a deployment or a workload, depends on if you do it on namespace basis, and then you deploy everything there. 17:26 Uh, you test whatever you want, and then it will be deployed for, I don't know, whatever hours, uh, or whatever time. And then if there is no activity in there, it will be, um, shut down. 17:37 And it's also, and I, I don't know if you, uh, test it out, but if you can connect it with the PRs, uh, stuff, so whenever, when everyone creates a pull request for testing a feature, you can just, uh, send it out to, to create the environment. 17:51 And then whenever, when that merge, of course, is closed, but then beforehand it can be also closed for, for that. What do you think of that approach? Exactly. No, no, same, same thing with PRs. 18:01 Uh, you know, as long as that is... And it doesn't really matter whether it's a PR or it's a dev environment or whatever it is. Mm-hmm. If it's not active, shut it down, right? Yeah. 18:14 And we have solutions for that these days, right? Uh, you can, um-Let's say if we stick with Kubernetes, you can use Knative that will scale down your application to zero replicas. Mm-hmm. 18:27 Uh, you can use vCluster to easily create cluster and destroy it. Uh- Mm-hmm... it's kind of a virtual cluster, uh, depending on some actions. There, there are plenty of solutions, right? 18:38 And, uh, of course, the, it all requires work. Uh- Yeah... but it's not that much of a work either. The biggest bottleneck usually in getting there is that our applications need to change. And this is- Yeah... 18:55 also the area where I feel that, uh, many FinOps, um, focused people are struggling, right? Mm-hmm. "Oh, I can do this, I can do that, but I cannot ask you to rewrite your application," for example, right? 19:11 Oh, that's a no-go. Nobody allows, nobody allows that. Mm. And very often your application is a bottleneck, right? Maybe, for example, let's say that your application cannot scale to mul-multiple re-replicas. 19:22 It is not fully stateless, uh- Yeah... whatnot, right? Uh, so if you cannot scale, we cannot optimize the usage of resources- Mm-hmm... because then we need to assume the worst-case scenario. 19:34 The, the whatever is the maximum that we expect ever to, to happen, right? Mm-hmm. Yeah. That's... I, I, and, and I totally hear you on that point. Like, um, if you, if you are not able to... 19:46 First, if you are not able to scale your application, you are, uh, bounded to the state, and if you are bounded to, uh, to the state, that's really, really, really problematic for, for the cost and especially for reliability as well. 19:57 Like, the moment that the state- Exactly... 19:58 has an issue, you are, you are- By the way, we are hearing a lot lately about, about what Viktor just mentioned, um, the, the thing that the application is being, uh, is not being looked at and th- that requires to do as well, where possible, right? 20:13 Um- Mm-hmm... to start doing optimization at the, at the software level. And, you know, most of the time, let's say, uh, if I stick with the application example, right? Let's say that- Mm-hmm... 20:24 we come to the conclusion we should change parts of the application, rewrite something- Mm... whatever. Yeah. Right? Uh, FinOps is not the reason to do that, right? FinOps is the best reason to justify it, right? 20:39 Because the easiest way to justify anything is to say, "If we invest this amount of money, then we will get this amount of money back," right? Right. That's the easiest way to justify it. Yeah. 20:50 But it's not really about FinOps. Uh, y- you don't want to scale because it's cheaper. Cheaper, being cheaper is just a side effect. 20:59 You want to scale because you will have no downtime releases, you will have better performance and so on and so forth. You will get many other technical benefits, and also- Mm-hmm... 21:10 flow benefits and human benefits, many, many benefits, and all those benefits will, at the same time, result in, in lower cost. Mm-hmm. 21:20 And, and I want to, but I want to highlight something that you mentioned there, and maybe that's a clarification that it needs to be needed, is that the FinOps is not just to make the application cheaper, um, to reduce the cost. 21:31 It's to... The whole idea is to maximize the value that you get from each of the dollars that you spend. So- Yes... that's why you can do, and you can do FinOp- you can justify it with FinOps. I mean, and you can... 21:44 It's the best reason in terms of in this spe- specific term. Like, you can maximize the ROI that you get for your application, and that's the whole idea. 21:53 Like, if you tell business, you tell your manager, the manager of your manager that this application will make, I don't know, $50 more per user with the same amount of money, that's enough justification to do any change, uh, that you need if the cost that you said, the human cost and infrastructure cost, it will be worth. 22:13 So if you are improving these, uh, KPIs and these, uh, ratios that we are seeing now, it's a very hot topic as well in FinOps. 22:21 Once you are improving that metrics, uh, that are related to business, which is the important thing, uh, that's all the justification [laughs] you need for, for your manager, right? Oh, yeah. Exactly. That's great. 22:33 That's great. 22:33 No, no, it's, uh, because it's a very hot topic because sometimes, you know, DevOps and, and certain people will think like, "Yeah, it's only because to make it cheaper, but it will make it less re-reliable," and it's not to make it cheaper. 22:44 It's to make it much more efficient in terms of business returns. Exactly. I mean, y- I think that we are, both of us are actually talk- uh, talking about the same thing- Mm-hmm... just not m-maybe using different words. 22:58 It could be. [laughs] But yeah, it's, it's not about... I think that we both agree it's not about reducing cost, it's about being better, right? Like that. And at the same time, that reduces costs great, right? 23:10 But yeah- Mm-hmm... it's about your users, uh, accomplishing the same res- outcome faster- Mm-hmm... that results in more income, right? Yep. 23:19 Uh, hey, if Amazon can, can, can make me shop, uh, buy something in 15 seconds instead of five minutes, that's, that, that's a business value, right? Yeah. And for them to do that, 23:33 their site needs to be blaz- lightning fast. It's kind of, it needs to happen like this. I'm not waiting for anything. I'm just shopping, shopping, shopping like crazy, right? Yeah. 23:41 [laughs] Exact- And you accomplish that by your application simply being better in ma-many different ways. Uh, faster- Mm-hmm... whatever the, the, whatever the better means, right? Yeah. 23:53 And regarding this topic, because I think it's very interesting to, to continue conversation, and now that we are recording this in a very, uh, you know, uh, shopping, uh- [laughs]... 24:03 uh, amount of, uh, of the calendar, in a very da- uh, shopping date, um, in terms of especially load testing and scalability and, and costs, like how do... Because that's what you're saying, right? 24:15 Uh, there is no good in Amazon that for reducing the cost, they will reduce the amount of orders that they will get on, uh- Yeah... Black Friday because they are selling like crazy, or any other, you know, uh, company. 24:26 So how do you think this load testing and this curve of-Of course, end users should be considered on the, like, load testing, uh, dev ops side, uh, especially in the Q&A and testing when someone is testing, like, Black Friday workloads. 24:43 Is- do you should consider, like, business metrics, KPI metrics also in this, in this Q&A, in this QA, uh, phase? Yes and no at the same time. Okay. I think that 24:58 testing is important, of any kind of testing- Uh-huh... including load testing or performance testing. It's, it's definitely important, but the only testing that really, really, really matters is testing in production. 25:12 Okay. That kind of... Until you reach production, you don't really know- Mm-hmm... because your tests are not... will never be able to simulate the real scale. 25:21 Your tests will never be able to fully simulate the randomness of your users and so on and so forth, right? 25:29 Um, I, I'm definitely in favor of test- I'm a testing freak, so I, I definitely think that everybody should be testing in any sort of way. Mm-hmm. 25:38 But what is m- just as important or more important is testing in production. And when I say testing in production- Yeah... uh, I don't mean, "Hey, uh, let's run our unit tests in production or load test in production." 25:53 What I really mean is that do you... Is your observability ready for that, right? Mm-hmm. And can you really see what's going on? Can you predict what will be happening based on production? 26:10 And ultimately, can you automate the actions behind those predictions, right? Uh, what you really want to accomplish, of course, you should be testing what will happen d- during Black Friday before Black Friday, right? 26:25 Yeah. But the ultimate goal is for your observability and your automation that uses those sa- that same observability to be able to react, right? Yeah. It's, it would be silly to say... 26:39 Imagine if Amazon would say, "Hey, we predict that, uh, for Black Friday, we will need million servers," right? Mm-hmm. "Let's spin up million servers and have them running just in case." Yeah. 26:53 That would make them bankrupt, right? Yeah. 26:55 What they do have to do is that they tested, uh, tested things that are not necessarily related directly with Black Friday, kind of, okay, um, how many, what is the number, how many nodes can we bring up online if the traffic increases, and how much does it take to do that, right? 27:16 Mm-hmm. Oh, we can, we can handle up to 100 new nodes with everything running, new replicas running in those nodes within, I dunno, five minutes' time, right? Yep. If that's what you tested, you're ready, right? 27:30 I mean- Yeah... th- that's not the only thing, just to be clear, but- No, no, no... kind of. Basically, you want to, you want to observe what's going on. Not you, sorry, not you. 27:41 You want machines to observe what's going on and be able to act accordingly- Yeah... uh, on any situation that imaginable, right? I mean- Mm... it's never 100% like that. You can never- Mm... 27:53 predict everything, but that's the goal, really. No, but that, that's a great point. You want to see that your environment can scale up and eventually, uh, uh, make it efficient 28:04 and be, um, and make sure that the production is ready for at least as much as possible, uh, is ready for the, for the real thing, right? 28:13 Because a- as you mentioned, uh, you cannot match it 100%, and even if you could, it would [laughs] it would make you bankrupt, even if you're Amazon. Uh- Mm... thi- this is interesting and, and interesting view. 28:27 And it's, it's also a calculation, right? Uh, it's easy to make something that is the most performant, right? Yeah. And what I said before, let me spin up million nodes and stuff like that. 28:40 It's also about the calculation. Okay, I want my users to have as good experience as possible, but not at any cost, right? Mm-hmm. So I w- I would make a calculation. 28:53 Okay, so yeah, I'm going to start spinning up nodes, not in advance, but maybe after I see that the traffic increases, and that will maybe make a, some percentage of users have slightly worse experience. 29:08 But that reduction of potential experience and the cost is maybe lower than, than, than the cost I would need to- Mm... pay for and so on and so forth, right? It's, the easy thing is to make it perfect. 29:24 Not, not perfect, but kind of to make it, hey, everybody has great experience. Yeah. But I need to balance that cost versus return of investment, right? Yeah. 29:35 You cannot either way to have 100% CPU consumption, but you cannot do it at 20%, of course. [laughs] Exactly. Exactly. 29:42 So some, l- at least in Kubernetes, I feel that it of course depends on the size, but I would probably aim for maybe 80, 90% resource consumption, kind of- Yeah... 29:54 that gives me a certain freedom without going crazy and saying, "Oh, 80% available for whatever happens." No, 10, 20% is the reasonable, uh, reasonable- Yeah... 30:05 amount of available resources that you can say, okay, if things start going up, I have, I have just enough resources- Time... available to keep me going until new resources are brought, brought to the cluster, right? 30:21 Yeah. And also very important that you should also consider the lower boundary for, uh, the down scale. [laughs] Oh, yeah. That, that, that's usually what people forget, kind of. [laughs] "Oh, we were very successful. 30:31 We managed to survive Black Fri- Friday because we were scaling up." And then a week later kind of, "But did we scale down?" [laughs] That totally happens. 30:41 Like, everyone, everyone uses the, the HPA to scale up, and nobody put the, the do- the cool down stuff. It's like y- there are two parts in scaling. 30:49 One, it's, uh, scaling up, but then you also have to scale, uh, down your replicas because otherwise if you are spending the same ti- the same amount, uh, in Black Friday than, I don't know, a week after, your, your cost per user will be definitely a problem. 31:05 Yeah. You know, Black Friday, s- Friday, big success. Survive. [laughs] Weeks after the Bl- Black Friday, we went bankrupt. [laughs] Yeah. That's a great thing. 31:14 You know, we were talking with Guillermo the other day, and he woke up- Yeah... and we were talking about the same thing. Where is the limit or where is that line that you say, "Okay, this is enough." Okay? 31:24 And then thi-this user experience is enough. We don't need to be perfect, as you said. We're just giving- Yeah... this enough for the user to, to continue to operate and the system to operate, yeah? 31:34 And the user will be, be okay. Eventually, this is only at peak times. It's not all the time. Mm-hmm. Yeah. You know, that's the same thing, like, when we talk about availability in general. Mm-hmm. 31:48 Most of the time we talk about how many nines you have, uh, 99 point- Yeah... and then how many nines. Nobody ever... No, no sane person or experienced person ever talks about 100. No. There is no 32:01 advanced company that tries to make it 100, because simply there is a li- there, there is a point after which the cost is simply too high. Yeah. It's not worth it anymore. Yeah, yeah, for sure. 32:16 And I mean, the, you only try to put another nine after the, the last nine, and that's it. [laughs] That's what you, what you try to do. 32:23 So, and, and continuing with this, like, because we discussed a lot of topics and a lot of, uh, practices, for someone that is maybe, you know, having a, a cluster, you know, a, a standard cluster stuff, it's alwa- always running for Q&A, dev, and, um, production, uh, what do you think could be a practices to improve, uh, the resource of, uh, usage and the resource optimization in general? 32:48 Something that you have seen in the industry or something that you would do for a, you know, a standard use case. 32:54 I'll, I'll give you the, the answer what I've seen and what I haven't seen, and I am surprised that I haven't. Um, what I've seen is, is more or less the topics we talked, right? Kind of- Mm-hmm... 33:05 if mainly around ephemeral, uh, usage of- Okay... everything, and about right-sizing, uh, stuff, right? Mm-hmm. And scaling, of course, right? If you, if you combine those three things, uh, you are going to be good. 33:22 Okay, right. Mm, yep. The, but the thing I'm not seeing, and I'm honestly surprised, uh, is usage of cloud providers that are not the big three, right? Okay. 33:36 So we have AWS, Azure, Google Cloud, and I do think that for anybody but small companies, they are the right choices because there are many things that they have that other providers don't. Mm-hmm. 33:48 Uh, like for example, almost no other provider has three availability zones in the same region, right? Yep. And without three availability zones, I cannot make it, um, 33:59 I cannot make it highly available, and it's normal that they have it and smaller ones don't. But many of the reasons why we want to use AWS or Google Cloud or Azure are not the same, uh, 34:13 for production as non-production, right? Um, people can, can really get amazing results in non-production by using, uh, Civo or DigitalOcean or Linode or Hetzner or any of those, right? Mm-hmm. Now, 34:32 in the past, that, that was not a good option simply because the, the effort and the work that we need to put to make same thing running in- Yeah... very different places was too high. Because, hey, if I'm... 34:49 Let's say that I'm running on nodes and what so not, running in AWS and in DigitalOcean is very, very different, right? The results are going to be different. The, the processes are going to be different. 35:00 The automation is going to be different. Everything is different. But now we have a standard, and that standard is Kubernetes. Yep. Right? If I... 35:09 I, now I know that Kubernetes in DigitalOcean and Kubernetes in AWS are not the same. I know that, but they're 99% the same, right? It's good enough, 35:21 and I'm not sure why, at least I don't see combinations of those, kind of, hey, let's go, let's go over there for production, and let's go somewhere else, but it's much cheaper, and at the same time, also faster in many ways. 35:36 Um- Mm-hmm... uh, if you spin up a node in Civo, you can spin up a cluster in literally seconds. Yeah. While in AWS, it, it can take, like, 20 minutes. Yes. 35:47 And 20 minutes is not too much if I'm setting up production because I'm not destroying and creating new production every five minutes. I don't care, right, how much it takes. Right. 35:57 But when I'm spinning up ephemeral environments, I, I do care how much it takes, 'cause I don't want people to wait. 36:04 So they're cheaper and faster, not necessarily as reliable, but reliable enough for, you know, development or pull requests. Yep. I totally agree with the, with the ephemeral system. And do you think... 36:18 So regarding that, do you think that the overhead, or when do you would start doing that because of the overhead that it would take for switching to a new Kubernetes provider? 36:29 I, I know you said that it now is less, but there is still, um, a little bit of an overhead, especially with the connections. I would say that it could be more problematic to the new providers and stuff. 36:40 Connections between applications or? 36:42 On the, so the storing the connection credentials for the cluster, um, all the setup for, for the CI/CD maybe, um, that, that kind of switches that could change from one environment, let's say one provider to, to the otherI mean, there is always work involved, but- Mm-hmm... 36:59 it's much, much smaller than before. Let's say- Yeah... I'm, I'll, I'll take one of the examples, right? You said, uh, keys to deploy stuff or something like that, right? Yeah. Yeah. We use Argo CD these days, right? 37:10 Yeah. And I just need to have Argo CD pointing to the repo. That's it. Oh, makes sense. Yeah. Kind of like- Yeah. That's a good point... I have a cluster over there. 37:17 I have Argo CD in that digital ocean, whatever it is, pointing to the repo. I just need to... I mean, yeah, it will take me time to set it up, but, uh- No, but, but you're right... 37:28 an hour, kind of like not a big deal, right? No. No, it doesn't. 37:33 Like, as you say, I didn't, I hadn't thought about the Argo CD for this, for this scenario, but I think it makes a lot of sense to switching for, for GitOps, let's say, um, to- 'Cause-... to do this stuff... 37:44 you know, that's, uh, security is very often the, the thing that people overlook and probably the, one of the biggest reasons for me for adopting GitOps. Mm-hmm. Um, and is that it is a pull-based mechanism, right? Yeah. 38:00 I, I don't want, I don't, I don't want my processes to, to go to a cluster and do something, because then I need to deal with keys that you mentioned before. Uh- Yeah... 38:10 and you were probably mentioning keys from the perspective of, uh, the effort needed to, to do it, but there is also security part of the story. Pull-based mechanism simply works so much better, right? Uh- Mm-hmm... 38:23 I just need... I have my pipelines that are building images and pushing them to registry and doing whatever- Yeah... they're doing, and when the time comes, they push to Git, and that's it. Yep. Yeah, yeah, yeah. 38:34 No, no, I totally agree that for this specific scenario, and now that people are doing more multi-cloud stuff and testing out new things, um, the GitOps approach can be so much efficient for, for, for these scenarios. 38:49 Yeah, I didn't fall into that. I was more focusing like, because since we-- I've been doing like CI/CD all my life, I, I am now adopting the, the GitOps. 38:57 Uh, recently I was always like, "Yeah, these secrets are driving me crazy." Where [laughs]... where I have to put it. I have to set it up to a new provider. The new connection is like, "Man, what's going on?" 39:06 [laughs] Yeah, secrets are driving everybody crazy. [laughs] Yeah. You're not the one, the only one. Yeah. But even secrets, right? Uh- Mm-hmm... so, you know, 39:15 there are so many tiny advancements that, you know, when you combine them, they make such a huge difference. Like, secrets was a pain before. Mm-hmm. But now we have, for example, external secrets operator. Yeah. 39:28 That you just need to deploy it in a cluster, and it will be fetching secrets from wherever secrets are stored, and doesn't matter whether that, uh, ESO, uh, operator is, is in this cluster or that cluster. 39:41 You just need to install it. Takes few minutes. Cool. You need to give him credentials to go to your secret store- Yeah... whatever you're using, and you're done, right? No, and- And secrets will automatically appear. 39:54 You don't need to worry about them. They will just appear out of thin air. Yeah. 39:59 And also with SOPS and KSOPS and all this stuff that you can also, like, uh, encrypt all the, the secrets within your code because it's already encrypted, and then you can decrypt it on the fly when you deploy. 40:11 I think we, we have found that solution as well to be one of the most effective, and nobody is, like, fighting again, like, where this is stored, uh, where the, this addresses. 40:20 You just, uh, encrypt it, leave it up there, and then the, the pipeline or Argo CD will do the, will do the work for you. [laughs] Exactly. Yeah. There, there is that on us. This- Mm-hmm? Just not to be misunderstood. 40:33 Yeah. I might sound like I'm saying that it, it's no work involved. No, no, no. There is always work involved. 40:37 I'm just saying that we got so many improvements, especially on standardization level, that, yeah, there is work, but it's infinitely smaller than 20 years ago. Like, I'm an old dude. I kind of like- [laughs]... 40:49 you know, I, I lived in an era, I, I don't know whether maybe Damian lived in that era- [laughs]... 40:55 when we would go over Friday night and, uh, sleep in the office for two days because it took whole weekend just to deploy a new release. Yeah. You know, like, uh, we're past that. [laughs] Yeah. 41:10 Now the, the times to, to deploy are much, much less, and also the, you know, the, the watch times, like, the availability is way easier to, to achieve, definitely. Oh, yeah. 41:22 One, one more thing that I wanted to ask you, Viktor, is, um, from, from what are you looking to every day? I mean, do you work with the FinOps? 41:30 Do you work on dashboard, any visibility, any things that you're looking into? Or, or do you have your own f- your own way of looking into usability and cost? So, um, I'm not, 41:50 I'm not much involved in, in whatever has the cost as a bird. Mm-hmm. That's not my focus. Mm-hmm. Uh, but I am, I'm very much involved in, in things that, uh, are about optimizations, right? Mm-hmm. Um, which I feel 42:08 end up having a lot of overlap, right? Yeah. I am very much involved into scaling, uh, observability and so on and so forth, and 42:19 I might be completely wrong because, as said, FinOps is not my area, but I feel that those are very, very overlapping. No, for sure. 42:27 We, we can tell you that, that, that those are FinOps related, and that's hugely impacting the, the FinOps practices because, as I told you, Kubernetes is one of the, the hottest topics, and scaling down and scaling up, optimizing workloads- Right... 42:41 um, it's, it's a core thing to, to- Oh, way-... this stuff... you know, whatever we were scaling until recently is nothing to the scaling that is coming now with people, companies training models. Now we are- Yeah... 42:54 jumping from hundreds to thousands or tens of thousandsOf, of nodes and processes and what so not And EBOs. 43:02 Now it's going to be go crazy Because now we are, we are going to be in a whole new metric, of course, which is the EBO consumption that often is forgotten, especially on Kubernetes, like you always focus on CPU memory stuff. 43:17 But now it's like distributed training, um, to a huge scale and the amount of time and resources, because I've been, you know, I, I was working on AI on 2018, which nowhere close to what we are reaching right now, and there was still a lot of consumption. 43:36 I cannot imagine the amount of money that could be invested now in... 43:41 Well, I said millions in a train, like in a train round for a model is like millions of dollars, so the, what we are going to be now is going to be crazy, especially if Kubernetes starts with the GPO, which is already there. 43:53 [laughs] Now, you know, all those people that were annoyed, that were not working in AI and were annoyed by you saying, "Hey, you should optimize this. You should do this," now they're fine. 44:05 Now nobody's going to ask them anything because now their expenses are gonna be nothing compared to- [laughs]... to expenses of the data scientists. Now, uh, now you're fine. You don't have to do anything. 44:16 Nobody cares about you anymore. Now we need to go after those guys. Yeah. And now, now we are seeing it at this, at Rendow as well, like there is a, a duality in FinOps now that is AI for FinOps or FinOps for AI. 44:29 Like what do you do? Uh, do you start using AI to, to your FinOps tools or to your FinOps practices? Or 44:37 you, you start doing your w- or the other part that is you do FinOps for your, your, your AI, you're trying to improve your prompts, you're trying to improve your training, you're trying to improve your RAG systems, how much, uh, each user costs in terms of the usage of the AI, and that's another whole level that is going to be there because AI is going to be 44:57 a huge part of the IT budget in the next few years. Yes. Oh, yeah. Well, there's both of them, right? [laughs] Yeah, we will see both of them. FinOps for AI and AI for FinOps. For FinOps. 45:09 And let's see, because now people are starting to do agents, which it seems to be working in the US and, you know, everything that works in the US that's going to be like all in our YouTube feed every now and then, so- [laughs]... 45:21 it's going to happen. 45:23 Uh, but yeah, um, that's going to be, that's going to be a whole new level to, to optimization and, you know, also reliability because, you know, to keep that training running, um, is not easy to have that amount of information running and available and training and not having any incident while training. 45:41 Oh, yeah. Uh, it's going to be a nightmare. I mean, it already is. [laughs] Yeah, indeed, especially- Which is good. We all get to keep our jobs. Great. Yeah. [laughs] Um, that's great, man. 45:53 No, I, I think it's a, it's a great point to, uh, to close the tab. So, um, thanks a lot for, for having today's call. It's been a pleasure. 46:03 And for those that, uh, don't know you, where you can say where, where they can find you and, you know, what stuff do you do online because it's, it's someone to check on, uh, if you haven't. Definitely. 46:12 If you're not following Victor- I do many things... start doing it now. [laughs] Yeah. I, I do many things. I have a YouTube channel. 46:20 I, I, uh, uh, there is a podcast that a colleague, ex-colleague of mine is doing where I'm always, I speak in conference. And I... Just Google my name. It's- Yeah, definitely... Google my name. You'll find me, I promise. 46:31 [laughs] Yeah. I, I can tell you you could find him. Um, so yeah, Victor, thanks, thanks a lot for, for being here today. 46:38 It was a pleasure talking to you from, from the DevOps perspective, and also from, from FinOps to DevOps with Damian, and from DevOps to DevOps. I think it was an amazing talk, man. [laughs] So thanks for coming by. 46:50 Thank you for having me. Thank you very much, Victor. Okay. And see you next time