Transcript 0:00 You read the book, you pass the certification, but your cloud is still a mess. Today, we are leaving the classroom, and we are entering the FinOps war room. 0:07 We have Olfa from Azure Cloud Academy to teach us why FinOps implementation normally takes three times longer than you plan. 0:15 You will learn how to stop drowning in cost alerts and the exact moment you should and when you should not automate a fix. If you are tired of building dashboards and alerts that nobody looks at, listen. 0:27 We are going to start right off the bat with, with a question. So for you, Olfa, what is the biggest gap between the, the FinOps theory and what really happens in, in production? Yes. Actually, FinOps, it's a framework. 0:41 It's a lot of practice and culture practice. It- Mm-hmm... uh, ensure to maximize the business value of using the cloud technology. Mm-hmm. And it mainly have... 0:55 has some principles, like six principles, and one of the principle I see a gap, important gap between the theory and the real production world is the business value, how the business value drives the technology decisions. 1:12 Mm-hmm. And here there are issue on the knowledge spread, what I see actually in, what I saw in different context of the implementation of the technology, 1:22 uh, is issue on the knowledge spread for the business value when implementing something. 1:30 In agile team, we implement incremental digital product, and team doesn't, does not have always the idea on the business value of the feature they are implementing. 1:45 Generally, this input comes from the product owner or the product team, and there is generally a ratio between the, the business value and the risk of putting this feature in production. 2:01 Generally, this ratio is not known in advance, let's say. So this one of the thing that makes a big gap between the theory and the production. Yeah, that's, that's a great insight. 2:13 I, I also believe that, you know, the product connection to the business value and, you know- Yes... 2:18 making it real happen, it's, it's a very difficult thing to, to do, and it's a real gap between, you know, the FinOps, like theory and what really happens in production. 2:27 There is a lot, let's say a long road between this idea and the, and the implementation. Yeah. It, it, it's one of the gap... 2:33 It's one of the gaps that can, that we can highlight between the theory and the, the real world. 2:39 Uh, but it's, in my point of view, is one of the biggest gap because project team, let's say, they implement a feature, but do they really know the business value of this feature? It's not always the, the case. 2:55 Y- yeah, generally to avoid really implementing things that with low business value is really to have, um, to question and let's say continuously asking for the, this ratio between what's the value versus the risk if I'm upgrading the digital product with this function. 3:18 Yeah. 3:19 That's, that's interesting and, and another question regarding like the, the product side is, you know, for you, uh, what are the aspects of, of a FinOps implementation that normally takes like two times, three times longer than, than the expected, uh, time to implement? 3:35 Yeah. There is from aspects which take consistently longer than what we expect is the ownership, how we assign ownership and how engineers feel it regarding to what they generally used to do. 3:52 Engineers generally, they do not feel the cost as an ownership. 3:56 They feel like we need to deliver something functionally speaking, working, uh, but d- they generally do not have this cost ownership because they say, "Yeah, if it works well and I can deploy it smoothly, so even if it costs twice time what it should, uh, cost, so it's not my problem," let's say. 4:22 [chuckles] Mm-hmm. It's more financial, uh, department problem in big, uh, organization, uh, generally. 4:29 This aspect to having assi- to assign the, the cost ownership to engineers is one of cultural aspect that implementation should, uh, generally lead and also changing the habits. 4:44 Yeah, we're all human, so we use to have a comfort zone and we used to do some tasks, and we are quite comfortable to do tasks, engineering, development, testing, et cetera. 4:56 But changing some habits to integrate this FinOps culture, taking some time actually. Mm-hmm. And so 5:07 this be- because this is, it's always a continuous enable, e-enablement, and it's not just one setup and we do not do other actions after. So we continuously need to do actions and, uh, to enable them. Yeah. Definitely. 5:24 Like it's, it's a continuous implementation movement and you don't need to, uh, you know, it's not a one-time thing, definitely. 5:30 And, you know, moving on to, to one of the things that is mostly common in, in FinOps, which is the, the anomaly. 5:37 So we have talked about an engineering side and we, especially I come from, from the engineering side on the observability. 5:43 We have a lot of anomalies, especially for, you know, something going down and for, for availability side, right? But FinOps also introduces the, the concept of, of anomalies for, for the cause, right? 5:55 So, you know, talking about the alert fatigue is, you know, how do you balance, uh, the sensitivity and being accurate in the, in the alerts with the alert fatigue that causes the, the anomaly detection?Yes. 6:08 One of the root cause for the anomalies is issue on provisioning. I-in a different context, we don't have always automatic provisioning templates or some parts are still done manually, and error is human. 6:24 We say to, for example, do a retention log, uh, duration or having a mistake instead of putting ninety days, for example, putting more, I don't know, adding zero [chuckles] in, in some duration- Mm-hmm... 6:41 which can happen if you do provisioning manually. We can also see the auto-scaling issue sometimes when working in, in strategy, in a cluster provisioning, for example, if the strategy is not defined and why... 6:58 w-when should it scale. Um, so this is also another issue that can happen. It can, for example, scale, scale in a test environment, and we should not have this scale. 7:11 Test environment is only done for typically continuous integration. Mm-hmm. 7:17 Also, another issue which is quite common is the fact of not deleting properly the orphans resources or, yeah, the resources are not used anymore, but they were not deleted c-correctly. 7:33 And most of anomalies detected generally eighty percent, like more than, even more than eighty percent are a configuration issue. This is to recap the three common root causes. Mm-hmm. 7:46 Yeah, and that's definitely like, uh, three, three of the typical enemies of, of any FinOps and any engineering in the practice. 7:55 And, you know, regarding the, these alerts, so they normally like happen frequently, so, uh, there is the issue of the, of the sensitivity. So how do you adjust the sensitivity for this anomaly detection? 8:07 Really, yeah, how to make we ask ourself when you're doing alert and dashboards, how to make a balance, uh, and do not spam, let's say the- Yeah... alerting people for something not pertinent. 8:21 So generally, we should have a method, organized method to start with the basic things and then refine, 8:29 make something basic, for example, monthly alert, and then refine, do daily, hourly, or per service or pertinent depending on what things you want to show, but refine, better refine the alerts. 8:45 Also, avoid the false positive. For example, we can have an alert for the day of a promotion day, and we made an alert, but it's quite positive. 8:56 So generally, to avoid that, we should configure the alert to have some absolute threshold plus a percentage, uh, meaning if we, uh, use for a period to have double traffic for the this scope, and so in this case, we should configure the alert to have this absolute threshold plus, uh, ten percent from, 9:22 um, yeah, if it increases with a per-percentage that we define, that should really be the alert threshold. And yeah, also that's quite what I said, uh, 9:37 uh, avoid making alerts during a period or we expect to have something different. 9:44 For example, the deployment phase, we expect to have downgraded, for example, CPU because some-something is, uh, we are, we're trying to upscale, so we 9:58 generally avoid to trigger alerts during the deployment phase because resources are... It's not relevant. And yeah, that's generally the things to, to see. We say generally not seen alert- Mm-hmm... 10:16 is more dangerous than a not defined one. Mm-hmm. Sure. 10:23 Because also with the automatic, the some mechanism of cloud, even in Azure, some alerts trigger some actions actually, and if, if they are not relevant, then also actions triggered, uh, could lead to big issue. 10:42 Definitely. Definitely. If you, if you are not, you know, handling metrics, it's definitely a, a problem. 10:48 And, and you know, to, to, you know, to, to work on the, on the anomalies and to, to fix that, uh, you know, what are the, the fastest remediation like wins that people- Mm-hmm... or the teams can, can implement? Yeah. 11:03 So definitely the fastest remediation is the automatic shutdown for the non-production instances. Mm-hmm. Uh, so we know these VMs are using only for QA activity or some performance B, uh, test. 11:19 The rule is to do the automatic shutdown on the weekend and do define office hours rules. This helps to save from twenty to forty percent from the cost. Yeah. Just, a-and it's quite fa- It's the fastest remediation. 11:36 Definitely. Simple. [chuckles] Very simple. It's simple, and, and you can save a lot. If, if you can afford to, you know, you know, have like twenty-four-seven, uh, availability requirements- Yeah... 11:46 and all of that, then I think it's like, uh, a good one. And, you know, talking about we just talked about what we should do, then what we shouldn't do, like what we shouldn't- Yeah... 11:57 do not automate the, the remediation. Yeah. If, for example, we don't know the context, it's doing automation without context. 12:06 We don't know what's the business case for this client.And we are trying to do automation for delivery process, for example, or let's say, yeah, trying to build a new metric, but we don't know actually the context, uh, or the business case. 12:23 In this case, yeah, it will leads to ineffici-ineff-inefficiency. Yeah. That's, that's totally true. And I think, like, it, it can inefficiently, like, you, you can totally- Yeah... 12:36 uh, avoid with, with a simple, you know, with that simple consideration, right? Ye- yes. It doesn't make sense actually to do, uh, this automation. 12:45 Imagine we, I don't know, we have a remediation on config rules or a user permission set to remediate something automatically without knowing what's the consequence of that. Mm-hmm. 13:01 So it, it will leads to, sometimes to correct it, it's takes more time to, than to s- Yeah... to set it up. Yeah. Yeah. [laughs] Definitely. You, you don't have... 13:11 You don't want to spend more time fixing automations that the time that is going to, to save you, you can apply that on, on everything. 13:18 And, you know, talking about the, you know, the, the problems with automations and, and the trust is like, how do you build trust for, for the engineering teams and the people to, for the automated remediation to trust? 13:32 Like, how do you build that trust for automated remediation? Yeah. Generally, the first thing is really to, uh, put the action to the service owner and not to the, to someone, I mean, in the financial or C- CFO. 13:50 So each action to the specific service owner, so this will give the ownership to the people. 13:59 And, uh, generally, we also need to add an automate- automation on the rollback in remediation, always thinking that remediation can lead to failure because there is a big drift, for example, between two deployments or I don't know what can be the issue depending on the context. 14:21 So here we need also to al- always think about an automation of the rollback in case the remediation doesn't work, and also to show always the saving evidence between the state before the remediation and after. Yeah. 14:38 And here we can build the trust. Yeah. That's, that's a good advice. 14:42 I, I think that, you know, building trust and also with, you know, having good policy and, you know, explaining it as well to, to the engineering teams and they being able to understand it and to, to see the value of that and see the process because sometimes, like, you have an automation and you don't get it and, like, what this is doing and, you know, if you explain it, what the process, what's the logic behind, and they will probably get some, some buy-in to, to, [laughs] 15:07 to that, uh, implementation, right? Yeah. And when you realize that it's saving you, uh, work time rather than, you know, giving you more, [laughs] then you- Yeah. [laughs]... you are automatically buy it in. 15:18 And, uh, you know, talking about, uh, the FinOps practice and, you know, building trust and all of that, so in your perspective, like, what is one thing that everyone or most of people consider in FinOps that do you think is like a, a waste of time or something that is not very, very efficient in, in the time space? 15:38 Yeah. What I see commonly happening is in the dashboard and observability, we have too many metrics, and doesn't help to have con- consistent decision. Mm-hmm. 15:48 So in many metrics, like error rates for this API, CPU usage, a lot of metrics shown with a lot of data without what's do, what's to do next actually. 16:04 And there is no ownership mapped because generally if we deploy resources and there is no tag for the owner, so we don't know who did what. So 16:18 to not have this ownership mapping also is a big mistake. Yeah, the fact of not having the context, there is something shown in the cost. 16:27 We know that this deployment, for example, upgrade this cost, but we don't know for which context, for which client in case we have a multi-tenant, and why, what was the feature who, which, I mean, introduced this cost gap. 16:47 Yeah. That's, that's also true. Like, I, I think also that, you know, metric need to be with context as well. Like, they need to mean something, and you can get the insights and, you know, like tons of metric. 16:58 Like, for example, if you check like Google Analytics, you have like thousands and thousands of, of values and metrics that, to be honest, if you're like there, uh, you're like you don't understand anything unless- What of these metrics, uh, if there is no do next, what to do next information, so actually you can see how many good paths, but you don't know the decision to do if we have some spike, some lot of data, but no quick decision next plan to understand in this. 17:28 Definitely. 17:28 And, you know, talking about, you know, you, you gave good advice on, on this, but if you had someone, you know, starting their, their FinOps journey, uh, today, and you can give, uh, them one piece of advice that are listening to us, what, what would it be? 17:44 Good question. Yes. Generally, someone who trying to understand and practice FinOps, he always need to keep in mind it's a practice. It's not a tool. It's not a set of tool. Mm-hmm. 17:53 So the point here is to continuously practice this enablement and being aware of the cost consequence on what we are developing, and not just, uh, I'm good in this tool, in that and that tool. 18:12 It's not a set of tools. It's a practice. This is the main point to keep in mind. Yeah. I think that's a good point to, to keep in mind. You, you need to be, you know, aware of the practice. 18:24 You need to implement it, but, you know, need to, to think about, uh, you know, dominating, uh, the, the whole thing and the whole stuff. It's, it's a practice, and you need to... You know, it's an ongoing, uh, thing. 18:36 If you are liking this episode, don't forget to check this automation special episode podcast with Ernesto and Austin so you can know more about how to automate your anomalies. 18:46 And I think with that, we can, uh, finish the, today's episode, so thanks. Thanks for being here today. Thank you. Thank you, Victor.