[00:00:00] Mark: He made a mistake. And consequently the switch went down and the, we lost the whole data center and that’s, the data center had all our, a bunch of homegrown applications. We did all our order processing, you know, on a mainframe in those days. So the company went down, basically the company was offline for several hours.
Introduction
[00:00:18] Jonathan Crowe: Hello, and welcome. Please come in. Join me. I’m Jonathan Crowe, Director of Community at NinjaOne and this, this is IT Horror Stories. Brought to you by NinjaOne, the leader in automated endpoint management.
[00:00:32] Jonathan: Hello everyone and welcome back to IT Horror Stories. I’m your host, Jonathan Crowe, Director of Community at NinjaOne, and I have with me here Mark Settle. He’s a seven-time CIO, including tours of duty at Okta, IHS, BMC. It’s clear that we didn’t settle for you, Mark. You are a top tier guest. I’m so excited to have you.
[00:00:55] Also the author of several books, including, your latest, “Truth from the Valley: A Practical Primer on Future IT Management Trends.” Obviously a very timely topic with so many aspects of IT, and really digital work in general, being an active state of flux. Thank you so much for taking time and being brave to join us here on the podcast.
[00:01:16] Mark: I’ve been looking forward to it. This should be fun.
[00:01:19] Jonathan: So before we get into your horror stories, let’s talk a little bit about your background. Those seven tours of duty, you as a CIO, major Fortune 500 companies. I mentioned that phrase because I was reading an article where you described, you compared military campaigns to IT.
[00:01:37] And you said everything that goes wrong in a military campaign is a near perfect mirroring of what goes wrong in IT initiatives. So I’m curious, when you look back at those seven tours there, do you think about them in military terms. Was there a successful campaign of operations, were there moments where it was touch and go?
[00:01:57] Mark: What that analogy refers to, you know, one either projects or just really routine day-to-day activities are disrupted in some way, the things that go wrong in battles. Now, this is historical battles – the technology today is so different, you know, in the military sphere. But, you know, historically, even during World War II, breakdowns in communication, just simple bad timing.
[00:02:23] You know, there was some piece of information that if it had only been known one hour earlier, two hours earlier, [it] would’ve been able to avoid, you know, a certain kind of problem. Misallocation of resources, you know, the wrong troops, like in the wrong place at the wrong time. And again, just a very, some defeats and some great victories are just so serendipitous because of the way resources were doled out.
[00:02:46] You know, just about every failed IT project, you can go back and tell yourself, “Gee, if I had, if I had just known we were falling this much behind schedule, or you know, if I had just gone in and asked for another three contractors, you know, so I had like the resources to get the data transferred and keep the schedule together.”
[00:03:05] I mean, there’s definite analogies for sure.
[00:03:07] Jonathan: Oh, for sure. I imagine it’s, you know, it’s very easy to be an armchair, a Monday morning, battle commander. Right.
[00:03:15] Mark: And that’s the best way. Yeah.
[00:03:17] Jonathan: It makes me think of another quote too, where it’s, you know, “We don’t rise to the level of our expectations. We fall to the level of our training.” Best laid plans, for leaders of any type, right?
[00:03:27] You can plan and have a great strategy in place, but it’s still gotta come down to execution in the end. Did you ever find yourself kind of looking back and saying, “Oh, you know, we’ve pinpointed okay, here’s what went wrong. We know how to be better prepared next time.”
Improvising at lightspeed
[00:03:43] Mark: Yeah. You know, I’m glad you asked that question. Very early in my career, I worked for NASA and I was involved in some early space shuttle missions. And we had one mission where we had this radar system that was in the shuttle. The shuttle was gonna invert when it was on orbit and point the radar at the earth and take various forms of data for different scientific research purposes.
[00:04:04] And there was a planning group based out of the Jet Propulsion Lab in Pasadena, and I worked with them. You know, I wasn’t responsible for this team. I was kind of like something of an informed spectator and, you know, this was well before the days of scrum or whatever, but we had meetings every morning and every night, at the end of the workday. And they were trying to plan like when to turn the radar on and how to tilt the shuttle and all this other kind of stuff.
[00:04:30] So the minute we got up on orbit, the whole thing got screwed up because there was a problem with the relay antenna to take the data from the radar system and get it down to the ground. And so it greatly constrained the position of the bird, the shuttle itself. There’s only so many attitudes or orientations that we could put the shuttle into for the data take.
[00:04:51] So basically we took the whole data plan, threw it out the window, and then like in real time, orbit by orbit, like, okay, what do we do in the next orbit? What do we do in like, you know, two orbits from now? That moment of realization was, you know, the planning that you put into any project or activity is almost like more practice and like a mental thought exercise to try to grapple with all of the potential things that could happen or go wrong or how to react to, yeah, something that is unanticipated, you know, in the course of that project or activity. And, that was a real object lesson for me.
[00:05:25] Jonathan: Looking back to that, like the types of leaders, you know, types of military commanders or leaders of any type. Would you describe yourself as on that spectrum of going into things with everything planned out, very by the book, procedures are ironed out, you’ve taken into account, everything super detailed. Or more on the side, the side of the spectrum where you’ve built things to be a little bit more adaptable on the fly. And, and I guess, have you had to occupy both of those roles in different ways at different points in your career?
Finding your leadership style
[00:05:57] Mark: Sure. I think, you know, as you take on larger organizations, you know, just like being a core commander or an Army commander, the battles are not fought at your level at all. I mean the people that are actually doing it are a couple of tiers down. So you have to make sure you have good people.
[00:06:13] I think one thing you do learn is asking good questions. So you may not be, you know, familiar with the peculiarities or the details of how we’re gonna get the data transferred from point A to point B or the, you know, the code testing we’re gonna do in 12 different environments and, you know, tire kicking that’ll get done along the way.
[00:06:32] But, my experience personally was, you know, you develop enough intuition over time that you can ask good questions. And if people are open enough, they’ll say, you know, I’ve had people say, I don’t know the answer to that, but I know I should know the answer to that. So I need to kind of go back and do a little more homework.
[00:06:50] But the counterpoint to that is, you know, earlier in your career you are that person and you do the best job of coming up with the plan and, you know, you have to be the one to know when to let go and kind of do a real time replan.
[00:07:04] Jonathan: Were there any times when you felt afraid or a little nervous about what you’re kinda stepping into?
[00:07:13] Mark: That’s another good question. You know, when I look back on…I’ve told this to other people. I’ve gone into some jobs where I’ve just thought, this is duck soup. I, you know, I look around and I think that we’re not doing this well. I know we need to change this. I can see what some of the systemic flaws are in the way the IT function is operating, where we have some talent deficiencies, etc.
[00:07:37] And then I’ve gone into other jobs where I’ve looked in the mirror after like two weeks, and I think, why did they hire me? You know, like I don’t have, and a lot of it has to do with the context, the business context of what you’ve gotten yourself into. You know, you just kind of get into something and you go, wow, I didn’t, I don’t really know anything about how this works. You know, I thought, and I’ll give you, you know, an example is worth a thousand words. I worked at Visa for three years and when I took the job, I was younger and cockier then, and I thought, how hard can this be? Okay. You just like authorize the transaction and at night the merchant banks and the member banks just settle up with each other, you know, like everybody gets paid and the money gets moved around and then we start the whole new day with the transactions.
[00:08:22] And of course it took me close to two years to figure out like what was really happening at the end of the day. It was part, and the other thing I, that job I would say frequently is like, how did they take something that’s so simple and make this thing so complicated at the end of the day?
[00:08:38] But, yeah, I’ve had both reactions to being overwhelmed and kind of underwhelmed at the same time.
[00:08:44] Jonathan: Those seven times being CIO. After the first couple were there things that you started to look for or look out for? I mean, maybe there’s, there’s red flags and there’s green flags.Were there certain times in that where you were looking for, okay, I wanna be able to go back into a place where I can run my playbook, vs. maybe you were drawn to an opportunity where you’re able to say this is gonna be completely new to me in some ways. What was going through your mind with some of those decisions?
[00:09:15] Mark: So just to kind of pick up on the last answer, I think one of the important things and other people would recommend this is when you come into a new job, especially in like a Fortune 500 or 2000 company, it’s probably more important to get out and spend time initially with the business people and kind of understand the challenges that they’re having because your people will give you a warped impression of like what the problem is.
[00:09:40] You know, it’s like, well, we gave them this, but they’re not using it properly. Or, we tried to help them with this, but they made it more complicated and it would cost too much so we didn’t do it, or whatever. So if you look at the IT function through the eyes of your customers. You know, that’s like one of the most healthy things you can possibly do.
[00:09:56] So I definitely recommend that. And then I think a lot of the direct reports that you inherit, what’s the best way to put this…
[00:10:05] They’re kind of, they’re bought in, but they’re not bought in, right? I mean, they’re kind of waiting to see, okay, the CEO brought you in. Like, are you gonna really influence the other executives and are we gonna get more money in headcount? Like they really trust you and, so you’re kind of in the back of their mind on trial.
Growing leaders, not just teams
[00:10:24] And each one of them will say, well listen, I know more about the data standard than Settle does. I know more about the applications here than I know more about the data warehouse or whatever. So I’m completely comfortable in, you know, my little pigpen here, you know. And like he can wander around and make PR prognostications of what we’re gonna do, etc, etc.
[00:10:41] But I can fall back on this thing, on this area of, of personal knowledge. So one of the things I have found to be very helpful is to try to get those people, your direct reports, out of their comfort zone. So an example would be, you’d say to the head of the data center operations, I want you to take a 90 day leave of absence and put together our budget for next year.
[00:11:02] And you’re gonna pitch it to the executive staff when we have the budget reviews that are coming up. And typical reaction is, oh, I don’t know anything, you know that? I’m not sure I want to do that. And I say, no, it’d be good, it’d be developmental. You know, like you’d learn about the whole organization.
[00:11:19] And then they’d say, Well, I think I can still do my current job and do that too. So I don’t need to really step out for the 90 days. And then I’d say, so you’re telling me you haven’t groomed anybody that can step in for 90 days and run your organization? Like that’s, you know, your absence of, you know, career development for your direct reports.
[00:11:40] That’s kind of a win-win if they do a good job at the alternative assignment, it’s more, you know, it’s a temporary thing. It’s not a permanent change in their responsibilities. You get them out of their comfort zone, you get a better appreciation of what their, you know, their real management or leadership skills really are.
[00:11:59] And you get exposure sometimes to some real talent within their organizations too. You know, when those people start to show up. So I think that it’s important to get them out of their comfort zone and in the best possible case, they come back, you know, after the 90 days and they say, that is like the best thing that ever happened to me.
[00:12:16] You know, like I would’ve never gotten up in front of the CEO before. I would’ve never gone around and asked questions of all the C-level people to try to get their reactions to the budget before the big pitch meeting and all that kind of stuff. So, yeah.
[00:12:28] Jonathan: Let’s get into a little bit more of that and let’s get into your, the horror stories that you wanted to discuss today because I know at least one of these is related.
[00:12:37] And it falls on the people that you have to execute. You come in, you own the strategy, you’re the new leader. You’re there to guide, but in the end, the execution comes down to the people that you’re delegating to you. And I know for a lot of folks, they’re very good technical individual contributors.
[00:12:57] They get identified as being good at their jobs. They get promoted, they get put into management, delegation and letting go is a tricky thing for a lot of folks to be able to do. It’s been hard for me to do. So I think the horror story, let’s tee this one up. The first one that you wanted to talk about, an employee that’s really highly skilled, and you’ve put them in a position to execute, and this hasn’t necessarily gone to plan.
[00:13:30] Mark: I think you’re referring to a situation many years ago where I had a member of my data center team who had, a very knowledgeable guy, and during a normal working day, I think it was a Thursday, he had gone in and done a software upgrade on the main network switch in the data center. And he made a mistake.
[00:13:53] And consequently the switch went down and we lost the whole data center and the, that’s the data center had all, a bunch of homegrown applications. We did all our order processing, you know, on a mainframe in those days. So the company went down, basically the company was offline for several hours.
[00:14:08] So when the head of the data center team came to report this to me, I said, you know, I’m not the most technically oriented guy in the world, but isn’t this something that we don’t do, like on prime shift? Like is, wouldn’t we normally do this after hours? So he said, well, yeah, no, that’s, that’s our standard operating procedure.
[00:14:26] He said, but you know, this guy had a bat mitzvah to go to over the weekend, and so he thought he’d get this out of the way, you know, on a Thursday. So he’d be free over the weekend to take care of this family obligation that he had. So, yeah, so it’s also, I think one of the telling points of that story is when we talk about human error, I think we frequently just assume, oh, it must be a junior person that just doesn’t know the right thing to do.
[00:14:52] If they weren’t properly trained, they didn’t have the right checklist or whatever, and in fact, in my experience it’s really the opposite is the case. It’s some of, you know, your most highly skilled and usually like your most overworked people that inadvertently just cut a corner or have a little, you know, momentary memory lapse or something.
[00:15:15] They are so remorseful. I mean, they take such pride in their work. That they almost don’t want to come to work the next day. Because they know the whole grapevine’s going, you know? Do you know what Jonathan did yesterday? Do you know that Jonathan touched the switch?
[00:15:29] Jonathan: Absolutely.
[00:15:30] Mark: Hide under your desk, you know, type of thing. So yeah.
[00:15:34] Jonathan: Yeah. I mean, a sign of taking things seriously. I mean, I think you mentioned, this person even went as far as to maybe even offer the resignation or suggest that.
[00:15:45] Mark: Yes, I’ve had that in a couple of instances where, or I think in this particular case, this individual suggested that he forfeit his bonus for the year. He felt that badly. There was another instance. Well, basically, same kind of thing. There was another individual, at a different company, was using Jamf, as a tool to, you know, manage a lot of the Apple devices that were out there.
[00:16:10] And again, he put in the software update that just happened to corrupt all of the working devices for the software engineers in the company. So they lost two or three hours of work, like through the whole company. They were just down like to the whole thing. And again, he wanted to, he offered his resignation the next morning. He came in and just felt mortified that, you know, he had done that.
[00:16:33] Jonathan: It just makes it more relatable because where, who among us has not made a mistake. But I mean, thinking about this, this issue and what the, you know, what the postmortem is, what the corrections are, there’s people, there’s processes, there’s tools.
[00:16:48] You had the standard operating procedure in place. That was right. It just wasn’t followed. You have someone who is clearly mortified and clearly repentant about it. But what was your reaction? Do you remember? Because I mean, you have, you know, this member of a team, that you need, it’s a high performing individual.
[00:17:14] You want them to continue to be high performing. Right? You also have other members of the team who are watching now too, and seeing, you know, what’s gonna happen? This is a, he broke the rules, they are rules for a reason. This is a a difficult thing to navigate as a leader.
[00:17:32] Mark: Yes. Exactly right because you can’t kind of write it off and say, oh, you know, you get one pass. You know, like we preach accountability, but you’ve clearly done something that had a severe business impact. But we’re just gonna overlook it this time ’cause we know how smart you are or how hard you work, whatever.
[00:17:51] So you need some kind of a, I don’t know if demonstrable is a good word, some form of punishment, that is recognized as such by the other members in the group. So, you know, in those two instances, the punishment that I devised was to tell them to go home for a week and not come in. And think about, you know, the jeopardy they had put the company into and to come back and don’t ever do that like again.
[00:18:17] And, I don’t know how effective that would be. You know, in modern times, people would say, wait a minute, if I screw up, I get a week off. And it’s, you know, all sins will be forgiven. But, no in those, in those two specific instances, those individuals needed like a little cooling off period too.
[00:18:34] And I think to kind of reflect and, and you know, in some ways it was an easy pass for them. I mean, eventually they had to face their coworkers who all knew what had happened, but at least they had a week to kind of get some perspective, you know, like they could handle it better instead of coming in, having to come in.
[00:18:50] Jonathan: I mean talking about another, you know, facing fears, coming back to work and doing that after some time must take obviously overcoming that fear too. With these cases, do you recall, were these, were they able to come back and be solid contributors again?
Pushing past fears
[00:19:09] Mark: Oh yeah, totally. Oh, totally, totally professional. I mean, yeah, absolutely. They recognized that it was just kind of a momentary lapse. Well, and you know, I remember the conversation specifically with the case of the Jamf person. I think his boss, you know, he was two or three levels away from me. So his, the boss came in, and we’re talking about what are we gonna do.
[00:19:31] And the boss says, I don’t think we could find another guy that knows this much about Jamf. Like if he goes, you know, like, it’s gonna be harder on us than it’s gonna be on him. He can go out and find a job tomorrow morning because he is, he’s got this high level of expertise, right. But, we’re gonna be stuck here, kinda sucking wind, to try to find a replacement.
[00:19:49] So I don’t, I don’t want to claim that that was the justification or reason for the, you know, the final punishment necessarily. But yeah, they were, yeah, they were totally professional. They didn’t. Neither one had any kind of a long-term negative reaction, you know, whatsoever.
[00:20:04] Jonathan: You know what’s interesting to me is, the comment that you made earlier on about, you know, a lot of people think about the junior folks making mistakes. It’s because of lack of knowledge, lack of training. But in your instances, and these are great examples, right? The, it’s highly skilled people… you know, you’re building your team.
[00:20:24] These are some of your top assets. And here they are, they’ve brought down the company. They’ve done, sometimes they do the most damage. You know, they’re the most skilled and the most, they have the most capability, and that can obviously be used for good, or when it’s a mistake, it can really impact things.
[00:20:41] So, did that change the way that you’ve operated with higher skilled level people? Do you put more checks in places in some cases, or anything that made you kind of change your approach there?
[00:20:53] Mark: Well, it’s funny you mentioned that, you asked that question ’cause we don’t talk about it as much anymore. But there used to be a lot of reference to the hero culture in IT. You know what the, there’s, I think Steve Jobs said it’s more fun to be a pirate than to be a member of the Navy, right? So in IT, it’s like, you know, I want to be known as the gun slinger. When things go wrong in that data center, I will wade in and I’ll get that thing fixed, right? I mean, I don’t. You know, process, what process? You know, like I know everything like the back of hand. And you know, if you look at the whole, oh my Lord, decades of investment of IT professional horsepower into so-called IT service management (ITSM) practices, idle practices that have been out there for a long time.
[00:21:43] It’s all about building more of a process driven kind of an organization. I mean, even in the Jamf case, the process would’ve been to do some testing of the upgrade on a very small subset of Apple machines before we rolled it out to the whole community. That would’ve been like standard procedure for whatever reason that individual thought there was more good to, well the it – either it was overkill to do the testing or it was gonna take too much time, or, you know, just wasn’t needed or he had too many other things to do.
[00:22:10] This was just gonna allow him to check the box and move on to his next task. So yeah. But yeah, so I think what you learn is to like go back and double down on process. And I’ve written a little bit about this too. You know, process is really hard. Because if you’re not careful, you’ve either over engineered or under engineered the thing. You’ve either, you know, made it so bureaucratic and convoluted that people just blow it off and don’t even follow it, you know?
[00:22:38] Or it’s so primitive that it just can’t comprehend some of the situations that you get yourself into. So it’s like, and very few organizations are really good at, you know, constantly kind of refining, modifying, updating processes. I’ve seen some of the best work around that, around project management. Like, you know, getting approval to start projects.
[00:22:57] Do we really need to take a project that’s gonna take six man months and subject that to 12 documents and five levels of sign off and, you know, three financial analyses and stuff like that. So.
[00:23:11] Jonathan: You know, this, thinking back to the heroes, right? I think about the Phoenix Project, that book. And I think I, there was definitely, the one engineer there, I believe his name was Brent. And it was, and it’s funny because, I mean, he was obviously – so much was revolving around him and it was so much was riding on him and on his shoulders.
[00:23:38] And, but the second – it just creates this kind of a brittle system with all this potential for horror stories. Where’s Brent? What happened to Brent? You know, Brent was sleep deprived. Brent made a mistake, and now it’s all gone down and nothing’s been documented. No one can repeat what he’s done.
[00:23:54] So it definitely resonates and makes sense that the antidote to that is process.
[00:24:03] Mark: You can overdo it as well. You can overkill the thing to the point where people just are really not even following the process. Like it exists, you can go find the docents, you can, maybe people were trained or whatever. I go back to the project management situation or scenario, where sometimes, you know, the whole justification process and planning process and whatever just gets over-engineered for two things.
[00:24:28] One, the amount of work that really is required or the risk exposure of the, you know, if you screw it up, what’s the real business impact? It’s gonna cost a little more money. It’s gonna slip in time or whatever. Yeah. But if you have this big standard, you know, 12 step process or 25 step process, you put people through, then they just don’t even come in.
[00:24:47] They just go do it on their own. You know, they say, well, we’re not gonna, we’ll never tell you.
[00:24:51] Jonathan: Is there something with, as a leader wanting to ride that line between, you know, having your people who are able to follow the checklist. Follow the process versus going back to the very beginning of our conversation. Another one of the horror stories that you mentioned wanting to tell was talking about making that point that, you know, a lot of success ends up being serendipitous.
Bracing for the unexpected
[00:25:14] And then, a lot of the failures come from things that, okay, yes, you know them in hindsight, but at the time, I mean, you were doing everything right. It was unexpected, otherwise it wouldn’t have happened. And so training, or enabling people to develop the skill of being resilient, being able to adjust and adapt on the fly.
[00:25:35] And one way, those two things kind of seem at odds, but is there anything in your experience that you’ve, you know, have you been able to operationalize that a little bit or is there a way that you kind of try to instill that ability in your teams?
[00:25:52] Mark: It’s interesting if you look at the security area. So the risk is so great in security that you do do that. Like you have red teaming exercises, you hire people that basically try to pretend to be the bad guys and test your team and your technology and your processes, etc, etc.
[00:26:11] And I guess I could argue like if you look at your application support team, and they’re maintaining Workday and NetSuite and ServiceNow, Salesforce, etc. You know, there’s less of that kind of, war gaming kind of an exercise that would go on in them. I guess maybe that’s just because implicitly the risk is less.
[00:26:36] I mean, you can take corrective action. You don’t wake up and find, geez, in the last two hours, you know, they just activated this thing that’s been sitting in the network for two years. They’ve pulled off all the employee records. So yeah. So that does go on in certain parts of the company. And to a degree, like if you’re still running a data center, there’s definitely some, you know, constant testing and just risk prevention procedures that are followed all the time.
[00:27:00] Jonathan: What about when things do go wrong?
[00:27:02] You were the one who had to deliver that bad news. This is what this podcast is all about. Let’s revisit that painful moment for you, Mark.
[00:27:11] Mark: So that one was an interesting one ’cause it was sort of a comedy of errors. So there I’m not gonna take the time here to go through the whole process. But it just turned out that we had a very small application that was not performing correctly. There were people that would normally have been in like the flow of this work process turned out one of them was on vacation and the stand in just wasn’t that familiar with what the, how the process was supposed to work.
[00:27:41] So to her it looked like it was working fine, you know? And it was just a concatenation of errors. If the viewers of this podcast or the listeners take away one thing, like, just one thing, here’s the deal. Bad news never gets better with time. So the minute, you know, ’cause if you go up there – so let’s say in this particular case, oh, gimme like 24 hours to really dissect this thing and figure out what went wrong.
[00:28:08] Because when I go up there, I’d have to tell them like, we have 600 impacted customers. You know, that’s gonna be the no, that’s not a good idea at all. Like, you go up there, right? The first opportunity you had where, you know, this information – you can always say, I’m, I’ll come back. Like I’ll come back like I’m researching it or whatever.
[00:28:27] I just want you to know, like, we got 600 unhappy customers, right? And we’re trying to identify them right now. In that particular case, I think we did some kind of a partial mea culpa. I don’t know if we gave them something free or, you know, whatever. But like we acknowledged we had the screw up and then we told them we’re gonna do something for them, in partial compensation. And we were, they were like, this is an order problem. And so we immediately, you know, prioritized their orders. But, yeah, bad news doesn’t get better with time. You’re better to immediately go up the chain of command and, yeah, just, it’s not, it’s not, it’ll only blow up in your face if you take longer time to tell somebody something bad. Yeah.
[00:29:09] Jonathan: Rip that bandaid off. Well, let’s shift gears to some of the things now. You’ve written some fantastic books. Looking ahead to the future and looking at the current trends in IT management, looking at how things are evolving. Really what the next stage of IT leaders, what they’re going to need. You’re doing a ton of advising. What are some of the things that you see IT leaders out there being very worried about or concerned about? What’s keeping them up at night?
Why IT leaders can’t sleep
[00:29:37] Mark: Yeah, that’s hard to figure out. You know, if you look at the, there’s these annual surveys that are done by like the Deloitte’s veins of the world every year where they ask people like whether you’re spending priorities for your IT budget. And historically, I’d say historically, the last probably five or more years, cybersecurity and data analytics have both, you know, been consistently, you know, at the top. And I think in both of those areas, people have purchased a lot of tools. They have a pretty complex tech stack. They feel like they’re overbought or over-engineered. They’re not getting, you know, the value for the collective investment that they’ve made in that stack.
[00:30:19] And they don’t really know how to unwind it, and they’re being barraged by new startup vendors or new people that come in and say, you know, I’ve got the missing piece of the jigsaw puzzle. Like, you’ve gotta, you’ve put most of the puzzle together. I have found the blue piece that goes in the sky that you have not been able to, you know, close the puzzle with.
[00:30:41] Jonathan: We have a whole new puzzle now it’s holographic and it’s got AI involved.
[00:30:45] Mark: Well that’s, yeah, well that’s, you know – so actually in the one, in the more recent surveys, the budget surveys going to 2025. Yeah. GenAI spending actually topped cybersecurity for the first time. But I don’t know how to interpret that because of course a lot of the vendors in the cyberspace and the data space are all offering up AI enabled products as well.
[00:31:06] So I don’t know the people that answer those questions, whether the questions are parsed, you know, effectively to figure that out. But yeah, obviously that’s sort of the big challenge of where and how to use AI and it’s – there’s a couple of problems there. So one is all of your existing vendors are pushing some form of AI enabled enhancement or, you know, new capability at you.
[00:31:30] Then there’s a new set of vendors who are saying, well, you know, forget about that like, we’re AI native, like we’re not ServiceNow building, you know, they’ve had a, they’ve had an IT service desk solution for 20 years, like in their – it’s big and cumbersome and expensive and hard to learn and hard to maintain and take specialized expertise, etc.
[00:31:49] We’re AI native, you know. It’s like we’re tackling the same problem, but we’re not encumbered with, you know, all that legacy thinking and the traditional way of tackling the problem. So you’ve got the legacy guys, you’ve got the new kind of startups that are going on and in many companies, especially like in a financial service area, you’re building your own AI applications or agents to do things – ’cause you, for security reasons in part. So that’s one class of problems. I think the other class of problems that is, I worry about, which I haven’t really seen discussed so widely in the tech press is AI explainability. Here’s the way I look at it.
[00:32:29] Almost all the applications that we’ve built in the past follow a certain set of rules or procedures and you put data in and then they have like known outcomes. And even if it’s like a machine learning model, you know, the model has been set up in such a way that for these inputs, there’s some kind of a forecast or prediction that has a probability associated with it or whatever.
[00:32:48] The LLMs don’t work like that. They evolve over time based upon the prompts they’ve received, what they’ve been asked to do, new training material that they’re given, etc. So they pose really more of an ongoing training – I’m sorry – testing problem. So they’ll, you know, well, you know this ’cause you’ve read about it or experienced it.
[00:33:10] You know, you don’t always get the same answer to the same question, like when you come back and why you got that answer instead of the other one. Or you and I ask the same question and you get one answer and I get another answer. I, so I think AI explainability. And I don’t, to some people, I think that has the connotation of, oh, these models are discriminatory in nature.
[00:33:34] You know, like they’re biased in the way that they treat certain people with certain languages or certain genders or whatever. And that’s a concern. I understand those concerns. I just mean explainability in a business decision. Like how, like you and I both submit an insurance claim and one LLM says, you know, give, Settle like $5,000.
[00:33:53] And they say, you know, give Jonathan like $50,000. But it was the same thing. Like, why did, so, why did that happen? And there’s a lot of evidence of this. This is not just me dreaming this up. There are, there are definitely, you know, instances where this is going on. So I don’t think too many groups are structured today, nor do we really have formal roles for that kind of ongoing, you know, deep analysis of why the models are producing the answers that they are.
[00:34:24] Jonathan: When it seems to me too that there’s an expectation of the ownership of this, ultimately will fall down to IT teams, right? It’s something that all of a sudden now IT teams are having to manage and, when users are having mixed results or things, the IT team is expected to be able to help with this.
[00:34:44] It just seems like for IT teams in in particular, I know that there’s, anytime there’s new waves of new technology and excitement around and hype around it, buzz, there’s all those factors of like, okay, but we’re the ones who end up having to actually roll this out, manage it, try to audit it, monitor it, all that.
[00:35:03] There’s a lot more work than the automagically-results that come in.
[00:35:08] Mark: Yeah, automagically. But the one thing that’s different this time around is that I think even big companies and small companies. I think have generally formed some kind of an AI executive steering group, you know, at the top of the company. So it isn’t necessarily IT just holding the bag and those executive governance boards or steering groups or whatever have tried to establish some kind of constitutional convention or rules of engagement. You know, like you’ve gotta come back through, not necessarily them, but some group that they’ve enshrined to make sure that you know, anything that you’re taking off the internet is not using DeepSeek because like, we’re gonna like to say that’s verboten.
[00:35:48] I can’t, that won’t happen here. So there’s, I mean, that’s encouraging. You think back like with the use of cloud computing resources. There was no executive steering group about how we’re gonna use AWS and like what workloads we can take up onto AWS and whatever, that was left to IT to hammer out workload by workload with the different functional departments and the apps that they were using.
[00:36:09] So there is that level of governance, but you kind of go back to the unpredictability of the technology and different people will be held responsible. I don’t think it’s gonna be like all IT at the end of the day. You know, and in fact, maybe there’ll be some new kind of organizational entities that are set up to actually do nothing but that.
Closing
[00:36:29] Jonathan: It is unsettling, of course, it’s also exciting. Mark, any other parting words to IT leaders out there who may be newer in their career who are facing some challenges, some, gosh, I mean, all the potential changes coming up. You know, you’ve had such great experiences, such a great career and you give advice all the time.
[00:36:55] Any parting words of wisdom for these folks?
[00:36:58] Mark: Well, maybe on the topic that we’ve spent, you know, our time on today is really try to, this sounds trite, but it’s true. You know, try to turn your organization into a learning organization. And so when things go wrong, don’t just like cover it up, you know, like, oh my God, okay. I won’t tell anybody if you don’t tell anybody. So you take the data center example, like, let’s all learn that yeah, there’s a certain level of process. Like, so even if you’re not involved in the data center operations, you’re over in the data warehouse or you’re running the support team for the Salesforce app platform.
[00:37:29] Like, can you look in the mirror and say, well, that would never happen to us because like we do, you know, you might wanna go back and look at like, how we’re adding objects to that platform, or some of the new APIs we’re opening up. Like, you know, are we doing the right testing and all everything else? So, I think you want to, well, yeah, you wanna learn from the things that go wrong for the benefit of the whole organization and not just kind of.
[00:37:53] You know, try to cover everything up and pretend that everything’s perfect all the time. And I’ll tell you the other thing is your customers outside of the IT organization will respect you for doing that.
[00:38:04] I think if any kind of, you know, post mortems or problem management processes that you have, the more transparent you are, and you will be respected. You learn from it internally and then you’ll be respected by your customers in the company.
[00:38:20] Jonathan: No. Covering things up. No hiding.
[00:38:22] Mark: No hiding nowhere, nowhere to hide. There is nowhere to hide in IT. That’s true.
[00:38:28] Jonathan: Mark Settle. Thank you so much for joining us for this episode. Again, your latest book, Truth from the Valley: A Practical Primer on Future IT Management Trends. I’m excited to check it out. I know our listeners will be too, and Mark, just very much appreciate you spending time with us. Thank.
[00:38:44] Mark: Thank you Jonathan. Appreciate it. I thoroughly enjoyed the opportunity to share some stories.
[00:38:49] Jonathan: And thank you everyone for listening in, and we’ll be back soon with another episode of IT Horror Stories.