[00:00:00] Wyatt: Nothing is sticking. Nothing is fixing this. The clock is continuing to tick down, and then another one comes in and then I see another one.

[00:00:13] Jonathan Crowe: Hello, and welcome. Please come in. Join me. I’m Jonathan Crowe, Director of Community at NinjaOne and this, this is IT Horror Stories. Brought to you by NinjaOne, the leader in automated endpoint management.

Introduction

[00:00:26] Jonathan: Welcome everyone to IT Horror Stories. I’m your host, Jonathan Crowe, Director of Community at NinjaOne, and I’m sitting down with Wyatt Hansen, VP of IT Services at LeaderOne Financial. He’s going to be sharing a little bit about an IT story that’s haunted him with us. And Wyatt, before we get into it, welcome to the show. Why don’t you tell us a little bit about yourself?

[00:00:51] Wyatt: Thank you very much. Happy to be here. My name is Wyatt Hansen. I’m the VP of IT for LeaderOne Financial Corporation. I got my education in theater at the University of South Dakota, currently reside in Olathe, Kansas with my wife and our Corgi, Luna. And, I think that’s all the important intro. There’s probably no other pertinent details about me.

[00:01:12] Jonathan: Oh, well, we will get into much more, I am sure, when you tell us your story. Thank you for being a brave soul and coming on the program. Let’s get into your tale at hand. And as I understand it, this is going to be set pretty soon after your theater major, or maybe while you were doing that. So help us set the scene here.

[00:01:32] What point in your career is this scenario taking place?

[00:01:36] Wyatt: It’s about 2017. I’m a few years into college, probably a junior at that point, I’ve been working for a small managed service provider, doing remote technical support for the last few years, getting some fantastic on the job education with my only real prep before that being a computer hardware class in high school and whatever I found on YouTube.

[00:01:55] A lot of credit to my supervisors and teammates for helping mold me into a really good remote technician. But yeah, just working as an entry level tech, doing remote support, working on a special client team with a very small subset of their customers, and that’s about it.

Nailing the audition

[00:02:15] Jonathan: Tell me about how you got that position and what the team looked like, what the work looked like.

[00:02:21] Wyatt: Funny enough, I met the guy who founded the business personally before I ever interviewed with him.

[00:02:25] And as I got to the end of high school, he had a job opening coming up. I knew that he did. We had a conversation and we did a formal interview. And I believe I graduated high school in May of 2014 on a Friday and then Saturday, I think we had the ceremony, and then I started with him on Monday. And so my primary role was to start helping customers over the phone who were hundreds or thousands of miles away with basic technical support. [You know] software isn’t working, a web page won’t load, phones’ acting up, something like that. Learning the ropes on best practices for communication, how to properly document what’s happening, learning how to use a ticket system, things like that.

[00:03:05] Jonathan: And so you’re doing this at the same time that you’re getting your degree. And I guess let’s go back. Your degree is not in computer science. Is that right?

[00:03:13] Wyatt: It most certainly is not. No. No, that is a bachelor’s degree in theater that I am going for at the time.

[00:03:18] Jonathan: So you had that going on. Your formal training is very much coming from this on the job experience. Were there signs early on that you were going to be involved in some, let’s just call them stressful scenarios maybe?

[00:03:32] Wyatt: I did not have an expectation going into it. The real defined conversations were you’ll be going in and doing some tier one support, basic troubleshooting. As time goes on, you learn more. Maybe we’ll get you into some more tier two support, getting into something fancy, you know, not just turn it off and on again, not just a remote session to look at what’s going on, doing maybe some registry edits, things like that.

[00:03:53] But nothing on, you know, you’re going to run disaster management and incident response when something pops up and coordinate across multiple teams, you know, nothing like that.

[00:04:02] Jonathan: For example. Were there any inklings, you know, in horror movies, you get the little false starts or the little foreshadowing. Were there any things that you saw – I guess when you’re the fresh person coming in right out of high school, you have your team. Were there grizzled veterans who were treating you as the rookie who is gonna have to like earn your stripes and all that?

[00:04:22] Wyatt: I don’t know that there were any caricatures to rely on, and whoever was directing my life at the time did a terrible job of foreshadowing anything. But no. I was surrounded by a couple of people in a small office most of them were in management and had actually helped found the business. And were doing a lot in terms of teaching, business administration, client relationships, things like that.

[00:04:43] And they had some pretty robust technical and business backgrounds. The other folks that I worked with, there was another college student who did most of my training, who I got to learn a lot from. He really took me under his wing. He did a great job of pushing me to learn a little bit more.

You have to have room to fail

[00:04:59] Doing some active coaching, listening in when I got on a call, letting me listen in on, you know, some of his calls to learn a little bit more. And the most important thing that I got from that environment was room to fail. And the ability to make small mistakes, and then when I make them, if they’re of any impact, to be told exactly what that impact was.

[00:05:17] But, no, I had nobody coming by and telling me, “Oh good, fresh meat!” Nothing, nothing quite so extreme. It was really a very collaborative environment. I would say probably relaxed as most offices go, but really just kind of a perfect atmosphere for learning and communication.

[00:05:37] Jonathan: This sounds pretty idyllic. It sounds like the perfect scene to throw a wrench into things and have a horror story happen here. So let’s get into it. Let’s go into code red. Tell us about the day… of destiny. Tell us about, about the scenario.

The day of destiny

[00:05:55] Wyatt: I believe it was a cold day in the fall, when I was working alone through an afternoon shift. And we had a service call come in through like a voicemail of like, Hey, you know, there’s, there’s an issue with the teams that I normally work with those special clients.

[00:06:13] They have a proprietary software that runs at their retail locations. Something that I’ve worked with a lot is at these retail locations, they have a sort of back of house server that goes out to front of house terminals. Those front of house terminals handle a lot of transactions with customers.

[00:06:29] And so when those back of house terminals go down, there’s something terribly wrong. And a lot of times that’s pushing folks to do cash only transactions, which even seven years ago, was still a big deal. Nowadays, even less people carrying cash, but then still a very large deal. But yeah, just a usual day until that call comes in that something is going wrong with the back of house unit. And I begin troubleshooting as I normally would. You know, I have a playbook to run through, I’ve done this before.

[00:06:56] It’s specifically just the software that’s bugging out. This is not something new. It’s also not a brand new error. It’s one that I’ve had a chance to troubleshoot before, so I’m feeling pretty confident going into it. Talking through with the contact, making sure that they’re being told, like, yep, I’m working on this, we got it, I’ve seen this before, let’s see what we can do.

[00:07:18] Jonathan: Got it. You have a sense of, okay, this could be an error that could lead into something more serious, but at the time with all the information you have ahead of you, you’re feeling confident, you know what to do. So let’s talk about what happens next.

[00:07:32] Wyatt: Normally troubleshooting this, like 10 minutes, you know, it’s one of a couple different issues, worst case scenario, reboot in a lot of cases is going to fix it. But in this case, we’ve done multiple reboots. We’ve checked a bunch of other contributing things. Connections to physical devices, connections to the internet.

[00:07:49] Latency, things like that. Pinging servers and it’s not really going anywhere. And it’s around the 45 minute mark on this call of troubleshooting that I’m realizing, okay, this is becoming an ordeal. This is something that is becoming very inconvenient to this location, and it’s going from afternoon, like mid afternoon, getting into evening.

[00:08:07] They see a lot of traffic around the middle of the day and the evening for their business, and so the pressure is kind of starting to mount. We know that there’s going to be more people coming through there. The importance of getting this up is paramount. And around that 45 minute mark, I, keeping an eye on our ticket system while I am checking with a few other people through instant messaging, checking on the web to see if there’s anything else that I might be able to use that I might not have thought of, and I look at my other screen and I look at our ticket system and I see an identical voicemail for the same problem.

[00:08:41] And I go, that’s odd. And then I ping a team member to go and take a look and they start connecting with the client, doing a little bit of troubleshooting. Maybe 15, 20 minutes later, we’re now like an hour into my first call on this issue and they’re reporting the exact same issue I’m seeing.

Nothing is fixing this

[00:09:00] Nothing is sticking. Nothing is fixing this. The clock is continuing to tick down and then another one comes in and then I see another one. And now we’re sitting at like four of these locations. And I know this business, they have between like 50 and 80 locations. Once you get more than one or two of them down, it can start having a little bit of a ripple effect where people feel a lot of pressure to get things working.

[00:09:27] There might be some side communications, people venting to somebody else in a district that they work with, something like that. And we’re starting to see this number rise slowly. And every single time it’s the exact same issue. It’s the exact same problem, and nothing is fixing it.

[00:09:46] Jonathan: So just to clarify too, this is one organization, different sites within that organization. Is that right?

[00:09:52] Wyatt: This company, as a managed service provider, varying levels of service to its many clients that numbered in the hundreds. And so some of them were a case of you paid us to install a router once and now you’re a client. We help you with that once in a while as needed.

[00:10:06] Others, you know, like the one I work with, there’s a full time retainer. We are acting as a supplement to an existing help desk they have of two people for all of those locations that are managing a lot of the hardware issues, whereas we are helping to deal with the software issues.

[00:10:21] So when it comes to physical security issues, site upgrades, physical hardware issues, they have a whopping two people in one central location that they can scooch out between, you know, what is effectively probably a 900 mile radius for, from their central location out to their furthest location and back.

[00:10:38] But yeah, there’s many other clients. I, again, because we have a retainer with this client for full time support, eight to five for multiple levels as well. I am pretty focused on this one and I have a lot of experience with it, with the unit managers with multi-unit managers, things like that.

[00:10:58] Jonathan: And so as this is going on, are you getting, basically managers or other people at this client. Are they escalating it? Putting more pressure on you guys at all.

[00:11:08] Wyatt: Within the first hour or so, no, there’s really nothing coming in. There’s nothing coming in from like a technical contact with that company. We’re not seeing some kind of widespread messages. My boss is not getting a message from their boss, anything like that. We’re still going through the usual troubleshooting methods.

[00:11:28] But that starts to shift around 90 minutes or so. That is where we start to see a need for proactive communication. My boss has already been made aware of the issue, you know, around that 45 minute/hour mark where I’m dealing with something. I’ve escalated, Hey, I normally can fix this within 10 minutes.

[00:11:46] It has been almost an hour. I am not there. Kind of rough. We have seen a second location, but I’m not sure. And they say, okay, well, we’ll take a look at it in a little bit. That 90 minute mark, that’s when we start to see a little bit of extra communication come in. Somebody with two stores that are starting to struggle has reached out and said, Hey, you know, is everything going okay?

[00:12:06] What’s going on? And, you know, we’ve had to quickly message them back and give them an update and let them know what we’re doing.

[00:12:11] Jonathan: You had the playbook, right? You knew how to handle the situation before, but now you’re going into uncharted territory. How are you feeling at that moment?

[00:12:19] Wyatt: My initial response is usually curiosity. When it comes to technical issues, most people have fight, flight and freeze. It’s just the human stress response. And I don’t like flight or freeze. I tend to go fight and I want to try and go head down and solve that issue. I am exceptionally curious and a little nervous.

[00:12:38] About what could be causing this across so many different sites, and that’s leading to a desire to start looking at other things. So it’s starting to look for news articles popping up in say, Google News, about other things happening. With not just this, this site, but, say, the technology that they’re using, because that’s not proprietary to them.

[00:12:59] The software that they’re using to run their retail systems is used by hundreds if not thousands of stores in that geographical area in the Midwest. And so we’re starting to see a couple of things show up, maybe on sites like DownDetector, where they’re just little murmurs. There’s nothing too crazy, but overall I am, I am still curious.

[00:13:20] I’m still nervous, but I am, I’m not ready to throw in a towel or run away with my hair on fire or anything like that.

[00:13:28] Jonathan: And also, too, I mean, this is, even though the situation has shifted and this is no longer one ticket that you’re trying to troubleshoot, you’re still staying responsible for addressing this issue? What did your response plan look like?

[00:13:41] Wyatt: Yeah, so usually if I was going to take that request, if I was going to take that ticket or call or whatever it might be, my job is to update the original ticket, change the status to note that it’s in progress, and then if it’s a retainer start working the ticket if it’s hourly and that customer is on bucket hours.

[00:13:57] It is to start a timer on that ticket and start recording how much time I am working and dedicating to that customer, to that ticket specifically, for billing purposes, really common with MSPs. And so I start with it. I grabbed the ball and then my job is to move it up the court, so to speak. I’m sure we have to throw a sports analogy in here somewhere.

[00:14:16] Jonathan: You have the ball. Let’s keep going. Where do you go from there?

[00:14:20] Wyatt: So I’ve been working with the supervisor and one other team member to let them know what’s going on. I am now, you know, now that I’ve seen four or five sites with this issue, I know what the impact is, I know it’s severe, and I know I’m starting to maybe see something on down detector and none of the usual fixes are working.

[00:14:34] This is where I’m letting my team know, hey, this is an active issue. This, this has gone from maybe something on the fridges to you know, for the 3 or 4 of us that have to deal with this, part time or full time, we need to be aware of what that is. Here’s our group chat or whatever, here’s an official communication from me indicating what the problem is and what the scope of it is.

[00:14:56] And then trying to guide mitigation efforts by showing people, here’s what we’ve done. This is our normal playbook, you know, we’ve tried this at 2 different sites, this is everything that hasn’t worked. As this stuff comes in, defer to my supervisor, but it may be best to let, you know, do some token troubleshooting with these sites. Try a couple of these things just to see if anything sticks, but really focus on the communication as it’s highly unlikely we’ll be able to solve this root cause as it appears to be something from the software provider itself.

[00:15:27] Something happening at the software level. And that’s obvious because, you know, all these physical troubleshooting, all the basic troubleshooting methods are not working. And we help a little bit with patch management on those machines. I can see through a basic, you know, remote management tool, Hey!

[00:15:43] There hasn’t been some big Windows patch that got deployed to all of these. I’ve thought to check that between those two. Like, is this the same, exact same system? Is this the same manufacturer? Is it the same model? Is it the same motherboard? Is it the same operating system? Is it the same version of the operating system?

[00:16:00] And after comparing two and thankfully getting up to three where the pressure mounts, but it’s helpful because I have more data points, more things to cross reference, I’m realizing there is no real rhyme or reason aside from the software and that version of that software, which cannot be easily removed and reinstalled.

[00:16:17] That is a more laborious process.

[00:16:21] Jonathan: Yeah. And as a reminder, I mean, we’re talking about a piece of software that’s critical to their business operations, that’s right?

[00:16:27] Wyatt: Yeah, if that software goes down, they go from having all of their business transactions run through their point of sale system, or POS, and having all of their cards – credit cards, debit cards, etc., gift cards – run through there, get recognized, and have payments processed. All of a sudden, they have to manually write down orders for goods and services, and calculate that out, and try and calculate the tax accurately, by hand. Which inevitably leads to not only data retention and reporting issues. It also leads to a lot of customers driving off and leaving that retail location because they do not carry cash on them at that time, or they don’t have enough for whatever it was that they purchased. So yeah, it’s a really, really big impact. The only thing that would have a bigger impact would be losing power or internet.

[00:17:22] Jonathan: How long is this going at this point? How long have the stores been having issues now?

[00:17:27] Wyatt: So for these four or five locations around the 90 minute mark, they’re sitting around consistently between 30 and 90 minutes of downtime. And thankfully they’ve all been communicated to. They’ve had either an email go out to them or a direct call. In most cases we have tried to do a direct call because that allows them to ask questions and it generally helps them feel more heard.

[00:17:46] Plus it makes sure that we don’t miss anything in terms of context. Because nobody who’s going through a major system outage wants to email back and forth with support? Especially not when they’re being told, Oh yeah, no, this is a known issue, hang in there. That is not going to give anybody any warm fuzzies, not by any means.

[00:18:04] We’re seeing this impact, and then it’s still growing, little by little. The whole team is aware, our management is aware, and the request is out from me to make sure that that company, that, that client, that their management is aware. Their multi-site managers who help with these retail locations, their technical support, their team of two people, and anybody else at the corporate level that might need to know.

[00:18:28] Because at this point, the impact is growing and we may see more tense conversations come up. There is a service contract with us to support them within the bounds of what we can do. And something that’s really tricky in technical support and even more so as a managed service provider, is what falls within your scope and what doesn’t. Because when it comes to keeping those units running, we go a step above best effort.

[00:18:57] And so now we’re getting our new playbook down, for this is the issue, these are the three things that we try first, and then we communicate what the fix is. And we’re going to people outside of our dedicated special client team. We’re going to our more general support team that handles the vast majority of our clients.

[00:19:14] We are in a very fortunate situation. The software that we’re working with is not proprietary to this client, but it is industry specific. For people that have point of sale systems. And so for the rest of our company, for the rest of the managed service provider, a lot of their other clients are doing professional services.

[00:19:33] And so what we’re doing is a completely different skillset. And so there’s also an onus on us to take pretty rigorous notes, so that if somebody from that broader team that does, you know, are the full time employees, and are doing the vast majority of the tickets – they aren’t part time college students like myself – they are able to jump in and assist.

[00:19:53] So now, in hour two, making sure that team is brought in, making sure they’re aware, and making sure that my running notepad list of whatever I can think of is thorough, because there is going to be a need for me to actually ask to go past my scheduled hours at this point. I am now hitting the end of what should be my day, and so I’m checking with my supervisor to say like, Hey, I know we’ve got somebody coming in for a little two hour shift here.

[00:20:17] Can I stick around and help them for a bit? And just kind of make sure things are okay, getting the okay for that or the denial and then proceeding on to the next step and just generally making sure the whole team is aware of exactly which sites are affected, exactly which systems are affected, and exactly what’s been done.

Seeing things through

[00:20:37] Jonathan: Yeah, and so we, it’s clear – you want to see this through. But there’s also, in addition to your natural curiosity, I mean, I think what you’re pointing out here is there’s real continuity value too. You’ve been central to documenting, to communicating all of that.

[00:20:53] And so there’s a real value too. In addition to, you know, You wanting to see this through to the end.

[00:20:58] Wyatt: I love a good puzzle, and I don’t like to leave work unfinished if I can avoid it. If you looked at my house projects right now, you wouldn’t believe that, but trust me, it is a thing. I don’t want to leave those unfinished. I do want to see them through the end, but I have to wrestle with this internally, like, this is more than I can manage at once.

[00:21:50] Jonathan: Gotcha. You’ve identified that pretty conclusively, this is rooted in a line of business software. Are you feeling beholden to the software? I mean, is there anything that you guys can do? You’re just hoping to get in touch with them. What’s the next play here?

[00:21:31] Wyatt: It’s really working with that software vendor as best we can to communicate the exact issue, the scope of the issue, and get their support as quickly as possible through the existing communication channels that we have. Luckily, with that one, I think that we had a better communication system, some good instant messaging, they had a pretty robust support staff.

[00:21:51] It wasn’t one of those things where their SLA for all requests is 24 hours. For those higher priority requests, we’re getting a higher service level agreement or SLA where we get a faster response. There is an urge to own it. There is, you know, a fire under our butts to work with that software provider and find a fix.

[00:22:07] And so we’re getting into, you know, probably two and a half hours at this point. We’ve done some work with them, that we’ve taken a look. They’re circling up with their team internally, and now, more things are showing up in DownDetector. More things are showing up in news articles, because again, this, this line of business software is not unique to this client.

[00:22:25] I’m starting to see, like, oh, all of these retail locations for, you know, fast fashion. For grocery stores, for fast food, they are suddenly starting to show up with outages. And these articles are saying, you know, major issues across retail as this, you know, point of sale system goes down. They throw in, you know, some fun little words like chaos and havoc and all kinds of dramatic stuff, just to make sure they get some clicks.

[00:22:53] Jonathan: Gotta get those clicks! Something that strikes me here is that, you’re talking about two and a half hours and in the grand scheme of things, that is not much time. And yet so much has happened behind the scenes. So many parties have become involved. Now we’re seeing articles on the news. This is all happening really fast.

[00:23:11] Wyatt: I was lucky that the news articles started popping up that quickly. Living in the digital age, having access to news that instantly. It is all happening really, really rapidly, and again, the only reason I’m getting the reporting is because it’s so widespread.

[00:23:25] In the past few years with things like the CrowdStrike outage, with some major Azure and AWS outages, because those providers are so large and they service so many clients across so many industries, the minute they start seeing wide scale impact it is the story of the day. And questions about how, you know, how does technology influence our lives?

[00:23:50] Is there something existential here? Or, you know, just some, something else to talk about while the news is slow. But yeah, it’s, it’s a really fortunate situation to have that documentation and know something is happening at a, at a wider scale.

[00:24:01] Jonathan: At this point, you should be able to say, well, it’s your software’s fault. Our job is done here. Right?

[00:24:08] Wyatt: Yeah, and so, once we start seeing those news articles come out, we’re getting something firm from the software provider, you know, they have a status page that we can look at. The pro tip to all of the people new in the tech space, status.io or Atlassian status pages.

[00:24:27] Those things are incredibly useful in figuring out what’s going on. Just generally when tech is not working and signing up for those, or especially getting a tool that gives them updates through RSS feeds, super handy. But yeah, so at this point we know that it’s an issue. We know that the provider is having some kind of major problem.

[00:24:45] And it’s a matter of crafting that careful, specific, professional communication to that client’s management first, probably through a phone call, and letting them know like, Hey. We have bad news. This has gone out of our scope. There’s something with the software provider for this point of sale software.

[00:25:03] We are letting you know, we will keep our ear to the ground. We have people staffed after hours. They are going to be keeping an eye and an ear on this. They will be listening for news and helping wherever they can. But until this provider gets a fix rolled out and a patch pushed, there’s not going to be much we can do.

[00:25:22] Jonathan: At this point, you spent so much time now you’ve become invested in it. You know, you don’t want to get off this case. You want to solve this and see it through. But now you’re having to wait. And I imagine too, it’s not as if you can just step away and check back in in the morning. You have no idea when this fix is going to go out.

[00:25:40] Wyatt: I really want to stick around as long as I can. I want to see the fix roll out. I want to help wherever I can. But I also have the presence of mind to say to myself, this is no longer within my hands to fix. This is squarely in another party’s hands. And, you know, my job of clear communication internally has been done.

[00:25:59] Now my job is listening and trying to see what I can learn for, for next time.

[00:26:06] Jonathan: Communicating, disseminating that information, the accurate information. It becomes so important because especially, I imagine with any kind of IT professional, but especially with MSPs where there is, we are servicing you, the client, and there’s not always that clear understanding of scope.

[00:26:26] They basically see like, Oh, tech’s not working. It’s your fault. You’re kind of responsible on the hook for it, right? And in a lot of clients’ eyes.

[00:26:35] Wyatt: Yeah, I mean, getting that communication across as to where the blame lies, but also being clear about it is tricky. I cannot count the number of times where I’ve had to go through three different parties that have all been pointing fingers at each other about something not working correctly. Still have things that I’m dealing with right now where I’ve got, I’ve got to work with multiple parties, and deal with that service finger pointing, but, honesty, clear communication. Those are just the best things that you can do.

Resolution

[00:27:03] Jonathan: So what happens next? How does this situation end up finally getting resolved?

[00:27:06] Wyatt: I have got to work a little bit later. I have got to make sure that all my notes were done and any other small requests that I might have missed while I was dealing with the big incident have been resolved.

[00:27:15] And that the team is ready to deal with it. So I’m going to head home and keep an eye on the news and see if I can find anything going on and ultimately try and get to the homework that still needs to be done. And the next morning, kind of flashing forward after the time decompressing from all of this and getting out of that stress response mode. I’m seeing, you know, thousands of stores back up after the provider pushes a fix and some formal apology goes out.

[00:27:43] And then I come into work the next day and I’m able to see the requests have been closed out. The communication has been sent out to those clients that things are resolved and there is that sense of closure. And that’s really helpful to see if not exactly what the fix was, just a guarantee that the fix has been rolled out and that those clients are now back up and running.

[00:28:01] Jonathan: And so you called on this story for a reason. Tell us a little bit about why. I mean, has it stuck with you? Were there learnings that you took from this that you put into action or that you feel like is important to call out to others?

[00:28:15] Wyatt: The biggest thing that made this stick out for me was, it was kind of one of my biggest professional leadership wins in the first few years of working in technology. Really just taking the lead, owning the problem, owning the communication and building on what I had learned over the course of several years to have a positive impact was something that has stuck with me the whole time.

[00:28:40] I’ve referenced this incident many times for learning and best practices, sometimes for my team as well. Anecdotally, it’s always fun to talk to one or two coworkers about “remember this time everything sucked.” There’s a lot to take away from it.

[00:28:55] And nowadays it really, in a mid to senior level leadership position, it’s a good reminder about how important the act of leading up is. If you want to hear more about that, you can listen to somebody like Simon Sinek talk about it. But communicating up to your leaders and giving them the tools that they need to succeed, And going a little above and beyond what is in your station and your exact job title.

[00:29:23] Jonathan: Beyond just the incident itself, putting, going back to your perspective as now you’re a VP of IT. You’ve got a team. You’ve got people who are in various stages of their career under you, I imagine. Including people who are just getting started. What struck me is, you had an incident where there was a potential crisis that was really an opportunity for you to flex some of those muscles, or really try to step into a leadership role, and that really helped you identify this is something that you want to pursue, right?

[00:29:53] But you were also in a position at this MSP where you had mentioned you were in an environment that encouraged learning. They encouraged, try things, fail, learn from them, get better. And that you had those resources around you. So, I’m interested if this has played a role in the environment that you want to create as an IT leader.

[00:30:13] Wyatt: Oh, it absolutely has. My team gets a little sick of me saying, I think the phrase is, ‘Fail wisely.’ Because I want them to use failure as a learning experience. It’s good failures, good mistakes, things like that.

[00:30:29] Because a lot of times when you’re starting out and you’re just trying to get through your day and learn as you go, maybe you don’t have the best training materials or hands on education. You are just trying to move as fast as possible, and you’re more prone to making mistakes. And you’re also generally not given a lot of context.

[00:30:47] You’re not told exactly what this business process means, or something as simple as a, you know, printing error, or a, you know, printer not feeding. What, what does that have? What business impact does that have? And really taking that time to slow down and say, here’s what it is. And I found that a lot of times, that really helps with giving constructive criticism.

[00:31:11] And it creates, in a lot of individuals, an artificial curiosity. Or enhances their natural curiosity. Because once you start giving this context, naturally it creates questions. Well, if this printing error is so small and it creates this ripple effect, what does this other error do? What does it affect?

[00:31:35] So on, so forth. And, you know, on one hand you need to manage that carefully and not let anxiety take over. But on the other, follow that path.

[00:31:43] Jonathan: I think for everybody listening to this story that’s not too much of a surprise, that you’ve let that curiosity lead you down the right rabbit holes. You can be really good at troubleshooting, you can get really in the weeds with the tech. But being able to understand the context of how this fits into a business, how to communicate that with people, is such a big part of moving up and taking a larger leadership position.

[00:32:08] Thank you for reliving your tale that was stress inducing, but also formative. And I, the fact that you’ve told it multiple times, makes me realize that we didn’t stress you too much out by having you on to talk about it.

[00:32:22] But thank you for doing that. Wyatt, that’s all the time we got. Thank you so much for, for coming on IT Horror Stories, sharing your story, sharing your thoughts. Really appreciate it.

[00:32:31] Wyatt: Thank you, sir.

[00:32:32] Jonathan: Thanks for listening to this week’s episode of IT Horror Stories. For even more information and resources on how you can beat IT misery and transform your IT, check out www.NinjaOne.com. Or pop by our IT Leadership Lab community. At www.theITleadershiplab.com. There you can connect with other IT leaders talk shop, and get access to the latest guides, templates, and documents from other experts in the space.

[00:32:55] Remember, whatever your IT horror story, just know you don’t have to go it alone. That’s all for this week. We’ll be back with more soon. Thanks for listening.