From an irked former contractor in Australia sabotaging a sewage plant in 2000, to the more high-level 2017 Triton malware attacks on Saudi Arabian petrochemical plants, operational technology (OT) for critical infrastructure has increasingly been a cybersecurity concern.
But now, the COVID-19 pandemic has led to an increase in shifts for employees as well as more remote access to facilities — which is presenting new, unprecedented challenges for manufacturers, utilities and other locations.
Threatpost talks to Lesley Carhart, principal incident responder at industrial cyber security company Dragos, about the biggest problems facing OT teams – and how they can formulate the most effective incident-response plans to keep up with sophisticated threat actors.
Lindsey O’Donnell-Welch: This is Lindsey O’Donnell-Welch with Threatpost. Welcome back to the Threatpost podcast, everybody. And I’m joined today with Lesley Carhart, to talk about operational technology security and incident response. So Lesley is the principal incident responder at industrial cyber security company Dragos. And she spent the last 13 years of her more than 20 year IT career specializing in information security, with a heavy focus on response to nation state adversary attacks. So Lesley, thanks so much for joining me today.
Lesley Carhart: It’s a great pleasure to be here. Thanks for having me.
LO: Just to start, can you tell us a little bit more about your background and how you really worked your way into the OT – or operational technology – side of cybersecurity?
LC: Yeah, so I’ve always worked in nonstandard information security systems. Before Dragos I was at Motorola and they have huge factory and radio over IP spaces. And prior to that, I worked for some telecoms with big telecom deployments. And before that I was in the Air Force working on aircraft and avionics. So I’ve always kind of been in the space of weird computer systems and what can be done with them. And I enjoy it very much. And it’s a very important space that a lot of people don’t think about.
LO: Right, right. I think it’s really top of mind, especially ot security right now, as it relates to nation state activity. You know, recently the U.S. Treasury Department sanctioned a Russia government research institution in late October, saying it was behind a series of cyber attacks that used the highly destructive TRITON malware. So I think this recent government activity shows how governments and organizations are taking OT and industrial control system – or ICS – security more seriously. And, you know, just before we talk about what that means, on the organization side, and in terms of incident response, can you kind of set the context here in terms of what the specific security holes or challenges are that really need to be addressed, that exist and kind of are affecting these companies?
LC: Yeah, so there are the technical and organizational challenges there. There’s both of those. First of all, these environments are very different from the IT environments in a lot of ways – yes, there’s some similar technologies, and they’re growing more similar over time, in some ways, but they’re very different environments. And they have different needs and different priorities. So there’s organizational and technical challenges here, because you’re dealing with situations where if a piece of equipment fails, it’s not ‘Oh, the computers down and need to go home for the day,’ it’s maybe somebody dies or gets injured, or there’s environmental contamination. So the consequences are very different of industrial failures that are caused by computer systems. And then again, you have the systems, the technical constraints, things like operations that have to run 24-7, 365 days a year with maybe one or two days off for for downtime and maintenance a year.
So you’re dealing with longer life cycles, less downtime, and a lot higher stakes in these environments. And a lot of these systems are in operations for 10, 20, 30 years, with only, you know, some some maintenance upgrades and not substantial system changes. So legacy systems, systems that are rarely patched because of operational needs, and then these great consequences that can potentially happen to the systems and the operators and the people in the surrounding area.
LO: Right. And I appreciate that you mentioned organizational, you know, issues or challenges. I know kind of talking to OT experts that there’s definitely a cultural difference there between OT and kind of the IT side of things, especially when it comes to patching, for instance, or if a factory needs to deploy an update or something that the fact that downtime is a lot more costly for industrial firms, and it really has kind of a bigger impact there.
LC: Trying to plug IT security on top of operations has caused the water to be bloodied for many years, often times there are very hostile relationships between the IT and the operational teams because IT security’s been trying to force what we consider standard cyber security models on top of the OT systems, and it just doesn’t work. We can do cybersecurity there, but we have to be adaptable and understand OT’s needs.
LO: On the technical side of things. Is there any one or two specific challenges or security holes that you could pinpoint that lead to attacks or anything that are commonly used as an initial attack vector there?
LC: Yes, so there’s a lot of assumptions we make about – as IT people or cybersecurity people – that we make that are incorrect about OT environments, and one of them is that they’re all air gapped or close to air gapped. And that’s really very, very, very rare. Usually what we see in semi-secure environments is simply a DMZ between IT and OT environments. And that just means a firewall with some access control. And that firewall might be full of holes, there might be tons of old rules in it that allow all kinds of things. So we see a lot of Swiss cheese DMZs between IT and OT there that are exploited and taken advantage of. And we’ll see a lot of the lack of understanding of those environments. So people think that their existing IT security planning from monitoring to incident response or covering those OT environments, and they’re not, they don’t see what’s happening inside them, they don’t have enough contextual understanding of what’s happening in OT. And therefore, when something actually goes catastrophically wrong, they either don’t see it, or it takes them a great deal of extra extra time to respond to it, because they’re just simply not prepared.
LO: That’s a really good point. I also wanted to briefly talk about kind of the threat actor sides of things as well. And I know that Dragos has done a ton of research around different types of ICS, or malware that targets ICS and OT, especially, obviously, TRITON or TRISIS, which you know, just for some background for our listeners is most notoriously known for a series of 2017 attacks on a Saudi Arabian petrochemical facility, where it targeted and safety systems, you know, with the aim of causing physical damage. But you know, there’s also been a ton of other malware families that have popped up over the years, including Stuxnet, or Industroyer/Crash Override. What are you seeing on the threat actor level, you know, especially as it relates to kind of nation state activity as well?
LC: First of all, in terms of incident response, and I’m out there responding to industrial incidents all the time; I’ll divide what I respond to into three categories. So first one is insider threats. And, of course, people who work at plants can cause a lot of damage – either purposely or not purposely – if they decide to mess with things. So that that’s very, very common, especially with the changes made post-COVID becoming a problem where we’ve been allowing a lot more remote work and a lot more remote access to systems. The second category, of course, is commodity malware, and ransomware. While they’re not targeted to impact low level industrial devices, in most cases, they can still wreak havoc in an industrial environment, because there’s Windows systems there. And there’s a lot of older and vulnerable Windows systems, embedded systems, etc., there that can be impacted negatively by ransomware. And cause things like loss of visibility or loss of control. And then finally, there’s those advanced adversaries and in the industrial space, they tend to be well resourced adversaries. I’m not going to say necessarily that they’re all states, they could be, you know, contractors, other well-resourced criminal organizations, but it takes a lot of resources to really launch a successful purposeful industrial attack. You know a lot about the system, you have to know a lot about how it functions at a process level, and how to actually cause chaos and damage, in a system that’s got a lot of redundancy and safety controls, physical safety controls, human safety controls in it. So they’re well resourced people. And it takes them a lot of time and money and people to launch their attacks. And the thing about that is they don’t want to just take every system in the universe down today, because that’s wasting money. What they want to do right now, and what we see them doing is they’re learning how to attack the people who they might want to attack in the future. And this is global. Adversaries targeting adversaries. They are learning how to attack verticals and pieces of equipment and specific providers and organizations, so that when they need to in the future, there’s not this huge lead time to understanding how their system works. So we see a lot of reconnaissance going on – so adversaries breaking into tons of different networks all over the place, just to figure out how things work, just poke around a little bit and look at settings and configurations and build footholds in those environments. So they can at least get into the IT network of those organizations in the future quickly. That’s what we see that the bulk of attacks being right now. And that’s tough for the operators because it doesn’t help build budgets. What helps build budgets and security, unfortunately, is being attacked and seeing a lot of attacks happening and we’re not going to see a ton of attacks happen today in industrial control system security, because the adversaries don’t have motivation – geopolitical motivation or whatever – to attack those environments right now this instant. But they are breaking into them, they are hacking into their their organizations just to learn how to attack them successfully in the future.
LO: That’s interesting. And it’s a good point too that it is just to take note of how it is, you know, very expensive and difficult for groups to kind of launch these types of attacks. Although I do think, you know, one concern has been, you know, with with a certain threat groups, like I know, the group behind TRISIS, has been able to attack industrial control systems, across geographic and industry boundaries. There’s a lot of concern that this might be a sign of more of what’s to come when it certainly relates to nation state activity, as well.
LC: Yeah, it’s only going to continue to grow. They’re learning more and more about their systems, they’re getting more and more effective at launching attacks, very specialized attacks against specific systems, devices. They’re learning. And some of that is just learning what attacks work and don’t work. Sometimes the targets we see attacked with industrial attacks were simply the easiest place to break into and test a piece of malware, or see what actually could be done in another environment. So it’s hard to tell, without boots on the ground intelligence doing spy versus spy stuff. But in a lot of cases, some of that might just be, ‘Hey, could we do this in another country or another city?’
LO: Right. But your point too about insider threats is really interesting, because I, I feel as though especially in security news coverage, that’s something that is not as widely talked about when it comes to industrial control system security. And like you said too, it’s not necessarily always malicious. I mean, sometimes it’s just more of an issue of human error, mistakes. So can you talk a little bit more about that, and what type of threat that can lead to?
LC: Yeah, really, something that a lot of people don’t know is the first thing I would call an industrial cyber attack against an organization was an insider attack. It was long before Stuxnet. It happened in Australia at a sewage treatment plant, where a disgruntled guy who hadn’t gotten hired full-time after a contract, and knew the system really well, decided to dump a bunch of raw sewage.
LO: Oh no.
LC: It was a disgusting, horrible mess. And yeah, and that was totally an insider, I would consider it a cyber attack based on the technology used. Others might disagree with me. There were a lot of semantic debates there. But insider attacks are a big problem that don’t get talked a lot about in the industrial space. Because companies don’t want to admit that they had some fired engineer who retained access. That’s an ugly piece of news to have out there. And for good reason. But yeah, I mean, a lot of it isn’t malicious, a lot of it is, hey, these guys are locked in their facility now, due to quarantine restrictions for weeks at a time, of course, they want to watch Netflix, of course, they want to look at their personal pictures on their operator workstation. And those are problems that exist everywhere, but they can be a lot more severe in a sensitive industrial environment that’s handling some some dangerous equipment.
LO: Right. And I think that also brings up another point, which is, how things are shifting during COVID-19 and the pandemic? And can you give us a sense of – for those factory floor workers or for people who work every day with OT – how things have changed, due to the pandemic and what that means for cyber security threats or risks.
LC: Our utility providers have gone vastly above and beyond for the last – What is it now – seven, eight months, during COVID quarantines and lockdowns. A lot of facilities are doing shift work. So they are, you know, taking a shift of people and making them work for a week or two weeks straight and live in the facility so that, you have isolated groups of people, if one gets sick, you still got another one you can pull in. And of course, that’s a lot to ask of those operators and those engineers, they’re doing an amazing job keeping things running even with people getting sick and things so they have obviously changed their operations, who they let into facilities, how long people work, when they work, where they work, all kinds of things like that. And there is more remote access being allowed to the facilities as well, so that remote repairs can be done, remote facilities, everybody’s adapting to lockdowns and travel restrictions, things like that.
LO: Right. And that’s across all industries. I’d imagine but I’m sure that you know, it’s kind of heightened with with OT and, you know, what those types of landscapes entail. So with all this in mind, you know, whether it’s insider threats, or kind of the risk of nation state threat actors – all coupled together with these weird COVID times – Can you talk a little bit about what that means for organizations and how they can best prepare an effective Incident Response Plan, and from their end goal that can effectively, you know, prevent these types of security attacks.
LC: Yeah, so I tell people, I’m a terrible salesperson. Because, if people don’t prepare either in IT, or OT, for incident response scenarios, what happens is, they have to call in the incident responder for X number of dollars per hour, which is a substantial amount of money. And that incident responder has to do all their preparation work before they can actually start incident response. We have to build out their asset inventories their network maps, figure out where their crown jewel assets are, figure out where their vulnerabilities are, we have to do all of that to do incident response, because that information is essential to figuring out how an adversary got in and what they did. So if you don’t have basic incident response preparation done in either your IT or OT environment, you’re going to see spending a lot more money when you have an incident to have it done by your consulting incident respondent, or even your on staff highly paid incident responders.
And then on top of that, you need to have an OT incident response plan as well as an IT response plan. Because they are not the same thing. They are not the same environments. They use different technologies, different protocols, different people are involved, different technologies are involved, and the consequences are so very different. So your risk management is totally different. And I heartily disagree with anybody who tries to claim that your IT incident response and security program can just wholly cover OT and your ICS systems, it’s just not possible. You need to specialize there, you need to understand what’s going on, you need to understand the people and what they do in that environment, the process and where the potential points of failures are in that in that environment. So that means different asset inventories, it means different network maps, it means different risk modeling, it means different lists of things like crown jewel assets, and you know, content that you can use for forensic evidence. That’s that’s all going to be totally different. And it’s going to have different implications.
LO: Right, I feel like you know, also, as more of these IoT devices are becoming more connected and digitized, it’s becoming, there’s just so much there to monitor and keep track of, whether it’s end devices, whether it’s different networks, as you mentioned, and protocols and so many different types of holes that can open up potential issues here. So it seems so vast.
LC: It is it’s a vast space. And a lot of people ask me, How do I get into industrial control system cybersecurity? And to me, it’s the same question of how do I get into cyber security? Well, what do you want to do if you if you want to learn how to attack systems, how to defend systems, in the industrial space, if you want to learn about electrical power generation facilities, you want to learn about manufacturing, there’s totally different vendors and protocols used in those spaces, different technologies, different standard layouts of equipment. So we really need to specialize and figure out where you’re going to start. And if you’re securing an environment with industrial devices in an OT environment, you need to understand what those environments are and what they entail.
LO: I’m curious too when you’re talking to industrial companies and maybe working with them or looking at their incident response plans. Is there any big challenges or kind of overarching issues and with these types of plans that you’re seeing, or any advice that you would give to organizations to kind of create the optimal type of incident response?
LC: Yeah, so everything that people expect for you to have in your IT Incident Response Plan and the basics there -things like network diagrams, contactless acid inventories, collection management for your various log sources, things like that – All those things you need to have on your OT site as well. And they’re going to be completely different. And I’ve walked into so many organizations as an incident response consultant, where they hand me the OT Incident Response Plan, and they say, Oh, it’s in here, right? And then I look through it and there’s nothing about an OT asset inventory, what devices are there, what they do, what vendors and firmware revisions are in place. None of that’s in there. And I say do you have this information. And then everybody stares at each other, and they’re like, ‘well, Joe Bob, who’s been here 30 years might know, but maybe Joe Bob’s on vacation, and none of this has ever been written down.’ So yeah, you need to do that preparatory work. You need to spend some time shadowing your operators and your engineers, you need to understand your processes. And then you need to start from the ground up doing those security hygiene things like understanding what devices are out there, what operating systems they’re running. And this is not a security audit, this is not tell them to get off Windows 95, because they might be on Windows 95, for a good reason. But you need to know that these Windows 95 systems are there, and that you’ll be able to do forensics on them. A lot of forensics tools are not going to run on your Windows 95 computer that doesn’t have USB ports. That’s a nasty surprise, if you’re not familiar with dealing with OT systems.
LO: Yeah, and I really feel like a lot of this too points to just the disconnect that a lot of organizations are facing between those IT teams and the OT side of things as well. I wonder if you know, if this type of disconnect could be integrated better in the future or something? Or if that would, would really help more with that type of incident response issue?
LC: Yeah, absolutely. We have to build better relationships between IT and OT, and not just the technologies. We talked about like OT-OT convergence. And that’s mostly about the techie stuff that they’re using more IT familiar devices in OT, yeah, that’s happening, but not everywhere, not at the lower level devices necessarily. And it’s human relationships we need to build right now. Sometimes I go into these environments, and I call myself a marriage counselor because their relationship is so bad, they don’t trust each other. They don’t talk to each other. They don’t have routine calls where they discuss cybersecurity. None of that’s happening. And it’s really unhealthy. And it bites them when they have an incident, when something goes wrong.
LO: Right, love that term “marriage counselor” because I think it is so true when it comes to IT and OT and really what needs to happen there.
So, Lesley, before we wrap up, is there anything else you wanted to mention about cyber security incident response in OT, or if there’s anything we should be looking out for in 2021, relating to OT, security or anything else there?
LC: Don’t be daunted by OT being an overwhelming space with a ton of different technologies and different protocols and things like that. Start somewhere, start today, go meet the people there, start learning about what they do, start learning about the technologies in their environment. It’s okay to only know part of it, as long as you’re trusting your experts in your IoT devices. Start somewhere, start today, even if you’re feeling overwhelmed with the vastness of it all. Yeah. And we’re going to continue in 2021 and beyond to see more exploitation of industrial systems, because the adversaries are clearly based on what we see getting very, very good at understanding those systems and having political from them.
LO :Right. Well, Lesley, thank you so much for coming on to the Threatpost podcast and talking more with us about OT security from the threat side and from the how organizations can get more prepared side as well.
LC: Absolutely, it was the pleasure to be here.
LO: Great. And that to all of our listeners. Thank you for tuning in today. If you have any questions or comments on what we discussed regarding ot security, please feel free to comment on our Twitter page at @Threatpost. And we’d love to engage in further conversation around this important topic. So thanks for tuning in and catch us next time on the Threatpost podcast.