It might sound funny, but these were a few talking points that came up last week during runZero’s two hosted fireside chats, where CEO and Co-Founder Chris Kirsch sat down with Lares CEO Chris Nickerson on Tuesday and then Fortinet Systems Engineer Roger Rustad on Wednesday.
If you’ve had the pleasure of hearing Chris Nickerson tell his pentesting “war stories,” you might already know some of the references here. But for first-time listeners, these narratives cover the potential dangers of a red team member’s (mis)adventures, and the role of asset inventory from an attacker’s perspective. As for natural disasters and time machines, our talk with Roger elaborated on his work with the Information Technology Disaster Resource Center (ITDRC), as well as his view on how runZero’s solution has been helpful to the incident response and forensics teams at Fortinet.
Chris Nickerson Recap #
The first fireside chat began over margaritas as Chris Nickerson (CN) joined Chris Kirsch (CK) on stage at our pop-up venue, the runZero Cafe, on Tuesday, April 25. Their chat covered:
- Why the recon phase is an important stage in pentesting
- The human element (and fallibility) of IT and security
- What tools Chris Nickerson uses in his pentesting
And sprinkled humorously throughout the dialogue were moments from Nickerson’s past exploits, including welding people to cars with killer robots.
Specifically, CN talked about how recon (for attackers) and asset inventory (for defenders) are two sides of the same coin. In answer to why the recon phase is important, he noted,
Video transcription
CN: "First off, right, Karate Kid Rule. Man can’t see, man can’t fight. Same exact words for any attacker. I can see things that you can't see. Good luck. And if that's what I'm looking for, right, I'm trying to find those lapses in visibility."
"So in general, right, when you're thinking about making a process in testing, it's not always like the voodoo magic and you just sprinkle your hacker dust and then magically like you win. It's a bunch of really crappy work."
"It takes a ton of time and you have to have a lot of process into it, it's not just a hit a button, hope that it expands to find the things. You have to catalog every single thing that you see and be able to start to index and understand this information and what starts to emerge is patterns, right."
"You start to see, oh, this is kind of where all the old stuff lives; this is where some cool, new stuff lives, this is where some I have no idea what this is. That might be interesting at some point. You might find names. You start to find, you know, indexed pieces of not only networking infrastructure, but I mean, engineers are good. They have naming conventions so that when somebody is like, hey, they want you to steal financial records, it's like D-E-N, Denver, F-I-N, financial, and then like a bunch of numbers and you're like, oh it's probably this server you know, like."
"So as you start to get yourself familiar, it's more about situational awareness to figure out what you're going to do in forward operation then it is go find a vulnerability, scan for something, exploit it, you know, move on to offense success, It's really about that process of getting that total view of the landscape because you kind of can't make plays on the field unless you know where the boundaries are."
In answer to what tools he’s using:
Video transcription
CK: "So how do you, how do you go about that? When you go on a pentest, what are your tools to figure out what's there, information for your pentest..."
CN: "So obviously lots of things, right? Because we have a great relationship being able to use runZero in that capacity, I think it's great, especially in massive networks. Because what you find is, you know, in a smaller network I can get a relatively high degree of success, if I'm just using basic, you know, nmap engines and I'm going to be able to find, you know, the scripts that I'm using to to be able to pull information."
"You don't get that rich bit of information. right? I know that the host is up, I know that these ports are open. I can probably go grab banners, but now I have to like grep through a bunch of shitty text files. And it's not super useful. Whereas if all those things are indexed, they are in a searchable database, you have ways to look at that information."
"It's now what's there, what's available, what's running, what version is it running? What other things can I start to collect and find out about that box?"
When it comes to testing more fragile environments, CN delved into the problem of legacy technologies lacking resiliency, and the importance of not only understanding the environment as a pentester, but also ensuring companies know what’s on their networks, including “what’s old and going to misbehave.”
As an example of misbehaving machines, here’s CN’s killer robot story:
Video transcription
CN: "We were working on manufacturing facilities, right? And the robotic welding arm things, right? Cool robots are just tech world stuff. Their TCP/IP stack was awful. And it's, like, I don’t know, somebody from the eighties built it. And it’s just half-open connections that make it harder for people. And I say that like in the most loving way, because like I portscanned it just started !@#$!@#, and just started shooting welds in the air going like this and I was like, ohhh shit, you know, like, I guess I didn't know but like the..."
CK: "Just to be clear, this wasn’t with runZero?"
CN: "No, no, no, no. No, this is bad scripts that, Chris, again 24 times unsafe, 25th time unsafe. I was like try three and it was now trying to kill people. So again, you know, like those types of tools, whether it's like the idiot guard for me, which, probably need it more often, especially now that I'm older, but but being able to understand and how you can interrogate a box safely is it's the hardest thing of testing because if you're wrong, you're really wrong."
"Like it's a super super bad moment because the whole thing that you're like, oh, I found the one box that I can compromise. Oh, yeah. Just turned it offline. That's it, start over, like two weeks of work gone."
While many companies understand how critical asset inventory is, they still face challenges when trying to implement it; they often lack the knowledge and resources to do it effectively. However, CN points out that if you have the proper tools, you can avoid making tragic mistakes:
Video transcription
CK: "Here's the thing that kills me, you know like, for a lot of that infrastructure. OT and also like the ERP system and those kinds of things, it's like it's both, this is absolutely critical for the business to survive, and this is so fragile and you can't touch it and never touch it. These two things don’t makes sense to me."
CN: "But this this is but this is where I appreciate the approach that's been taken with runZero because they think that not not only are we looking at this like central source of truth and system of record, but the idea that the logic is built in for the grouping and for some of those things starts to create that that map of of where severity could be without having to get into them, you know, robots killing people."
Yes, getting those parameters is important, and luckily, runZero can give you that right out of the box.
As a final note on the importance of asset management, CN told us:
Video transcription
CN: "I’ve also worked in a lot of other enterprises and consulted all over the planet, and everybody's trying to change stuff in their network. Well, if I can just come in and give you an inventory. But let’s say, I mean, even if I'm a tester or I just run the network or I'm part of ops in engineering, if if what I can do is come back because you hired this, like, whatever some $4 billion consulting company to come in and like, upgrade your SAP system, they're going to be like, oh, give a map of everything and the people who run it will give them the maps of like a couple of interfaces and then everything else won’t be there."
"But if you can add value to go back and go, oh, this is absolutely every single thing that we have that as a SAP vendors, be able to group them, be able to categorize them, be able to explain to them that like, well, this one was from the 90s, this one was from the 2000s, all of them don’t follow the naming conventions, half of these aren’t in DNS."
"Like you're now making a graceful transition, which is huge because being a consultant, like the worst problem is information right? And if you can do that, you can give them accurate inventory, like they might actually get the job done on time. Probably never on cost, but at least quickly."
So happy hunting to you, Chris! And many thanks for your entertaining insights on asset inventory from an attacker’s perspective.
Roger Rustad Recap #
During Roger and Chris’s fireside chat, we heard about Roger’s journey in finding an asset management solution both for Fortinet and the volunteer group the Information Technology Disaster Resource Center (ITDRC).
Video transcription
CK: "Now for asset inventory. I think you, well, you brought in runZero, that's why you’re here. But can you tell us a little bit about how you were doing asset inventory before you brought in runZero?"
RR: "I think probably the easiest way to put it is very poorly. We leveraged a lot of open source tools, mainly the command line tools, you know, nmap and mass scan are kind of something we use regularly. And we went through a lot of logs manually, you know, to go back and try to find things. I think that became very laborious. And doing our threat hunting sessions one time we had to kick off an nmap scan that was going to take forever. One of us said there's got to be a better way than this. and so we started Googling and found you guys and here we find ourselves today."
Roger elaborated that other methods and solutions involved waiting for results, and interpreting the data – even though there was often consensus on his team, sometimes the interpretations got lost in translation when presented to other teams.
As Roger and his team looked to find different approaches to the problem, they looked at attack surface management solutions. Unfortunately, many of these tools require agents or APIs, and because Fortinet is more of a hacker culture internally, they preferred command line tools. They wanted to start there and wanted something that started there, too. He noted that runZero’s agentless solution made it very easy for his team to get a quick 30,000 ft view and then trim it appropriately.
As for first steps on how they began their runZero journey, Roger stated,
Video transcription
RR: "Literally, we just downloaded it and played with it. Each one of us ran it in our home network and we were just amazed at what it found. You know, we liked the fact that you can export everything straight into nmap format or XML format or interact with the API. I think that made it really easy. Then it was really just kind of figuring out how we were going to start implementing it internally."
Once they had runZero up and running, Roger provided some insight into how the solution has been helpful in specific use cases:
Video transcription
RR: "Yes. So oftentimes we need to find an owner of an asset. I mean, everyone has the challenge of on certain networks finding owners is difficult. The extra information that we can look through or see who maybe was on that IP first. You know, I don't think of runZero so much as an asset tool but sometimes as a time machine where we can go back and see who was on that network or on that device at a particular time. That's been incredibly helpful for our incident response and our forensics team."
CK: "How do you, give me an example of when you have an incident that you are investigating, how would you leverage runZero in that respect?"
RR: "So there could be a time in which we saw that a certain IP, let's say, certificate on an IP, we could see what the certificate was. We could then pull that certificate and pivot across and see who else had that certificate."
"I think when it comes to our FortiGates, we can tell by that type of certificate what version it is, what this may be running, and then that's helped as we’ve gone through and patched certain things. Just seeing them, getting more details. But even the web page itself, being able to get a screenshot on that web page has been really helpful with runZero."
We’re so glad we could help you at Fortinet, Roger, but we’re also happy to help with your work at ITDRC. This volunteer group is a nonprofit that builds IT solutions in areas affected by disasters, with no cost to the communities using these solutions. Roger explained that a lot of the work involves setting up simple connectivity, including setting up satellites and access points so first responders, shelters, kitchens, and churches can have access to their networks.
How does runZero help the ITDRC?
Video transcription
RR: "And runZero has been really good for helping us kind of figure out what's on the network before we put stuff on, once we put stuff on. We often forget where we put stuff because as you can imagine, asset inventory is a bigger pain in the butt. Whenever you're, you know, it's a volunteer thing at the end of your day that you’re not keeping good tabs on."
And for how the ITDRC plans to use runZero in the future:
Video transcription
CK: "When you think about how you want to mature and evolve that, looking to the future for disaster relief, etc., how are you planning to use runZero in the future?"
RR: "So I think, you know, one thing we're starting to see is, as we start to partner with bigger companies like ZPE and other companies, we're starting to leverage edge compute devices a lot more."
"So the fact that runZero can run on such a tiny footprint becomes really helpful in figuring out what else has been added or taken off of the network. As we start to at some of these sites, do things like check the fuel levels of the generator or check the voltage level of the battery, we can do all that right off of runZero console access."
"So as we start to do those things, it just makes sense to just throw a container on it, just see what else is on the network and it might be compromising. So I think when we talk about security for a lot of our other projects, you know, the CIA triad, the one we're most concerned about is availability. The others don't matter so much, and we kind of see runZero being really helpful for just making sure things are up and we know what else is running on the networks that we kind of throw out spontaneously."
With all of the work that Roger does, we’re so happy that we can take off some of the strain in both his day-to-day job and volunteering. Thank you, Roger, for chatting with us during RSA!
RSA Venue Recap #
In summary, the runZero team had a great time at our venue during the RSA conference, and we were grateful we could host these informative discussions with Roger Rustad and Chris Nickerson. We were also glad we could welcome many other cybersecurity professionals throughout the week to join us for drinks, tacos, digital caricatures, and faraday bag giveaways.
If we were lucky enough to see you at the venue, thank you for stopping by! We hope you had a wonderful time!
If you are interested in learning more about how runZero can help your company with cyber asset management, please let us know by reaching out via our contact us form.
