SolarWinds CISO Tim Brown spoke with ISACA Board Director Rob Clyde at the ISACA Conference North America last month about lessons learned from the high-profile software supply chain incident in 2020. Find out what Brown and his team learned from the incident and how they have recalibrated their security approach since then.
RC: Tim, can you share some tips on how you learned about the incident and how you break this kind of news to the public at large, shareholders and the board?
TB: The SolarWinds CEO was called by the CEO of Mandiant the morning of December 12, and I called the Mandiant CTO for specifics. Code was discovered by FireEye Mandiant in an investigation. The code attempted to do some things that Orion never should do. Once they saw that, they started digging in. They decompiled it and saw it didn’t look like our code and immediately gave us a head’s up that we were looking at a supply chain attack.
It turned out to have affected three versions of product produced between March and June 2020, and 18,000 customers had downloaded those versions. Later, it turned out that 100 were possibly affected, but we initially responded as though it was 18,000 to get our customers information as quickly as possible.
The initial research was able to be done very quickly. Once we proved it was true, we had calls with leadership, the board, legal and everyone else we needed to speak with very quickly. We were working remotely due to the pandemic but went to the office and created war rooms to do the necessary research, communications and customer outreach. We realized we had sales contact information and not security contact information, which is something we’ve fixed.
Communication was critical for us. Priority-wise, it was how do we prioritize the customers? We were having a bad week, but our customers around the world were having bad weeks too, trying to figure out if they were running infected versions.
RC: It can take a while to understand the full magnitude of an incident, often more than the new SEC rule of four days. How do you deal with that?
TB: It’s a matter of exposing what you know and what you don’t know. You don’t want to overexpose or underexpose. It was 2 a.m. on Sunday when we published what we knew and what we didn’t know. At that time, we didn’t know the attacker or who to attribute it to.
Don’t overstretch or go beyond what you know—just share what you do know. We decided to be as transparent as possible—with what we knew, with who was affected, with the affected versions and with sharing code.
RC: Who was involved in the response to this situation?
TB: Before this incident, we’d use normal incident response exposure models. This one was just so big and so intense. Our legal partner has a very, very good cyber team, and that cyber team acted like the quarterback. The quarterback cannot have other jobs to do and needs to focus solely on the incident, so it was helpful to have the legal team in this role. CrowdStrike and KPMG did investigations and forensics. Internally, our legal, engineering, security, IT marketing and customer relations were all involved. We often stayed until 3 in the morning—that is what you have to do in these types of situations.
RC: Did you see any cases of organizations using this as an opportunity to whitewash their own sins and blame it on SolarWinds?
TB: I felt outmarketed and outgunned by a lot of folks. A number of motivations were in play, and they were very strong and very real. But commercialism never bothers me as long as the research behind it is good.
RC: Who was the attacker, and what was the motivation?
TB: Attribution was not what we were looking for. We were busy trying to help the customer. The US government attributed it to Russia SVR [Foreign Intelligence Service of the Russian Federation]. The characteristics of the attack said you had to be able to connect to the internet and a command and control server. The majority of customers were not configured that way, so we believe the actor was after a certain set of customers, like government agencies. They could have written code to just do harm or go after IoT systems. They specifically said you have to be connected to the Internet to get command, which is a rarity around Orion, so we believe they knew their targets, and we were the route to those targets. We were a means to an end, and it was a very sophisticated and motivated actor.
We don’t have exact specifics on patient 0. For over a year, the adversary had access to the system. We know that they did very specific spear-phishing against very specific people to collect data. The amount of noise they did in our environment was very minimal. They came in, they got access to our email, they did a test run in October with no code, they came back in March and put some code in, and in June they came back and removed that code. They were mission-centric to not be discovered.
RC: What was your security culture like prior to the attack, and what is it like now?
TB: When this came out, we were like, “What did we miss?” We had a very good program. We were spending a little more than industry standard. But now we’re significantly above standard.
Leadership was incredibly supportive. I was asked, “Tim, what did you need to be exemplary?” Exemplary costs money and takes investment in development process, security team, everything across the board. We got those resources we needed and started security by design, which is people, process and technology, and made that a cultural essence of the company.
RC: Often the first instinct after an incident is to fire the CISO because someone has to be held accountable. You’re still here and with SolarWinds, so obviously that didn’t happen. Why not?
TB: A number of reasons. I’ve done a number of different things throughout my career, and I’m very comfortable talking to people—from Fortune 500 companies to conferences like this one. With an incident of this magnitude, you don’t need a background CISO. You need someone who can take the hard questions, get yelled at, be blunt and take ownership. If I wasn’t me, I would fire me. If you were a background CISO, you wouldn’t have been able to be helpful. You would have been helpful on the side, but you wouldn’t have been able to get in front of the problem. You need a CISO who can be outward-facing in this situation.
Also, I was in the middle of it. I talked to media, press, governments, industry forums, customers, nations. If you’re helpful, if you’re playing a central role in finding the solutions, you won’t be fired. When asked why he didn’t fire me, my boss said at a conference “If I was hiring a CISO, I’d hire Tim, so why would I fire him?”
RC: My guess is you can’t just suddenly decide to learn how to be an outward-facing CISO right after an incident occurs. What’s your advice for people about how to be prepared in advance?
TB: Getting out there and talking. CISOs now have products and conferences and groups—get out there and talk. Get training, run through scenarios. The best training we had was speaker training earlier on in my career. And after you get the training, keep practicing. Talk to industry forums and boards.
Remember that your customers are going through as much as you are. It is not just you. I had to wreck so many people’s Christmases and New Years.
RC: You said “I had to wreck” those holidays. You really took accountability.
TB: I ran a good program. But my program was not sophisticated enough to combat the Russian SVR. It just wasn’t. Was I doing OK? Yes. But did I want to do more? Absolutely. Our security program was on par with the industry. But now we’re not on par. We’re exemplary. We stopped development for six months to accelerate a lot of different efforts. Now we’re doing things like triple SOCs, we have our own red team.
RC: Give us an idea about how you convinced and persuaded to go on this cultural and budget journey with you.
We had great support. We had a planned CEO change in the middle of this. Our former CEO had said in August that he was retiring, and our new CEO started January 1. We had full support from the CEO and the board.
RC: How did you rebuild trust with your customers?
TB: A lot of the effort has been things like this ISACA conference—forums, where we’ll share everything. At first, we would share information with customers but were reluctant to share all of the details. Now we will share as much as they want and much more. That has changed how vendors are evaluated by all of our customers. They are now asking for more detail because we set that precedent. In setting that precedent, we showed our customers that they could trust us.
More transparency throughout the supply chain is critical.
RC: How likely do you think it is that this is happening in other organizations?
TB: We’d be naïve to think this isn’t happening in other places. Do I think it is probably in some other organizations? I’d be surprised if it wasn’t.
RC: What is a key lesson you want us to take away from this?
TB: Always practice your incident response practices. Have people ready. We did incredible things in two days, but we had to have those contacts. This happened on a Saturday. Make sure you have the right contact points to reach people beyond the office.
It is OK to talk. The more you can share, help your customers and help your community, the better off everyone will be.