A few years ago, a large US healthcare insurance company got surprising results from an internal information security (InfoSec) audit. The new chief information security officer (CISO), who had directed the InfoSec team to conduct the audit shortly after joining the company, was alarmed to find that the firm’s compliance with security standards was very low and the number of vulnerabilities detected in its systems was very high. Change was needed—and fast—to avoid the potentially catastrophic consequences of failing to meet regulatory requirements and exposing customers to a heightened danger of data breaches.
The enterprise needed to build a DevSecOps pipeline to integrate security best practices across all its applications and networks at every stage of the software development life cycle. The task would be formidable. Through an interdepartmental collaboration that spanned a period of approximately 18 months, the company was able to improve its security posture with a zero trust-based DevSecOps solution that used automated tools to reduce risk and increase compliance without slowing development.
Searching for a ‘Just-Right’ Solution
The InfoSec team provided an overview of the core problem: the low audit score represented high security risk for the company. The InfoSec team’s immediate response to the findings was to block deployment of the next version of the company’s core claim management application because the development team lacked the visibility to identify critical vulnerabilities in the pipeline. The application was essential to the company’s operations, as it was used to streamline the medical claim process for the purpose of providing better patient care.
A planning team was quickly assembled to brainstorm possible solutions. The InfoSec group, the development team, and the operations team participated in the initial conversations. Someone from the business team was also involved. Arun Mamgai, now a cybersecurity and data science specialist for a large software integrator, was tapped to function as project lead for the effort. His department manager also participated in the early meetings.
Mamgai recalls team discussions of three possible approaches to remedying the problem:
- Forego internal development and acquire a new claim management software system from a vendor that could provide a high-quality product along with expert implementation and ongoing support.
- Direct existing IT staff to build additional firewalls around the insecure application to reduce the risk level it posed.
- Design a new zero trust-based DevSecOps plan to build security protections into the application design at the earliest stages of development.
The first approach was impractical. Getting a core application from a vendor posed hurdles in terms of cost, customizability, and implementation time. Some vendors offered accelerated deployment and impressive enterprise-level support, but they tended to be extremely expensive. And even if the company had decided to make the necessary investment, none of those solutions would have been ideal. They were not designed to easily accommodate the frequent modifications the organization required to meet its specific business objectives. More affordable applications tended to require lengthy implementations and typically did not come with in-house expertise to accommodate the health insurance provider’s needs for customizations and ongoing support.
The second approach—building more firewalls—would have been comparatively fast and affordable. Firewalls would have eliminated SQL injections, cross-site scripting, and cross-site request forgery. However, they would not have protected against zero-day exploits or complex custom attacks. Also, they would not have eliminated third-party upstream vulnerabilities already deployed in the application. As a permanent solution, building additional firewalls would not have been sufficient to raise the company’s security posture to the desired level.
The zero trust-based DevSecOps solution was considered the best option because it would build security into the application development workflow.The zero trust-based DevSecOps solution was considered the best option because it would build security into the application development workflow. Vulnerabilities could be detected and addressed early so that risk could be reduced without slowing down development or bringing it to a halt.
Assembling the Ingredients
The planning team devised a software supply chain-based solution to be built in house. The resulting zero trust-based DevSecOps solution:
- Was developed in the Node.js open-source, cross-platform JavaScript runtime environment,1 because its support for event-driven and microservices architecture enhances developer productivity and resource efficiencies
- Used Istio, an open-source service mesh tool,2 to run a distributed microservices-based architecture and provide a resilient service mesh architecture along with visibility into the health of microservices
- Used Cosign, a command line tool3 used to sign and verify software artifacts such as container images, to enhance trust in the software developmentpipeline
The solution incorporated role-based, zero trust access. By default, users and devices were not trusted. They had to provide their identities and be approved before they could submit access requests to any microservices in the system.
The zero trust approach also required that the overall software supply chain be monitored on a regular basis. There was continuous assessment of strong identity verification and enforcement of least privilege access for authorized users.
The new zero trust-based DevSecOps solution gave customers the necessary information to identify vulnerabilities when running the application. In turn, the customers were able to relay actionable information regarding detected threats to the teams engaged in building the code and infrastructure. The information gave the developers the visibility they needed to understand which modules were dependent on specific vulnerabilities so they could make necessary corrections.
The solution also provided software composition analysis—by inspecting the package manager, manifest files, binary files, and container images—to identify open-source components included in the codebase. It required that assets be digitally signed and verified.
The solution supplied relevant information about the software supply chain, enabling compliance with the SLSA security framework.4 It generated a software bill of materials (SBOM) while building an image of systems or processes that contained hierarchically structured information. The SBOM could be downloaded and analyzed for vulnerabilityassessment.
Clearing Project Hurdles
The project team faced a number of challenges during development of the solution, including how to assess risk appropriately. The risk for InfoSec might not be the same as the risk for developers, or vice versa. Calibrating the application to avoid false positives and false negatives was a high priority, due to their potentially devastating consequences:
- False positives—Developers can be overwhelmed by the level of noise coming in if they are bombarded with thousands of risk factors. For example, a firewall blocking access to a legitimate website or a network traffic pattern mistakenly identified as a potential threat could prompt the system to flag a completely authenticated and validated application access as a vulnerability. If that happened multiple times, developers would likely begin ignoring the notification because they would not trust the data.
- False negatives—Failure to identify vulnerabilities that are present is hazardous to an organization’s cyberhealth. Bad actors can exploit the resulting loopholes, introducing actual risk. One example of a false negative is a security system ignoring the threat of a malicious email that could allow a virus to infiltrate the enterprise network and cause damage.
To address false positives and false negatives, the project team decided to use the National Vulnerability Database (NVD),5 which serves as a repository of risk data, as “the golden source of truth.” If an identified risk was included in the NVD, it was considered a risk for the application. A risk that did not appear there was dismissed as a risk for the application.
The length of the implementation cycle posed another challenge. The systems had been running an unsecured application, and that had to be corrected as soon as possible. The team expedited deployment by breaking down the zero trust-based DevSecOps solution into multiple phases and adopting an agile-based methodology. That strategy compressed the implementation time to six months. It could have taken more than two years to implement an alternate approach.
Zero Trust-Based DevSecOpsin Action
After four months of planning, six months of development, two months of testing and training, and six months of deployment, the healthcare insurance company had a zero trust-based DevSecOps solution in place that was clearly the right choice for addressing its low compliance/high vulnerability problems.
First, and perhaps most important, the solution secured the supply chain. If the InfoSec team wanted to know how many applications were impacted by a specific vulnerability, the solution could provide that information, enabling the organization to act quickly.
The Log4j vulnerability, which was identified in 2021, had been a big part of the firm’s problem. A popular Java logging framework, Log4j is used by millions of computers worldwide running online services across multiple industries. The Log4j vulnerability allowed attackers to execute arbitrary Java code that resulted in the leakage of sensitive information.6 The healthcare insurance company’s new zero trust-based DevSecOps solution identified which applications were impacted by the leak.
The solution also kept track of the open-source libraries the company used and was able to continuously scan them. Prior to its deployment, developers using open-source libraries did not have a way of knowing if they contained malicious code. Those libraries might have been accessing data without anyone’s knowledge.
A dashboard functioned as a self-service portal to help identify trust factors. Project lead Mamgai offered this example: “Someone from the InfoSec team could ask, ‘Out of thousands of clusters, how many are being infected?’ Or ‘How many have been scanned completely?’ Or ‘What’s the trust factor in those applications?’” Without needing to reach out to developer or operations teams, the InfoSec team could determine the health or insecurity of the application.
With the new solution, users were able to trust the Kubernetes platform—which automates the deployment, scaling, and management of containers—and were thus able to modernize their applications.
Zero trust used to be an afterthought process for developers. It was not until the testing stage that serious attention was given to who could access the application and what risk was involved. With the new solution, zero trust became an integral part of the application design. Instead of being applied reactively after development was done, it was introduced proactively during the earliest design discussions.
Concrete Benefits
The new zero trust-based DevSecOps solution built and implemented by Mamgai and his team delivered some impressive results:
- In six months, the Log4J vulnerability was reduced by 100 percent. This was determined by analyzing the total percentage of systems running the latest version of Log4J, as the vulnerability was fixed in the latest version.
- Developer productivity increased.Developers learned new technology and security principles and incorporated them early in the development process rather than at the end. They were no longer worried about the lack of security practices for their application, because the zero trust-based DevSecOps solution took care of that. Consequently, they were able to focus on alignment with the healthcare insurance company’s businessrequirements.
- Overall costs were reduced. Because it was an internal software supply chain solution, it cost much less overall than a commercial off-the-shelf (COTS) solution.
Conclusion
Security is a primary concern for enterprises as they start adopting digital and cloud technologies. It has become proverbial that a security breach “is a matter of when, not if.” Organizations must proactively develop technologies that safeguard their interests by shifting security toward the left. A common platform that brings developers, operations, and InfoSec teams together enhances collaboration and expedites enterprise software development. Based on the experience of the healthcare insurance company that needed to address the weaknesses uncovered by its InfoSec audit, it is apparent that the zero trust-based DevSecOps project led by Mamgai paid off. Its adoption enabled the organization to go beyond plugging security gaps to enhance its enterprise modernization efforts overall, making in-house software development faster, less costly, and better aligned with business objectives.
It has become proverbial that a security breach “is a matter of when, not if.”Endnotes
1 Node.js,“About Node.js"2 Istio, “The Istio Service Mesh”
3 Chainguard, “How to Sign a Container With Cosign”
4 SLSA, “Understanding SLSA”
5 National Institute of Standards and Technology, National Vulnerability Database, USA, 2023
6 IBM, “What Is the Log4j Vulnerability?”
MICK BRADY
Is a freelance technology communicator with experience writing and editing articles and books on a broad range of topics, including cybersecurity, artificial intelligence, data privacy, and digital transformation.