Editor’s note: A recent case study explored how a routine audit revealing hardcoded API keys in a multinational conglomerate's CI/CD pipeline led to a transformative revamp of their secrets management, highlighting the importance of addressing blind spots for increased security, efficiency and agility. See a Q&A interview below with case study author Ed Moyle and Shilpi Mittal, lead IT security engineer, who explore some of the case study’s main themes.
Ed Moyle: Can you walk us through the incident where API keys were discovered hard-coded in the pipeline? What was the potential business impact?
Shilpi Mittal: The incident that really brought the issue to attention was during a routine internal security audit of our CI/CD pipelines. Our DevSecOps scanning detected several build configurations that contained hard-coded API keys these were service credentials used to talk to external systems and internal microservices. They had been put in temporarily by developers to accelerate testing but were never removed.
Technically, they were plaintext in pipeline configuration files, and anyone with repository or runner access could have extracted them. If an attacker had insight into those builds or compromised a developer account, those credentials would have granted unauthorized access to production data or cloud resources. At worst, it would have led to data exfiltration, service disruption or even regulatory non-compliance with standards such as SOX or PCI-DSS.
The moment the issue was found, we addressed it as a high-severity incident. Our first response was to rotate all affected credentials, audit similar repositories for embedded secrets and add pre-commit scanning to prevent recurrence. More importantly, though, this incident served as a wakeup call. It made us recognize that our then-current approach to secrets distribution would not scale. That recognition served as the catalyst for our company-wide secrets management modernization.
EM: How did the audit finding on inconsistent credential use influence urgency or executive buy-in?
SM: The internal audit finding was really the tipping point that moved the discussion from a technical problem to an organizational priority. The auditors highlighted that our credential management wasn’t standardized across business units. Some groups rotated secrets on a schedule, others stored them in local config files and a few had static credentials with no obvious ownership. From a compliance standpoint, that lack of standardization meant that we couldn't demonstrate clear effective control over privileged access, which is a basic expectation under SOX and PCI-DSS. When the audit team presented their report to management, it was clear that this was not just about security hygiene; it was a governance and regulatory exposure issue. The risk statement was straightforward: without centralized visibility and lifecycle management of credentials, the firm faced audit deficiencies and, by extension, reputational and financial risks. That context immediately attracted executive attention.
EM: What was the role/purpose of the legacy secrets management platform you mentioned in our prior discussion?
SM: The legacy secrets management platform was originally deployed to centralize control over all application credentials, certificates and API keys across our environments. The intent was sound. We wanted a single, secure vault where sensitive information could be stored, encrypted and managed rather than having developers embed secrets directly into code or configuration files. It also provided basic capabilities like access control policies, audit logging and static secret rotation.
For several years, it served as the backbone for protecting credentials in both our on-premises and cloud environments. It gave us encryption-at-rest, a centralized access layer and a way to enforce least-privilege principles through role-based policies. Essentially, it helped us move away from scattered secrets in pipelines and repositories toward a more structured model.
However, as our application footprint and automation matured, the platform’s original design started showing limitations. It wasn’t built for the scale, velocity or multicloud integrations we operate today. Over time, maintaining high availability, synchronizing instances and supporting continuous integration workflows required more manual effort than the system was designed for. That’s what ultimately drove us to re-evaluate and pursue a more scalable, cloud-native approach.
EM: What limitations did it present (scalability, operational overhead, licensing, usability)?
SM: The legacy platform worked for us in the early days, but as our environment grew, several limitations became obvious. Scalability was the first problem; each new business unit or environment required its own instance, its own replication setup and its own maintenance cycle. This added layers of complexity every time we moved into new cloud regions or introduced new applications.
Operational overhead was also an issue. The system had to be patched constantly, undergo manual failover testing and undergo continuous monitoring in order to provide high availability. We had to invest engineering time to simply keep the service up, which ran counter to our automation-first philosophy.
Usability-wise, the platform was not adapted to the pace of modern DevOps. Developers found the request and approval workflows cumbersome, leading to ad hoc workarounds that bypassed the vault. The friction slowed delivery cycles.
Finally, the licensing and infrastructure costs increased as we expanded. New environments or clusters meant new servers, storage and operating costs. As time passed, it became apparent that the model wasn't feasible, either technically or financially, for a rapidly expanding, multi-cloud environment. That set us on the path to a more services-based, flexible architecture that could scale elastically without the costly operational footprint.
EM: What was your decision process for selecting the new solution?
SM: The decision process was structured and collaborative from the beginning because we wanted to make a long-term, strategic investment, not just replace one tool with another. We started by forming a cross-functional working group that included representatives from security, DevOps, compliance and architecture. This team defined the business and technical success criteria: automation readiness, scalability, zero-trust design, strong integration with identity and delivery systems, and clear auditability.
Once requirements were finalized, we conducted a proof-of-concept evaluation where multiple platforms were tested side by side in a controlled environment. We focused on how each handled key areas like dynamic credential issuance, role-based access control, secret rotation and integration with existing CI/CD pipelines.
Parallel to the technical evaluation, we performed a risk and compliance assessment to ensure each candidate aligned with frameworks like SOX, PCI-DSS and our internal security standards. Cost modeling was also performed early comparing total cost of ownership, operational burden and future scalability.
The final step involved gathering feedback from pilot teams and documenting measurable outcomes such as ease of integration, performance and administrative overhead. These findings were presented to our internal governance board, which approved the platform that demonstrated the strongest balance of security, simplicity and long-term adaptability. The result was a unified decision supported by both business leadership and the engineering team.
EM: How did you manage the rollout?
SM: Adopting a centralized vault strategy caused us to think about architecture in totality – not just where secrets are stored, but how they are moved, who accesses them and how you build resilience and trust between systems.
Security by design was the very first consideration. We sought a model that gave us zero standing privileges and prevented any single point of failure. That meant using identity-based authentication and temporary, dynamically created credentials instead of static keys that persist in memory or in config files. The second was segmentation and fault isolation. We designed the vault architecture to operate in logically separated environments for production, non-production and disaster recovery. Each had its own access boundaries and policies so that an issue in one realm could never spread into another.
High resiliency and availability were also top priorities. We needed a distributed deployment pattern that would withstand outages, require minimal maintenance and gracefully recover from regional failures. The architecture had to support multi-cloud redundancy without us having to manage complex replication manually.
The other major requirement was integration with our identity and automation systems. The vault needed to trust the same sources of identity that our organization already trusted our IAM and CI/CD systems, and use those trust relationships to authenticate workloads automatically. Secrets could then be issued and revoked dynamically without the need for human intervention.
Finally, we prioritized observability and compliance visibility. All access requests, secret retrieval and rotation events had to be fully auditable and exportable to our enterprise monitoring and logging tools. This design choice ensured the solution not only secured secrets it also provided the traceability required for audit readiness and governance reporting.
