After past breaches to similar CI/CD platforms, such as Sumo Logic and Travis CI, those vendors also recommended users rotate their keys. 

CI/CD platforms play a vital role in modern development and cloud-native pipelines. Breaches in these platforms' secrets are serious risks because these platforms often have access to long-lived credentials. If these tokens are compromised, the result can be data leaks, account hijacking, or unauthorized access to multiple cloud accounts. 

A recent NSA report on securing CI/CD pipelines recommended minimizing the use of long-term credentials by using ephemeral tokens, like OpenID Connect (OIDC), where possible. Additionally, the NSA advised organizations to secure their CI/CD pipelines with strong access controls, up-to-date tools, log auditing, security scans, and proper secret management.

In our recent webinar featuring Rob Godfrey, Senior Technical Architect at the Financial Times (FT), we gained invaluable insights into how they navigated the aftermath of the CircleCI breach. The strategies implemented by Rob and his team at FT closely align with NSA's recommendations. Rob shared firsthand experiences, detailing:

• The monumental task of rotating thousands of long-lived secrets. 
• The critical role OIDC authentication played in securing their pipelines. 
• The importance of selecting tooling in the build pipeline that supports OIDC, like Cloudsmith, to ensure robust security measures.
• Implementing alternative processes where OIDC isn’t supported to mitigate risks effectively.

Why do we have long-lived secrets in our CI/CD?

Long-lived keys and credentials are often found at the heart of CI/CD pipelines, because they facilitate access to cloud services for publishing artifacts, software deployment, and cloud resource management.

The following is a classic use case of adding long-lived secrets to your CI/CD pipelines:

• A user is created within a cloud service that integrates with CI/CD, such as AWS or Cloudsmith, with permissions to perform specific actions.
• A secret access key is generated for this user, serving as the long-lived credentials.
• These credentials are then integrated into the CI/CD pipeline in a secret store and used to execute various tasks, like infrastructure provisioning or artifact publishing.

These long-lived credentials often have very extensive permissions, since they need to create, delete, and update resources and cloud services. This increases the blast radius if the long-lived credentials are leaked.  It’s hard to detect a leak, and rotating keys takes time and effort.

Rotate all your secrets

When Rob returned to work at FT after New Year's 2023, his plans for January took an unexpected turn. In his inbox was an advisory from CircleCI, urging the rotation of all secrets stored in their pipelines.

With over a thousand pipelines running in CircleCI and numerous environment variables to inspect, Rob faced a daunting task.

To address the issue, the team at FT took a series of actions:

1. Identifying Affected Secrets: Leveraging the CircleCI-Env-Inspector tool, they enumerated all tokens stored in their pipelines, resulting in identifying over 14,000 environment variables to manage. These were narrowed down to 5,000 that represented long-lived secrets.
2. Assigning Severity and Remediation: Secrets were then categorized based on potential impact, with a plan to rotate critical and high-severity secrets first. Manual rotation proved laborious, requiring weeks of effort from a team of 30 to 40 engineers, but they eventually got through all of their secrets.

Enhance the security of your pipelines

After evaluating lessons learned, FT embarked on a set of long-term initiatives to fortify pipeline security:

1. OIDC Integration: The best way to avoid the risks of long-lived secrets is to not use them at all.  FT adopted OpenID Connect (OIDC) wherever possible, since OIDC enables use of short-lived authentication tokens.
2. Automated Key Rotation: Where OIDC was not supported, FT implemented automated key rotation using Doppler. 
3. Internal Infrastructure Inventory: FT enhanced their internal inventory system to track secrets and their responsible owners, ensuring better visibility and management.
4. Build Tool Selection: FT made changes to their build tools, opting for tools that are built with security as the simple or default mode for developers. It was at this time that Cloudsmith was selected for artifact management, based in part on our support for OIDC.
5. Training and Awareness: FT built training and awareness programs to educate employees about security best practices and response procedures.

After the initial shock of the CircleCI breach, Rob and the team at FT realized that a similar breach could happen to any of their vendors. 

In response, they proactively fortified their CI/CD pipelines against future breaches. Embracing OIDC authentication and selecting secure tooling like Cloudsmith, which supports OIDC, became cornerstones of their long-term security strategy. 

Through these decisive measures, Rob and his team transformed adversity into an opportunity, ensuring their pipelines are robust and resilient against evolving threats.

To learn more watch our webinar with Rob about how the Financial Times recovered and thrived after the CircleCI breach.

Liked this article? Don\'t be selfish (:-), share with others:   Tweet