TL;DR: We disclosed to Chainguard in December 2023 that one of their GitHub Actions workflow was vulnerable to “pwn request”, potentially impacting the integrity of Docker images signed by their cosign Terraform Provider. Fortunately, this ended up being a near-miss incident. We also introduce the Living Off The Pipeline project, which inventories tools used in build pipelines that have RCE-by-Design features.
In addition to conducting Software Supply Chain Security (SSCS) research, my team at BoostSecurity.io handles Product Security. In that capacity a major change we’ve championed early in 2023, was to migrate all of our production Docker images to use Wolfi-based images from Chainguard because we genuinely believe this is the best way to minimize the toil of patching CVEs in “traditional” base images (like ubuntu). We are big fans of what Chainguard does for SSCS, but also adhere to a Trust But Verify approach. As we do for all mission critical software components that go into our product, we like to peek at the Supply Chain security of those upstreams. Over time, we started to automate more of this analysis and scaled up our machinery to cover a larger blast radius. This now makes it easy for us to just point to a given GitHub organization and get Supply Chain insights derived automatically.
For those of us researching GitHub Actions, discovering a workflow with pull_request_target is akin to a shark sensing blood, we tend to get agitated and cannot keep calm until we’ve exhausted all attack paths stemming from that potential Initial Access. That’s why our automation does its best to exclude scenarios where the conditions for a meaningful impact are not present, leaving what’s left to an expert analyst to finish the triage. In this story, we stumbled upon this GitHub Actions workflow in a Chainguard repository used to test a Terraform Provider wrapping cosign, the awesome, modern code signing tool by the Sigstore project. So we found a potentially high risk workflow inside of a high value repository…
Our automation was right, that is indeed a workflow which is vulnerable to the classic “pwn request”, where a workflow triggered on pull_request_target events (as opposed to the much safer by design pull_request) contains a Job which will do a git checkout of the untrusted code coming from a pull request from a fork and also does not explicitly scope down the permissions for the automatic token (GITHUB_TOKEN). This means that, unless the repository or organization level settings were changed to make the token read-only, it would get the default permissions with contents: write , pull-requests: write , etc… That being said, GitHub keeps hardening the default configuration, but is always very cautious to avoid breaking changes in existing repositories or organizations. Looking at the logs of a recently executed job, we could confirm the effective Write permissions.
Here we’d like to introduce what we came to call Living Off The Pipeline tools. Those are seemingly innocuous tools that are commonly used in the context of build pipelines and that have lesser known features that can lead to Remote Code Execution (RCE). The most classic example, often cited along with “pwn requests” is npm install, which will silently execute various scripts contained in package.json when installing dependencies. In the case at hand, the culprit is the go generate command, which executes any command in single-line comments (i.e. //) with the go:generate directive inside *.go files. Needless to say that all the conditions were in place to justify setting up a lab environment to validate our hypotheses.
Now that we know we could plant a backdoor in the codebase of this Terraform provider, we want to see the downstream victims of such a hypothetical compromise. We quickly realized that Chainguard was essentially the only user of that provider and were using it inside of workflows for their mission critical chainguard-images/images repo. We quickly realized that not only were they using it, but that the .terraform.lock.hcl file was absent, which is essential to pin the expected hash of the provider. As recommended in Hashicorp’s documentation, this file should be committed to source control so that the build pipeline behavior is deterministic. Without it, it will just look up the provider by its version (i.e. 0.0.17), which is merely a Git commit tag that the Terraform registry inventories. At this point the only thing standing in our way is the GPG signature that the provider needs to be signed with. The GPG public key is pinned in the registry and passed along to the Terraform CLI as Trust-On-First-Use (TOFU), added to local .terraform for later reference.
If you are a regular reader of our articles, you know how much we love Deciduous attack trees, so it was only fitting for us to create such an attack tree to visualize the the facts we’ve uncovered thus far, the attack chain and the, hypothetical, end goal — to compromise Chainguard Docker images. We demonstrated, in a lab environment, the following attack scenario. We opened a PR from a fork containing malicious // go:generate directives (Stage 1). The directives pulled (using curl) our Stage 2 Bash exploit script. That script created a branch on the victim repo with our backdoor, then force pushed the 0.0.17 tag pointing to that branch as well as poisoning the release artifacts. Additionally, but dependent on Github Actions settings on the target repository and organization, we demonstrated approving and merging to the default branch even in the presence of Branch Protection with a single (bot) approver.
We hoped we could trigger the release.yml workflow which has access to the GPG_PRIVATE_KEY secret, but we were stopped in our tracks by two things. The automatic token (GITHUB_TOKEN ) has two fundamental limitations. First, unlike organization members or repository collaborators with write permissions, the automatic token DOES NOT get the workflow scope which is essential to push changes to files under .github/workflows. If we had workflow scope, given that environment secrets are not used here, it would be possible to exfiltrate the private key or modify the behavior of the workflow. Second, collaborators could, unlike the token, trigger the Release workflow, as originally intended by the maintainers by pushing a new tag (Tag Protection Rules is the recommended mitigation here), but here is where the second design limitation throws us a monkey wrench — it is documented by GitHub that “the automatic token CANNOT accidentally create recursive workflow runs”. So even though we could push a tag, this event is NOT propagated to the GitHub Actions subsystem, thus preventing us from creating a signed release of the provider, shutting the door at compromising the Docker images through that path.
GitHub Actions is becoming the de facto choice for the vast majority of Open Source projects as it is free to run on public repositories. While it is, in our opinion, probably the best choice for Build Pipelines as they have made great design choices and continue to roll out incremental hardening improvements, the fact remains that build pipelines are increasingly complex. This is especially true, when considering so-called Living Off The Pipelines tools that have lesser known powerful features leading the RCE in critical workflows. In the case we’ve discussed here, our attack fell short ultimately by a product limitation initially added to prevent abusing infinite free compute time by crypto-miners, so not truly a security mitigation, but it acts like one in some scenarios.
Thanks to the Chainguard team for collaborating with us so well and taking this seriously and applying mitigations right before the holidays.