Blog Articles

Opening Pandora’s box - Supply Chain Insider Threats in Open Source projects

François Proulx

Spotting an Insider Threat isn't always as obvious!

TL;DR: Granting repository "Write" access in an Open Source project is a high-stakes decision. We delve into the risks of insider threats, using a responsible disclosure for the AWS Karpenter project to demonstrate why strict safeguards are essential – branch and tag protection, code review, and especially controls around the publication of release artifacts. Also GitHub may be lacking in terms of auditing capabilities to help spot Indicators of Compromises (IoCs) in some scenarios.

Trust, but verify: Setting the stage

Traditional information security models often address insider threats by defining clear boundaries between corporate secrets and public information. In contrast, Open Source projects embrace transparency and collaboration. However, this doesn't eliminate insider threats. The project's reputation and the integrity of its software artifacts remain vital assets. As a core maintainer, granting dozens (or even hundreds) of contributors "Write" access requires careful consideration of risks. Segregation of duty remains crucial – identifying privileged actions and limiting them to a select subset of collaborators.

A responsible disclosure opens Pandora's box

As we mentioned in a previous article, we like to probe at the dependencies we consume in our own supply chain, which led us in November 2023 to peak at Karpenter, an Open Source project started and primarily maintained by AWS that facilitates dynamic provisioning of compute resources in a Kubernetes cluster. Given its privileged role within Kubernetes, Karpenter demands a high level of trust. As we discussed, we have a large scale data infrastructure that helps us identify supply chain issues in Open Source packages at scale. Once we’ve identified a certain vulnerability class, we can quickly spot all similar issues across other projects. In this case, we started by looking at workflows in the project and a curious case of a workflow injection popped fairly quickly. It was obvious that this one was only relevant for insider exploits, though.

“Life is like a CTF”

Looking at this workflow injection immediately brought to mind a quote once heard amongst fellow Capture The Flag (CTF) players: “Life is like a CTF”. Sometimes vulnerabilities seem almost designed to challenge security analysts. In this case, the challenge was to craft a malicious Git tag that could be pushed to GitHub.com. Successful exploitation required injecting valid JavaScript through Git tag within the constraints of a workflow step, ultimately leading to unconstrained RCE. The prize? Access to the GITHUB_TOKEN with repo write permissions as well as the ability to tamper with Karpenter Helm Charts releases by abusing authentic OIDC claims granting access to an AWS ECR registry.

Crafting a first POC

Constraints and Considerations

Tag Length: A maliciously crafted Git tag must remain under 255 characters (including refs/tags/) and, in this case, begin with a semantic version format (v*.*.*).
Character Restrictions: Standard Git tags are limited to alphanumeric characters, hyphens, underscores, and periods. This prevents the direct use of many special characters commonly seen in exploits.

Just a POC

This is the context in which the tag is injected:

const result = await github.rest.pulls.create({
  title: 'chore: Release ${ {steps.tag.outputs.TAG }}', 
  owner,
  repo,
  head: 'release-${ {steps.tag.outputs.TAG}}',
  base: 'main',
  body: [
    'Stable Release Changes for ${ { steps.tag.outputs.TAG }}.',
    'Please disregard this PR if it is for a patch release.',
    'Please remove the branch after merging.',
    'This PR is generated by [StableRelease](https://github.com/aws/karpenter/actions/workflows/stable-release.yml).'
  ].join('\n')
});

First, we need to escape out of the Javascript string with a single quote. Single quotes ('), along with other special characters like spaces, double quotes ("), and non-printable characters, are usually avoided in tags, however, Git itself does not strictly enforce these naming conventions at a technical level.

Let’s start with validating that this works:

$  git tag "v1.2.3'+console.log(333)+'"
$  git push origin "v1.2.3'+console.log(333)+'"

Yup, we have RCE! Moving on.

Elevating privileges

The true goal is arbitrary remote code execution (RCE). In this context, RCE would allow us, for instance, to use the aws CLI (already present on the GitHub Actions runner) and benefit directly from the ephemeral AWS ECR credentials freshly fetched by the workflow via OIDC (id-token: write) immediately before the injection point.

To simplify our exploit development, let's add some scripting to generate a random tag and use Base64 encoding. This allows us to bypass character restrictions within the tag:

https://gist.github.com/fproulx-boostsecurity/90f07cd2987f63901d39b3ec51bea7c7

The full exploitation chain

As mentioned, this workflow leverages OIDC for access to an AWS ECR registry. Though we didn't test a full exploit, we're confident based on reviewing GitHub Actions logs that a threat actor could've done so.

Imagine a nation-state actor who gains the maintainers' trust over time through consistent contributions. With Write access, they're free to create, push and immediately delete the malicious Git tag (unless Tag Protection rules are in place). They can then clean up, by deleting GitHub Actions logs, leaving minimal traces. This is true as well for many classic Insider Threats in enterprises.

Indicators of Compromise (IoCs)

GitHub Public Events: Malicious tag creation/mutation/deletion might be spotted (e.g. https://api.github.com/repos/aws/karpenter-provider-aws/events )
Audit Logs:
- Deleting workflow run (logs) would leave a workflows.delete_worklow_run event.
- Tampering of release artifacts (Adding, Updating, Removing), on the other hand, WOULD NOT leave any audit log event.

A sophisticated attacker would certainly use a multi-stage payload, where the malicious tag fetches a secondary script containing the true malicious payload. Even more alarmingly, GitHub release artifacts can be tampered with using basic Write access (that GITHUB_TOKEN had permissions to in our disclosure - i.e. contents: write). This poses a severe risk to projects like Homebrew taps, Terraform modules, goreleaser and others that rely on GitHub release artifacts integrity. Interestingly, using cosign keyless signature with Fulcio transparency logs could potentially help spot strange tags.

Limiting the blast radius: Effective mitigations for Insider Threats

Karpenter’s GitHub Actions workflow vulnerability highlights the need for Open Source maintainers to implement robust safeguards against insider threats. Here are key strategies to mitigate risk:

Implementing strict Branch and Tag protection

Treat branch and tag protection as your first line of defense. Enforce policies that prevent unauthorized code changes. Require status checks, mandate code reviews by at least one senior contributor and carefully restrict who can push tags or create or force push branches.

Enhancing code review processes

A rigorous code review process is vital for maintaining the integrity of the project. Maintainers should ensure that all changes, even those from trusted contributors, undergo thorough scrutiny by a fellow reviewer. This practice not only helps catch potential security issues but also encourages a culture of accountability and transparency. Leverage .github/CODEOWNERS file to ensure CI workflows and all of their dependencies (scripts, package manager manifests, etc.) are always reviewed by trained contributors. Prefer this exact file location to avoid attacks exploit the search order for the various locations CODEOWNERS can be located under.

Tightening controls on release artifacts

Release artifacts are highly sensitive. Implement strict controls and automated checks to verify their integrity before release. Leverage signatures and cryptographic hashes to ensure artifacts haven't been tampered with. It could be worth implementing out of band checks in workflows in a separate repository so that not all GitHub Actions jobs run in the context of the repository where you have numerous direct contributors.

Securing workflow automations

Minimize the use of secrets in GitHub Actions and other CI/CD workflows. Require explicit maintainer approval for actions needing elevated permissions. Carefully manage secrets ($) and follow GitHub's best practices for environment protection and OIDC trust configuration. Regular inspection of workflows, review of audit trails and events as well as running static code analysis tools out of band on build pipelines, are crucial.

Leveraging security assessment tools

Proactively analyze your project's security posture with tools like OpenSSF's Scorecard. Its actionable recommendations will help you identify areas for improvement to harden based on common Best Practices, though not necessarily specialized for insider threats.

Understanding your Build-time dependencies

Dependency choice and review is, of course, a key aspect of supply chain security. Use SCA tools that track Build-time components to assess the security of potential dependencies (direct and transitive). Evaluate results carefully to make informed decisions, choosing secure options or advocating for improvements upstream when your dependencies might affect your security posture.

Conclusion: A call for vigilance

Open Source software development thrives on collaboration, but the Karpenter case underscores the ever-present risk of insider threats. The SLSA framework offers a solid foundation, emphasizing provenance, tamper-proofing, and secure environments. Yet, insider threats demand even more specialized safeguards to maintain community trust.

The OpenSSF and similar initiatives are vital for establishing best practices and guidelines to counter insider threats. Existing resources like their "Source Code Management Platform Configuration Best Practices" provide a foundation, but we must expand them to cover more persistent threats. Secure environment secrets management and strict adherence to the principle of least privilege within OIDC integrations are, for instance, essential additions.

Timeline of disclosure

Disclosure to AWS: 22 November 2023, 20:22 UTC
AWS Response Received: 22 November 2023, 22:25 UTC
Initial Mitigation Implemented by AWS: 24 November 2023, 14:59 UTC
Ongoing Improvements by AWS: Between November 2023 and March 2024
Release of This Article: March 2024

Bonus

To clarify, while we talked a lot about tags, the same exact type of injection would work just as much with malicious branch name (so any Git ref). This is documented by GitHub.

An interesting quirk of workflows triggering on tags being pushed, is that as long as you have such a workflow present anywhere in the Git history of a branch, a malicious insider that can push tags, when Tag Protection is missing, and trigger the vulnerability again. We came to call this the evergreen scenario. You just need to push a new tag, which points to the old commit SHA with the vulnerable workflow! That’s also documented, though quite cryptically, as the official behaviour. Go protect your tags, now!

While doing this responsible disclosure, we discovered many other, very similar vulnerable workflows for other Open Source projects (almost always release.yaml, which often trigger on Git tags), so we're sharing other payloads as Gists to help fellow researchers: