Last fall, my security research team at BoostSecurity published two articles on supply chain security, initiating an in-depth exploration of the Supply chain Levels for Software Artifacts (SLSA) model. Our first article, “SLSA dip — At the Source of the problem!” concentrated on Source Control Management (SCM) systems like GitHub. There we analyzed the role of SCMs in the supply chain from both Red Team (Attackers’) and Blue Team (Defenders’) perspectives, culminating in an attack tree built using Deciduous, an open-source security decision tree tool. Since then, we gave a talk entitled “Broken Links : Behind the scenes of Supply Chain breaches” at several conferences, including BSides NYC and NorthSec.
A noteworthy observation from our research was the importance of implementing GitHub’s “Tag Protection Rules”. This little-known feature, added in March 2022, can prevent contributors with mere write access to a repository from pushing remote tags that match specific sensitive patterns, such as a release’s semantic version scheme. Without a protection rule in place, a threat actor could maliciously manipulate release tags to point to Git commits containing vulnerable code.
Our attack trees also highlighted another vulnerability, this time much more well-known, so-called “pwn requests”. This situation arises when a GitHub Action workflow triggering on the pull_request_target event is abused. Under a very specific set of conditions, it is possible for malicious PR authors to gain repository write permissions or steal repository secrets. GitHub has a series of articles on this topic.
Moving forward to the beginning of 2023, our team embarked on a new quest — detecting malicious packages in package manager registries. While the likes of NPM, RubyGems, and PyPi have been heavily scrutinized by many, we noticed that Hashicorp’s Terraform Registry hadn’t received the same level of attention. Considering Terraform’s critical role in automating Infrastructure As Code (IAC) deployments for cloud infrastructure, we felt it warranted a closer look.
Terraform configurations may refer to two kinds of external dependencies — Providers and Modules. Providers are plugins that extend Terraform with support for interacting with various external systems. Modules, on the other hand, allow splitting out groups of Terraform configuration constructs into reusable abstractions. Each dependency type can be published and updated independently of Terraform itself and the configurations that depend on them.
At the time of our investigation, the public registry hosted over 3000 providers as well as over 13000 modules, some of which have been downloaded millions of times, often within highly sensitive Continuous Deployment environments.
Our analysis revealed a crucial weakness. Unlike providers, modules are not subject to the same protection granted by the Dependency Lock File. When specifying the version of the provider, terraform will fetch the effective binary artifact and pin its exact SHA256 hash in a lock file (.terraform.lock). Modules are simply fetched using a version constraint requirement, which provides no cryptographic guarantee in the face of changes in the underlying artifact. This means that even if one were to review the terraform plan output in a Pull Request, there is no guarantee that a plan generated seconds later as part of terraform apply, which would generate a new plan using the same manifests, would behave the same way. A seemingly benign module used to configure VPC firewall rules could potentially add a backdoor to the firewall configuration.
Also, it’s easy to imagine that a malicious module could use local-exec to execute arbitrary commands in the context of the CI environment where it runs. Another option would be to use the generic http provider to exfiltrate sensitive values from your state.
Given this weakness of modules in mind, the fact the Tag Protection Rules is very rarely used, and that a growing amount of terraform modules published on GitHub use GitHub Actions, we set out to unearth vulnerable modules to exploit. To uncover these vulnerabilities at scale, we downloaded all providers and modules and ran various static code analysis tools, such as Semgrep, looking for GitHub Actions workflows vulnerable to “pwn request”. Our efforts led to the discovery of several hundreds of vulnerable modules, and in many cases, we found that successful exploitation could enable a threat actor to hijack already published, trusted terraform module versions.
We responsibly disclosed these vulnerabilities to the maintainers of the affected modules, leading to collaborative efforts to rectify the issues. This cooperation not only led to the resolution of the identified vulnerabilities but also prompted a comprehensive review of their workflows, resulting in significant security improvements.
One particularly intriguing case involved a vulnerable workflow that automated the formatting of a module’s markdown README file using a template engine called gomplate. By default, gomplate looks for a file named .gomplate.yaml in the current working directory, which can include a postExec configuration for executing arbitrary commands. A subtle modification of the README could trigger re-formatting, and by planting a .gomplate.yaml file containing a postExec command, an attacker could exfiltrate the GITHUB_TOKEN and gain the ability to push Git tags. This discovery underscores the need for diligent security practices, even in seemingly mundane tasks such as formatting documentation.
Through our exploration into the Terraform Registry, we’ve highlighted a critical vulnerability in the security of its modules. Unlike providers, modules do not benefit from the cryptographic guarantee provided by the Dependency Lock File, resulting in potential security threats. This disparity raises significant concerns, especially considering the sensitive environments in which these modules often operate.
In light of these findings, our immediate advice to users is to either:
While both alternatives maintain the integrity of the code post-review, they unfortunately forfeit the benefits offered by the registry reference syntax. Nonetheless, these are merely stopgap solutions to a deeper, systemic problem. The real solution lies in enhancing security measures for Terraform modules to ensure they match the level of protection that is currently extended to providers.
This research underscores the ongoing need for vigilance in software supply chain security, especially in widely used tools like Terraform. We urge both Hashicorp and the broader community to take these findings into consideration, reinforcing the security measures around Terraform modules. To be fair, Hashicorp highlights that malicious modules and providers can exist and should be in your threat model, but our research highlights that even the most well-known, well trusted and used modules can be trivially hijacked. It’s only through continuous investigation, reporting, and rectification of these issues that we can enhance the security of our software supply chains.