What is the Software Supply Chain — and how do I secure it?

In a previous blog article, we described the modern software supply chain. In a nutshell, this is everything from the developer, their laptop, your source code management system, the continuous integration machinery, all the 3rd party open source software used, the generated deployable artifacts (docker images for example), the repository where those images are stored, all the way through the continuous deployment system (if present) that orchestrates those images as they are deployed to the production environment.

This software supply stack is complex. Your code is complex as well. There is frontend code, backend code, infrastructure (as) code, container images, API’s, build systems, test systems, code management systems, 3rd party dependencies.

Each of these layers has its own nuanced security body of knowledge, potential for security weaknesses, and resultant set of security controls to check for and address said weaknesses. For example:

1. Application Code: Concerns around secure coding practices (e.g. OWASP Top 10), Architectural concerns (e.g. design of authentication flow)
2. Infrastructure (as Code): Security misconfigurations such as open/public access, weak/no encryption, etc…
3. Containers: Known vulnerable components
4. Source Management Systems: misconfigured repositories (e.g. missing or incomplete branch protection settings), Missing 2FA for developer accounts on company repos, users with excessive privileges
5. 3rd party dependencies: vulnerable libraries, malicious libraries, Vulnerability to dependency confusion
6. CI systems: vulnerable CI jobs, misconfigured CI jobs

Note: The list above is not exhaustive, but meant to illustrate the breadth of security concerns across a technology stack and build pipelines.

In addition, modern regulatory requirements (e.g. SOC II, the Executive order) require the production of SBOMs at a minimum. In many cases, they require the presence (and subsequent evidence collection) of specific security controls.

Now, in order to provide the coverage across the width of the entire software supply chain, and across the depth of the technology stack — one will need to leverage several specialized security tools. Some of our customers have over a dozen different security tools to get the coverage they need across all technologies used in their organization. For example, using open source examples:

1. Infrastructure as Code: such as checkov, terrascan, or KICS
2. Container scanner: such as Trivy
3. Static Analysis: brakeman, bandit, semgrep, CodeQL to name a few
4. Secrets scanner: gitleaks, trufflehog
5. Dynamic Analysis: OWASP ZAP, BurpSuite, Acunetix
6. Github Security Settings: Reposaur
7. 3rd party libraries: bundler audit, npm audit, dependabot, snyk, etc..
8. Mobile apps: MobSF

Great. So now we know that our software is produced in a wide and deep software factory, and that we will likely need to bring in various security tools at the right location to secure it.

What’s involved? At a high level:

1. Get visibility across your organization — where do you have software and how is it built?
2. Provision the necessary security tooling — get the right technology in the right space across your software development systems.
3. Configure the machine — harden your systems, tune your tools.
4. Engage the developers — to ensure they are acting on high quality findings, at the right time, and that you are indeed building a security culture that is scalable.
5. Generate Reports — to track progress, identify gaps, and determine priorities.

Visibility

Before you set out to secure the software supply chain — you need to understand where your supply chain is. This means you need visibility into all the systems where your source code is stored, how the code is built, the various stages of the pipeline, the security settings spread across the stages, and so on…

Provisioning

This is usually a multi-step process — once you know where your code lives, and how it is built and released — you need to instrument the build environment with the necessary security tools to cover the technology stack and build pipelines. This is where you check for hardcoded secrets, run SAST tools, inspect container images, ensure IaC is secure, and so on.

Configuration

First you need to start with hardening your systems. Do we have the right security settings across our source code management systems? How about our build systems?

Next — you need to find a way to provision (potentially) a dozen scanners of various types across (potentially) hundreds of repos, and then you will have a secure supply chain?

Wrong.

You see, getting the tooling and instrumentation in place to begin with is only the starting point.

To properly secure your software, and the software supply chain, you need to be securing everything as code is being written. Developers need to address security concerns as they are writing code, before that code goes to production.

If you fail to get the developers in your organization on board — your security efforts will be always be dwarfed by the amount of new code being developed on a daily basis.

How do we accomplish that? Well, this means that those tools that we provisioned have to provide near real-time feedback to developers on the code they are writing, as they are writing it.

Great. So we’ll get those tools we just provisioned to emit their output straight to the developer in the form of Pull Request comments, or perhaps in the IDE itself.

Are we done?

This too will not yield the results you are expecting. You see, these tools are built for security experts, not the non-security-expert developer. They have potentially hundreds of rules each, not all of which are important. Not all their output is equally accurate either. Their output was meant to be used by human experts to triage and filter the noise, to find the few gems in the dirt.

Developer Engagement

This is one of the hardest steps in any DevSecOps effort. Many efforts fail at this step exactly. Too many alerts to too many developers, and they start ignoring all the tooling. Rightfully so. If they don’t understand what is asked of them — similar outcome. Lots of false positives? Same.

In order for the security team to get traction with developers, they need to ensure that the security alerts meet certain criteria:

• Fast: You need to provide your developers with feedback shortly after they push their commit. Waiting until the time to merge the PR is too late. Waiting until code has shipped and is in main is definitely too late. Sprint ends tomorrow. Code must ship. Cannot go back and address security issues raised at the last minute.
• Actionable: It should be clear what the action to take is. If a developer was trusting user input, then how do they sanitize the input? If the developer was using a dangerous function, then what is the alternative function they should be calling?
• Accurate: Especially at the early stages of DevSecOps maturity, the reported findings must be accurate. Traditionally, AppSec tools had notoriously high false positive rates because they were trying to find the needle in the haystack. While that may be fine for an AppSec expert doing a thorough investigation of an important code base for the first time — when trying to win developer trust it is important to recognize that perfect may be the enemy of the good. You need developers to trust 90% of the reports, than ignore 100% of them because they are mostly false positives.
• Relevant: Am I seeing a finding about a line of code that I wrote? Is it part of what I am working on now? If not, I am focused on hitting my sprint deliverables, and someone else needs to deal with this — but this issue is not relevant to me, not right now at least…

In order to go from just all tools turned on, to fast-actionale-accurate-relevant results — that is, findings with a high probability of developer timely action — each of these tools needs to be configured/tuned to accomplish that goal. Furthermore, this configuration changes: not every codebase or pipeline is created equal.

• Some repos are critical; because they are internet facing, touch sensitive data, and potentially have financial or reputational damage in case of a breach
• Some repos are less critical than others; perhaps because they are internal apps for ad-hoc reporting, never touching sensitive data. A breach has no reputational or financial concern.

Similarly, you may want to treat public repos different than private repos, etc…

Treating all pipelines and codebases equally is akin to saying all my repos, without exception, must have 95% test coverage. This does not make sense most of the time.

Finally, this goal post changes over time. What may be unimportant today, may become very important next month. Issues and vulnerabilities that you (or the industry as a whole) did not know existed, become critical in the future.

Reporting

Ok. So now we’ve mapped out our Software Supply Chain, Provisioned the necessary scanners, configured all of them appropriately, surely, now we’re done with the tooling part of DevSecOps.

Nope again. You see, you need a reporting layer. You need to be able to easily ask, and answer questions such as:

• Do we know where we have our most pressing and critical issues?
• Are we actually addressing security issues at the same speed of development?
• Are we getting developer engagement? If not, where? Why?
• Which vulnerability classes should we be targeting next month?
• What is the biggest gap in our DevSecOps tooling coverage today?

What’s next?

Ah, finally — we’ve arrived.

Sorry to be the one to tell you — but there’s still more to go. We need to worry about compliance, the particular challenges around 3rd party code, and how tampering of your pipelines can occur, and what you can do to protect against all of that. But we’ll leave that for another day…