What is the Software Supply Chain — and how do I secure it?

In a previous blog article, we described the modern software supply chain development process. This encompasses everything from the developer’s tools, source code management systems, continuous integration machinery, all third-party open source software used, and the generated deployable artifacts (e.g., Docker images). It extends all the way to the deployment systems that orchestrate these images as they are moved into the production environment.

The software supply chain stack is highly complex, similar to your code. It includes various layers like frontend and backend code, infrastructure-as-code, container images, APIs, build and test systems, code management systems, and third-party dependencies. All these components play a vital role in software supply chain development and security.

Each of these layers has its own nuanced security body of knowledge, potential for security weaknesses, and resultant set of security controls to check for and address said weaknesses. For example:

1. Application Code: Concerns around secure coding practices (e.g. OWASP Top 10), Architectural concerns (e.g. design of authentication flow)
2. Infrastructure (as Code): Security misconfigurations such as open/public access, weak/no encryption, etc…
3. Containers: Known vulnerable components
4. Source Management Systems: misconfigured repositories (e.g. missing or incomplete branch protection settings), Missing 2FA for developer accounts on company repos, users with excessive privileges
5. 3rd party dependencies: vulnerable libraries, malicious libraries, Vulnerability to dependency confusion
6. CI systems: vulnerable CI jobs, misconfigured CI jobs

Note: The list above is not exhaustive, but meant to illustrate the breadth of security concerns across a technology stack and build pipelines.

In addition, modern regulatory requirements (e.g. SOC II, the Executive order) require the production of SBOMs at a minimum. In many cases, they require the presence (and subsequent evidence collection) of specific security controls.

Now, in order to provide the coverage across the width of the entire software supply chain, and across the depth of the technology stack — one will need to leverage several specialized security tools. Some of our customers have over a dozen different security tools to get the coverage they need across all technologies used in their organization. For example, using open source examples:

1. Infrastructure as Code: such as checkov, terrascan, or KICS
2. Container scanner: such as Trivy
3. Static Analysis: brakeman, bandit, semgrep, CodeQL to name a few
4. Secrets scanner: gitleaks, trufflehog
5. Dynamic Analysis: OWASP ZAP, BurpSuite, Acunetix
6. Github Security Settings: Reposaur
7. 3rd party libraries: bundler audit, npm audit, dependabot, snyk, etc..
8. Mobile apps: MobSF

Great. So now we know that our software is produced in a wide and deep software factory, and that we will likely need to bring in various security tools at the right location to secure it.

What’s involved? At a high level:

1. Get visibility across your organization — where do you have software and how is it built?
2. Provision the necessary security tooling — get the right technology in the right space across your software development systems.
3. Configure the machine — harden your systems, tune your tools.
4. Engage the developers — to ensure they are acting on high quality findings, at the right time, and that you are indeed building a security culture that is scalable.
5. Generate Reports — to track progress, identify gaps, and determine priorities.

Visibility

Before you set out to secure the software supply chain — you need to understand where your supply chain is. This means you need visibility into all the systems where your source code is stored, how the code is built, the various stages of the pipeline, the security settings spread across the stages, and so on…

Provisioning

This is usually a multi-step process — once you know where your code lives, and how it is built and released — you need to instrument the build environment with the necessary security tools to cover the technology stack and build pipelines. This is where you check for hardcoded secrets, run SAST tools, inspect container images, ensure IaC is secure, and so on.

Configuration

First you need to start with hardening your systems. Do we have the right security settings across our source code management systems? How about our build systems?

Next — you need to find a way to provision (potentially) a dozen scanners of various types across (potentially) hundreds of repos, and then you will have a secure supply chain?

Wrong.

You see, getting the tooling and instrumentation in place to begin with is only the starting point.

To properly secure your software, and the software supply chain, you need to be securing everything as code is being written. Developers need to address security concerns as they are writing code, before that code goes to production.

If you fail to get the developers in your organization on board — your security efforts will be always be dwarfed by the amount of new code being developed on a daily basis.

How do we accomplish that? Well, this means that those tools that we provisioned have to provide near real-time feedback to developers on the code they are writing, as they are writing it.

Great. So we’ll get those tools we just provisioned to emit their output straight to the developer in the form of Pull Request comments, or perhaps in the IDE itself.

Are we done?

This too will not yield the results you are expecting. You see, these tools are built for security experts, not the non-security-expert developer. They have potentially hundreds of rules each, not all of which are important. Not all their output is equally accurate either. Their output was meant to be used by human experts to triage and filter the noise, to find the few gems in the dirt.

Developer Engagement

This is one of the hardest steps in any DevSecOps effort. Many efforts fail at this step exactly. Too many alerts to too many developers, and they start ignoring all the tooling. Rightfully so. If they don’t understand what is asked of them — similar outcome. Lots of false positives? Same.

In order for the security team to get traction with developers, they need to ensure that the security alerts meet certain criteria:

• Fast: You need to provide your developers with feedback shortly after they push their commit. Waiting until the time to merge the PR is too late. Waiting until code has shipped and is in main is definitely too late. Sprint ends tomorrow. Code must ship. Cannot go back and address security issues raised at the last minute.
• Actionable: It should be clear what the action to take is. If a developer was trusting user input, then how do they sanitize the input? If the developer was using a dangerous function, then what is the alternative function they should be calling?
• Accurate: Especially at the early stages of DevSecOps maturity, the reported findings must be accurate. Traditionally, AppSec tools had notoriously high false positive rates because they were trying to find the needle in the haystack. While that may be fine for an AppSec expert doing a thorough investigation of an important code base for the first time — when trying to win developer trust it is important to recognize that perfect may be the enemy of the good. You need developers to trust 90% of the reports, than ignore 100% of them because they are mostly false positives.
• Relevant: Am I seeing a finding about a line of code that I wrote? Is it part of what I am working on now? If not, I am focused on hitting my sprint deliverables, and someone else needs to deal with this — but this issue is not relevant to me, not right now at least…

In order to go from just all tools turned on, to fast-actionale-accurate-relevant results — that is, findings with a high probability of developer timely action — each of these tools needs to be configured/tuned to accomplish that goal. Furthermore, this configuration changes: not every codebase or pipeline is created equal.

• Some repos are critical; because they are internet facing, touch sensitive data, and potentially have financial or reputational damage in case of a breach
• Some repos are less critical than others; perhaps because they are internal apps for ad-hoc reporting, never touching sensitive data. A breach has no reputational or financial concern.

Similarly, you may want to treat public repos different than private repos, etc…

Treating all pipelines and codebases equally is akin to saying all my repos, without exception, must have 95% test coverage. This does not make sense most of the time.

Finally, this goal post changes over time. What may be unimportant today, may become very important next month. Issues and vulnerabilities that you (or the industry as a whole) did not know existed, become critical in the future.

Reporting

Ok. So now we’ve mapped out our Software Supply Chain, Provisioned the necessary scanners, configured all of them appropriately, surely, now we’re done with the tooling part of DevSecOps.

Nope again. You see, you need a reporting layer. You need to be able to easily ask, and answer questions such as:

• Do we know where we have our most pressing and critical issues?
• Are we actually addressing security issues at the same speed of development?
• Are we getting developer engagement? If not, where? Why?
• Which vulnerability classes should we be targeting next month?
• What is the biggest gap in our DevSecOps tooling coverage today?

What’s next?

Ah, finally — we’ve arrived.

Sorry to be the one to tell you — but there’s still more to go. We need to worry about compliance, the particular challenges around 3rd party code, and how tampering of your pipelines can occur, and what you can do to protect against all of that. But we’ll leave that for another day…

Frequently Ask Question 

What is software supply chain development and why is it important? Software supply chain development refers to the entire process of creating and managing software, from the initial development phase to the deployment in production environments. It involves multiple stages, including coding, testing, integration, and managing third-party dependencies. Securing each stage of this process is crucial to ensure that the software remains robust, free from vulnerabilities, and compliant with regulations.

How does software supply chain development impact security?  Software supply chain development significantly impacts security because it involves multiple layers such as application code, containers, and third-party dependencies. Weaknesses or vulnerabilities in any of these areas can compromise the security of the entire system. Implementing security measures at each stage helps protect sensitive data, prevent breaches, and comply with industry regulations.

What is software supply chain and why should I care about securing it? The software supply chain includes all the systems, tools, and processes used to develop, manage, and deploy software. Securing it is essential to prevent cyber-attacks, data breaches, and compliance issues. By safeguarding the software supply chain, you can protect your organization’s digital infrastructure, ensure data integrity, and minimize risks associated with third-party software components.

What are the key stages of software supply chain development? The key stages of software supply chain development include the initial code development, integration with third-party libraries, containerization, deployment, and continuous monitoring. Each of these stages requires specialized security tools and best practices to ensure that the entire software development process remains secure and compliant with industry standards.

How can I secure my software supply chain in development? To secure your software supply chain development, you should implement robust security practices at every stage. This includes using secure coding techniques, scanning for vulnerabilities in third-party libraries, configuring security settings in source code management systems, and utilizing automated tools for continuous monitoring and testing throughout the development process.