In a previous blog article, we described the modern software supply chain. In a nutshell, this is everything from the developer, their laptop, your source code management system, the continuous integration machinery, all the 3rd party open source software used, the generated deployable artifacts (docker images for example), the repository where those images are stored, all the way through the continuous deployment system (if present) that orchestrates those images as they are deployed to the production environment.
This software supply stack is complex. Your code is complex as well. There is frontend code, backend code, infrastructure (as) code, container images, API’s, build systems, test systems, code management systems, 3rd party dependencies.
Each of these layers has its own nuanced security body of knowledge, potential for security weaknesses, and resultant set of security controls to check for and address said weaknesses. For example:
Note: The list above is not exhaustive, but meant to illustrate the breadth of security concerns across a technology stack and build pipelines.
In addition, modern regulatory requirements (e.g. SOC II, the Executive order) require the production of SBOMs at a minimum. In many cases, they require the presence (and subsequent evidence collection) of specific security controls.
Now, in order to provide the coverage across the width of the entire software supply chain, and across the depth of the technology stack — one will need to leverage several specialized security tools. Some of our customers have over a dozen different security tools to get the coverage they need across all technologies used in their organization. For example, using open source examples:
Great. So now we know that our software is produced in a wide and deep software factory, and that we will likely need to bring in various security tools at the right location to secure it.
What’s involved? At a high level:
Before you set out to secure the software supply chain — you need to understand where your supply chain is. This means you need visibility into all the systems where your source code is stored, how the code is built, the various stages of the pipeline, the security settings spread across the stages, and so on…
This is usually a multi-step process — once you know where your code lives, and how it is built and released — you need to instrument the build environment with the necessary security tools to cover the technology stack and build pipelines. This is where you check for hardcoded secrets, run SAST tools, inspect container images, ensure IaC is secure, and so on.
First you need to start with hardening your systems. Do we have the right security settings across our source code management systems? How about our build systems?
Next — you need to find a way to provision (potentially) a dozen scanners of various types across (potentially) hundreds of repos, and then you will have a secure supply chain?
Wrong.
You see, getting the tooling and instrumentation in place to begin with is only the starting point.
To properly secure your software, and the software supply chain, you need to be securing everything as code is being written. Developers need to address security concerns as they are writing code, before that code goes to production.
If you fail to get the developers in your organization on board — your security efforts will be always be dwarfed by the amount of new code being developed on a daily basis.
How do we accomplish that? Well, this means that those tools that we provisioned have to provide near real-time feedback to developers on the code they are writing, as they are writing it.
Great. So we’ll get those tools we just provisioned to emit their output straight to the developer in the form of Pull Request comments, or perhaps in the IDE itself.
Are we done?
This too will not yield the results you are expecting. You see, these tools are built for security experts, not the non-security-expert developer. They have potentially hundreds of rules each, not all of which are important. Not all their output is equally accurate either. Their output was meant to be used by human experts to triage and filter the noise, to find the few gems in the dirt.
This is one of the hardest steps in any DevSecOps effort. Many efforts fail at this step exactly. Too many alerts to too many developers, and they start ignoring all the tooling. Rightfully so. If they don’t understand what is asked of them — similar outcome. Lots of false positives? Same.
In order for the security team to get traction with developers, they need to ensure that the security alerts meet certain criteria:
In order to go from just all tools turned on, to fast-actionale-accurate-relevant results — that is, findings with a high probability of developer timely action — each of these tools needs to be configured/tuned to accomplish that goal. Furthermore, this configuration changes: not every codebase or pipeline is created equal.
Similarly, you may want to treat public repos different than private repos, etc…
Treating all pipelines and codebases equally is akin to saying all my repos, without exception, must have 95% test coverage. This does not make sense most of the time.
Finally, this goal post changes over time. What may be unimportant today, may become very important next month. Issues and vulnerabilities that you (or the industry as a whole) did not know existed, become critical in the future.
Ok. So now we’ve mapped out our Software Supply Chain, Provisioned the necessary scanners, configured all of them appropriately, surely, now we’re done with the tooling part of DevSecOps.
Nope again. You see, you need a reporting layer. You need to be able to easily ask, and answer questions such as:
Ah, finally — we’ve arrived.
Sorry to be the one to tell you — but there’s still more to go. We need to worry about compliance, the particular challenges around 3rd party code, and how tampering of your pipelines can occur, and what you can do to protect against all of that. But we’ll leave that for another day…