# Roles and Responsibilities of Signing, SBoMs and Security Scanners

As cloud native development continues to automate the consumption of upstream content providers, the ability for automation to make realtime, informed decisions becomes critical to keep the automation wheels spinning. The speed of development and consumption has reached the point where cracks in the system are always present, even if not immediately seen. The ability to leverage new information against facts on artifacts already in our environments are key aspects of analysis. The time shifted analysis continually improves the hygiene of the artifacts we consume, build and distribute.

Signing, Systems Bill of Materials (SBoM), Security Scanning and Registries play major roles in how you build, secure, distribute and consume artifacts. But, what are the roles and responsibilities of each? What should be included in an SBoM, and what are the expectations of signing? If you have a signed SBoM, do you even need to run security scanning software? In this article, I'll cover an opinion for the roles and responsibilities of each as security is an always evolving standard, with multiple lines of defense.

> Side note: the [SPDX](https://spdx.dev/) and [MITRE](https://mitre.org/) groups use Systems Bill of Materials, as opposed to Software Bill of Materials to reflect IoT and other hardware based device scenarios.

## Systems Bill of Materials (SBoM) & Tacos

You're planning a party for the end of COVID-19. Some of your friends are gluten-free and some are vegetarians. The menu is Tacos, 'cause who doesn't love tacos?

You order gluten-free corn and flour tortillas, a variety of vegetables, and of course, avocados to make fresh guacamole. You even plan separate serving areas and utensils to not cross contaminate.

<img src="https://www.missionfoods.com/uploads/Mission-Super-Size-White-Corn-Tortillas-Back_kRJTToV.png" alt="Food ingredients" width="300" align="right"/>

How do you know what products are gluten-free, and which are made from non-vegetarian ingredients or packaging? Food products have a list of ingredients. The list doesn't elaborate the full recipe, nor state an opinion of the quality of those ingredients. It factually states the ingredients for consumers to make educated decisions.

In addition to the factual perspective, where is the list of ingredients generated? The ingredients are a factual list of what went _into_ "assembling" the product.

Can you analyze a completed food product and know what went into it? You might perform a DNA analysis to understand amylase was in the final product. But was amylase the ingredient, or the result of mixing other ingredients and its what formed when the product was complete? These are important distinctions, and both are valuable.

The list of ingredients is the equivalent of an SBoM. It states the factual aspects consumers need to know.

## Vulnerabilities

[As of late December 2021, 13 people across six states were diagnosed with the outbreak strain of E. coli O157:H7](https://www.foodsafetynews.com/2021/12/publishers-platform-53-sick-30-hospitalized-4-with-kidney-failure-and-3-dead/). It seems the concern over corn and flour tortillas were less of an issue, compared to the bag of lettuce that was used.

The **Simple Truth Organic Power Greens** ingredients (SBoM) lists:

> _Organic Spinach, Organic Mizuna, Organic Chard, Organic Baby Kale. Mix May Vary by Season._

Not surprisingly, there's no mention of E. coli O157:H7, as there was no intent to include E. coli, nor was it known at the time that some of the greens were contaminated.

At the time of packaging, and the time of purchase, everything was fine, or rather it was believed to be fine. Only later, when consumers were diagnosed with E. Coli was an analysis completed. The affected people, across 6 states, all consumed the same product, labeled _"Best if used between Nov. 30, 2021, and Jan. 8, 2022"_. The packaging wasn't damaged, it remained sealed, nor was it contaminated along the distribution supply chain. It was a failure in the preparation of the product (the build system).

### In the Future, We Learn About Things of the Past

With the knowledge around E. coli contaminations, we can block consumption of the specific product: _Simple Truth Organic Power Greens with a Best if used date between Nov. 30, 2021, and Jan. 8, 2022_ label. Is that enough? What can we learn from this event and other outbreaks that help us prevent this _vulnerability_ from making its way into future supply chains? If the risk and recurrence is high enough, manufacturers might choose to proactively test for E. coli before their food product is shipped. The industry might even require testing (scanning) for known vulnerabilities prior to distribution. A scanner may identify known ingredients that are _subject to_ a type of vulnerability and perform further testing. If specific types of lettuce, from specific regions of the world have proven to be more prone to E. coli contaminations, it might be worth more expensive testing. In the E. coli case, the food processor would need to [test their greens as part of the assembly line](https://www.foodnavigator-usa.com/Article/2012/08/08/Firm-secures-US-certification-for-E.coli-kit). Manufacturers could attest they follow the certified process, just as manufactures endorse products as [gluten-free](https://gfco.org/certification/), [non GMO](https://www.nongmoproject.org/), [Kosher](https://www.ok.org/), or [USDA Organic](https://www.usda.gov/topics/organic).

Shifting the focus from contamination response, the FDA recently announced the [Food Safety Modernization Act (FSMA)](https://www.fda.gov/food/guidance-regulation-food-and-dietary-supplements/food-safety-modernization-act-fsma), which shifts to proactive testing measures. The FDA is taking the historic knowledge to apply to future potential situations, as the risk/reward is justified.

## Scanning for Best Practices

In the above case, we explored the lack of information in an SBoM, and the need for scanning to test for potential risk. The SBoMs didn't include E. coli, metal fragments or other vulnerabilities. Products that are known to contain these vulnerabilities would be recalled, and removed from the supply chain. I'll explore revocations a bit further down.

SBoMs can be used to assure best practices are applied. The inclusion of an item in an SBoM is factual information that enables a scanning policy to make an informed decision, based on what's known at a point in time. Scanners may output endorsements (badges), when specific tests were validated as they implemented a best practice. However, a scan of factual information may yield a different decision as new information is applied to previous facts.

### Asbestos Fibers for Fire Suppression and Material Bonding

<img src="https://www.asbestos123.com/image.php?&image=/uploads/large-1.jpg" alt="Asbestos curtain" width="400" align="right"/>

Between the 1950s and 1980s, Asbestos curtains were the standard for theaters. The curtains _boldly advertised_ they were made with Asbestos, to give comfort to attendees in the case of an on-stage fire.

In the 1970s, Asbestos was preferred in some industries as a risk mitigation. Contracts _required_ Asbestos in the bill of materials. Flash forward to the 1980s. The same SBoM, for the exact same product, is now considered a high risk that _requires_ abatement for the product that was previously considered a safety requirement.

Did the product change? It was our knowledge about the product that changed. Future information was applied to past factual data. If the SBoM of a theater stated an opinion the theater was safe from stage fires, without stating the facts as to how that requirement was met, the SBoM wouldn't have the same long term value. It was the static facts contained in the SBoM, at the time of creation, that enabled future opinions on the factual information.

### Risk and Remediation

<img src="https://www.asbestos.com/wp-content/uploads/asbestos-tiles-336x0-c-default.jpg" alt="Asbestos floor tiles" width="200" align="right"/>

Asbestos is an example of a known risk, that is avoided in future products, but doesn't require immediate abatement for existing products. Existing products that contain Asbestos are only a risk under certain conditions.

Asbestos was so prevalent, most residential and commercial buildings built between 1950-1980 likely contain [Asbestos floor tiles](https://www.asbestos.com/blog/2018/07/13/asbestos-floor-tile-diy-removal/) or [Asbestos ceiling tiles](https://www.mesothelioma.com/asbestos-exposure/products/asbestos-tiles/). Are these buildings unsafe? Should all buildings that contain Asbestos be immediately shut down until the Asbestos has been removed, and replaced with whats _currently believed to be_ safe products? What is the risk (cost) and reward for such an undertaking?

### When to Remediate

Identifying a risk doesn't necessarily require immediate abatement and remediation. It's a reference point to decide if the risk applies to the usage. As long as the Asbestos floor and ceiling tiles are left in-tact, the risk is managed. The costs of removing the tiles would entail shutting down the building, relocating the occupants, with an impact to the business. The tiles require special handling to remove, increasing the costs, and then the reconstruction must begin. When new construction (deployments) are begun, the new products are validated against known vulnerabilities. Construction of a theater in 1970 would require Asbestos curtains, or Asbestos tiles as they were cheap and sturdy. However new construction (deployments) assure Asbestos is not in the SBoM.

### Measured Responses

In the software supply chain, we might identify a known vulnerability, which exists in the registry and is currently deployed, but has no exploit vector. For example, Asbestos is not a risk, until it's disturbed.

The fictitious Wabbit Networks software company produces network monitoring software, which is consumed by Acme Rockets. When the Wabbit Networks `net-monitor` product was initially built, a throttling profile package was used to provide usage insights, while also preventing denial of service attacks. Wabbit Networks advertised their software integrated with api management systems, because it included the throttling profile package. Scanning software at Acme Rockets required the package as a best practice, integrating with their API management software.

Several months later, the profiler package was found to send PII data to the [Vulgarian government](https://en.wikipedia.org/wiki/Chitty_Chitty_Bang_Bang). The security team at Acme Rockets had to evaluate whether they needed to shut down their operations, delaying the launch of their road runner rocket, or proceed with operations and remediate the exploit at a later date.

While the exploit is considered a [critical level risk](https://nvd.nist.gov/vuln-metrics/cvss), the usage within Acme Rockets is mitigated as production environments are locked down, and all nodes, with the vulnerable package, have no egress to send the PII data. As the launch commenced, their kubernetes environment needed to scale up more instances to satisfy all the telemetry processing.

The security team evaluated the risk and determined they could proceed. They flagged the vulnerability in production deployed nodes requiring mitigation within 5 days. The scanners were configured to block _new_ deployments that contained the throttling package, but existing deployments were allowed to scale. To mitigate the risk of scaling to newly added nodes, a security rule was added requiring nodes must be in a private network with no egress. The data exfiltration exploit prompted a proactive best practice, locking down nodes by default.

The development team was given time to replace the package within 5 days, enabling the team to watch the afternoons launch. Of course, the road runner dodged the rocket, and Acme Rockets was back to the drawing board.

## Signing, Identity, Quality and Revocation

What does a signature imply? When an artifact is consumed, do you know who created it?
The location by which you acquired the artifact should not reflect the sole source of trust as the [Best Practice for Consuming Public Content](https://opencontainers.org/posts/blog/2020-10-30-consuming-public-content/) is to take possession of the artifacts you depend upon.
In addition to the promotion of public content to private registries, the container community recognizes the challenges with consuming public ([see Docker Hub Rate Limiting](https://www.docker.com/increase-rate-limits)).
Alternate sources include the [Amazon ECR Public Gallery](https://gallery.ecr.aws/), [GitHub Package Registry](https://github.com/search?package_type=container&q=package+registry&type=RegistryPackages) and the [Microsoft Container Registry](https://aka.ms/mcr/).
Just as you can find the Mission gluten-free tortillas from several grocery stores, the same debian image may be available from multiple registries. The location you pull the artifact from, should not reflect the identity. See [Separating Identity from Location](https://stevelasker.blog/2021/09/24/separating-identity-from-location/) for more info.

Signatures represent identity, but how is the signature implemented? As a best practice, detached signatures are preferred as they enable validation of the signature, prior to pulling the content of the artifact. Signing solutions, such as [Notary v2](http://notaryproject.dev/) with [ORAS Artifacts](https://github.com/oras-project/artifacts-spec/), enable multiple signatures to be independently associated with an artifact. As the artifact is promoted within and across registries, users can filter which signatures are promoted.

The detached signature contains the signature and a hashed reference of the signed artifact. The hash assures the signature is associated with a specific version. By associating the signature of the entity with the hash of the artifact, you no longer need to track the independent hashes of each artifact. You choose to trust the artifact from a signing entity, until you don't.

The signed artifact could be the `net-monitor` container image, the SBoM of the image, or even a persisted scan result of the image. If we're going to evaluate SBoMs as a source of our security decisions, don't we need to know who created them? And, based on who created them, do we trust them?

### Does a Signature Imply Quality

A signature, by itself, is not intended to guarantee quality.
There may be hundreds of releases of the `net-monitor` image that contain various levels of quality. Some builds may even have known or unknown exploits.
Being able to identify the author is a piece to the secure supply chain puzzle.

Using the favorite [Solar Winds Exploit](https://www.solarwinds.com/sa-overview), it was the signature that provided insight the exploit wasn't a distribution hack. The signing key wasn't stolen. The artifacts were signed by the SolarWinds build system, enabling auditors to focus on how the artifacts were built.

The next question is how are consumers notified of the exploit? Should the signing key be revoked, causing the signature validation to fail, because a particular build was found to be a lower quality, or contained a vulnerability?
If we're going to revoke signing keys because of vulnerabilities, what level of a vulnerability should constitute revocation? Is the revocation more damaging? Should the scaling of the Acme Rockets telemetry service fail, just as the launch commenced? Is revocation the right tool for the job, or a blunt hammer that can cause more damage and confusion than the initial exploit?

Key revocation is a debated and controversial topic. SolarWinds did [revoke their signing key](https://www.solarwinds.com/sa-overview/new-digital-certificate), which caused a wider range of problems.
> ...the code-signing certificate used by SolarWinds to sign the affected software versions was revoked March 8, 2021. **This is industry-standard best practice for software that has been compromised.**

Is it the "best practice". Should it be the best practice?

The article goes on to note:

> Regretfully, the same digital code-signing certificate used to sign our Orion Platform software affected by the SUNBURST vulnerability was also used to sign additional SolarWinds products not known to be affected by SUNBURST. While this **does not mean all products are compromised**, it does mean the day-to-day operation of any software signed by the compromised digital code-signing certificate may be impacted

Which prompted questions from users:

> [Can I just replace the revoked digital code-signing certificate with the new one and keep my software running?](https://www.solarwinds.com/sa-overview/new-digital-certificate#question8)

Of course, replacing the code-signing certificate doesn't mitigate the problem. It doesn't automatically download and upgrade to remediated versions. Users must install a new version of the software, _and_, they must enable new versions of the signing certificates.

So, what is the best practice? What are the roles and responsibilities of signatures, SBoMs and security scanners? When should you revoke a signing key. Is it the job of signing keys to enforce vulnerabilities or quality, when the vulnerability was distributed by the owner of the signing key?

### Identity Theft = Revocation

Lets compare to the identify of a person, Morgan. If Morgan does a bad thing, as a mistake or an intentional act, does that mean Morgan is no longer Morgan? Should Morgan get a new identity each time they make a mistake? In fact, if Morgan does get a new identity each time, and continues to intentionally do bad things, how do you know if Morgan is continuing to do bad things? Or, do you have a list of bad actors, that happen to all be traced back to Morgan?

In comparison, if Morgans identity is stolen, then we need to know Morgan is still Morgan, but the information used to identify Morgan is no longer valid.

A signature is representation of an identity. As long as the identity is valid, the identity should be maintained. When the identity is stolen, and used to identify bad actors, the identity of the signature should be revoked as it no longer accurately identifies Morgan. If your drivers license, passport, or credit cards are stolen, you contact those authorities to have them canceled, and new documents and/or credit cards are issued. Anyone attempting to use the stolen identities are blocked, as each of these systems have their validation schemes. However, there's no central list to track all drivers license, passport or credit cards. And, if you purchased a bad product, or cancelled a subscription, you don't cancel the credit card, you file for a credit.

### Security Scanning Identifies Vulnerabilities

Using the SolarWinds exploit, I'd suggest revoking the signature wasn't the appropriate approach. It created a wider blast radius, causing [collateral damage](https://www.merriam-webster.com/dictionary/collateral%20damage). And, it didn't actually remediate the problem. It was a blunt hammer.

Security scanners were the appropriate tool for the job. Users didn't know where the SolarWinds software was running. However, companies should deploy security scanning products, which monitor all environments. The security scanners look for known vulnerabilities, enabling users visibility and tools for remediation. As users were notified of the problem, by the security scanner, they were notified to remediate by deploying a new version. The new version would install, as the SolarWinds signing key still accurately reflected SolarWinds built the update, and the security scanning software would approve the update, as it no longer contained the exploit. Unaffected SolarWinds products would continue to operate, as they were never impacted.

## Scanning

> TODO: Based on the great feedback, I'm working on an update to this section to reflect inputs and outputs from a continual stream of upstream updates. The outputs are a combination of factual insights and opinions. The may become SBoMs, while the security scans are continually evolving opinions.

> Security scanning can, and should be done at each stage of the supply chain (build, distribute, consume). Imported artifacts for the build, staging and production environments must be scanned. The source code, and the patterns within the source must be scanned. The output of the build system (the collection of artifacts) must be scanned as the aggregate can surface new vulnerabilities.

> Scanning takes inputs and generates outputs. As the saying goes, the output is only as good as the collection of inputs. Since we're continually chaining the consumption of upstream artifacts, scanners are always at the mercy of what it can consume.
> Scanning provides insight, based on what is known at that point in time, and what has been learned, up to that point in time. It evaluates the information it has and states an opinion, based on the knowledge at that point it time. It doesn't predict the future, nor may it see the cracks just below the surface. Scanners provide a subjective opinion that will change, over time as it learns of new exploits and vulnerabilities. As a result, we continue to improve our knowledge, and often we find out information that changes our perception of what was known prior.

## Security Scanners and SBoMs, PB&J

Security scanners use information to make decisions. These include the list of [Common Vulnerabilities and Exposures (CVE)](https://www.cve.org/) and proprietary information security companies maintain. They actively scan content for what they can see. Using the Tacos example, scanners perform DNA analysis on the completed food product. Wouldn't it be nice to know what went into the food product? If you knew [Red Dye No. 3, which causes Thyroid Cancer](https://nutritionfacts.org/2015/04/30/coloring-to-dye-for-dangers-of-red-no-3/), was used in creation of the food, you would know to avoid it. You would have insight to what went _into the build_, and wouldn't be limited to after the fact, DNA style analysis.

If scanners new a particular package was used to compile a native binary, they could be far more effective and efficient. When a new vulnerability is found in a particular package, or a build configuration, scanners could query the list of SBoMs of software that's actively deployed providing more insight. Scanners wouldn't have to pull terabytes of container images across the network to scan for patterns in the compiled code.

If you have an SBoM for building you're purchasing in 1970, you would require Asbestos curtains. Purchasing the same building in 2022 wouldn't stop the purchase and usage. But, you would use that factual information to negotiate the price of abatement. Without an SBoM, you're faced with lengthy and expensive inspections to check for Asbestos. The inspection may require non-destructive testing, concealing potential risks. The use of an SBoM, from an entity you trust, verified by a signature, makes the entire process efficient. When the signature is from a construction company, that was found to do shady work, the SBoM may be considered invalid, and exploratory testing may be required before purchasing the building. or the identity may represent a company so shady, that you either walk away from the deal (block the deployment), or know the building must be torn down and replaced before you can continue.

## Summing It Up

### SBoMs Should Contain Factual, Non-opinionated Insight That Will Last Over Time

They represent information, frozen at the point they are created. They must provide information that went into the build, not after-the fact, DNA style analysis. They should provide information about the build, the environment, and any other insight that _can be used for future analysis by scanning systems_. SBoMs should be easy to create, if they're created at the point of the build. There's not much logic, as it simply captures factual information. You might argue a particular SBoM may not capture enough information, but the facts for what it captures should not be opinionated.

I'd suggest any analysis done after a build shouldn't be considered an SBoM, rather it's a scan result, attempting to inventory what it may have found in the build, based on what it knows for how to evaluate an artifact. It is not factual information, rather it is an opinion, only as good as the tools are able to reverse engineer the artifact. Just as DNA analysis is far harder than capturing the list of ingredients, SBoMs are simple documents of facts.

### Scanners Are Time Based Opinions

Scanners inform, based on what’s known at a point in time, providing oversight, where time is a factor of evolving information

Scanners can use the factual information for what went into a build to be efficient and deeply informed. You may provide an SBoM and scan at the time of a build, as two separate pieces of information; one fact, one opinion based on what was known at the time.

If the build system knows its building an artifact that contains a known vulnerability, it should list the package in the SBoM, and the opinion in the scan result. The opinion may change in the future, and a new scan can represent the newly informed opinion, based on the static facts of the SBoM. You may likely choose to keep the history of SBoMs, as you may be asked: _"why was `foo:v123abc` deployed on Jan 12, 2021, when we know it contains `vul 456def`"_ To which you can demonstrate on Jan 12, 2021 it had scanned clear of that vulnerability. It wasn't until September 10, 2021 that `vul 456def` was even identified.

### Signatures Are Identities, That Are Maintained as Long as the Identity Is Accurate

Signing certificates may be revoked, but only when the identity has been stolen. Signatures which contain verifiable timestamps for when they were issued, may be invalidated from a point in time. If you believe an identity was stolen on April 15th, you should consider the confidence in the date of theft, increasing the range providing a margin of error based on the confidence, blocking signatures for days or weeks prior. Signatures should not be revoked as a blunt instrument to impose an opinion of quality. I'd suggest it is the overuse, or abuse of key revocation, and the central management of revocation that has led to the concerns of revocation.

### A Suite of Tools, Each With Their Own Purpose

Scanner are the tools for quality and opinions on security, which use identity and SBoM insight to inform that configurable opinion.

SBoMs are an excellent new tool, that will make future software validations more reliable and efficient. The ability to use these new tools are only as good as the quality that goes into them.