Collect and Manage your Binary Metadata using Build-Info

Our modern life depends on software from the most trivial to critical task. How software is built, behaves and what it actually contains are fundamental questions that almost all stakeholders of the Software Development Life Cycle (SDLC) need to know. Being able to effectively manage your binaries (aka software packages, artifacts, containers, images…) provides full control over your binary lifecycle. This includes knowing which binaries were created in each build, what the available release candidates are, which of them have issues and so on.

To achieve even more control over the software development life cycle, it’s not enough to just host binaries in repositories and provide fast access to them, it’s also important to store as much metadata about them as possible. Metadata like their creation and modification date, the user who uploaded the binaries, their checksums and a lot more.

What is binary metadata?

Metadata is a term that in the SDLC suggests going “beyond the actual data”. Any binary file contains actual data that is used by an interpreter or processor to execute, display or take action. Metadata is a set of entries that describes extended data, without the need to actually process it, allowing for said extended data to be quickly searched, filtered, classified or managed according to some criteria that is exposed as a metadata entry.

Why is binary metadata important?

Managing metadata about your binaries allows you to efficiently group your binaries according to any criteria, and then perform operations on them. For example, you can create retention policies for binaries according to their creation or last download time. You can mark binaries with properties, to reflect events that they go through during their lifecycle. Furthermore, record  a set of tests  passed, or if they were found free of security vulnerabilities. You can now probably imagine many more useful scenarios where metadata can really come in handy.

Where can you find and collect binary metadata?

There is a considerable amount of  information about binaries that can be collected during the build phase when they are being  created. We developed an entity in JFrog Artifactory called build-info, JFrog’s SBOM own format, available as an open source tool. Build-info is practically a recording of the build. It is saved in JSON format, and includes most of all the information that can be collected during the build process. For example it includes the build modules, artifacts (the binaries created by the build), the dependencies (the binaries used to build the artifacts), the source code repository which includes the code to build the binaries, the branch, commit hash and message, environment variables, and a several additional data that is extracted during the build process.

How build-info works

The build client records and generates the build-info data during the build process, and then stores it in Artifactory. This is in addition to the actual artifact binaries. Resulting in plenty of information for each binary stored in Artifactory in the form of a build-info. To give a better sense of how the build-info looks, see the following example.

{
  "agent": {},
  "buildAgent": {
    "name": "GENERIC"
  },
  "modules": [
    {
      "type": "go",
      "id": "github.com/jfrog/build-info-go",
      "dependencies": [
        {
          "id": "github.com/stretchr/testify:v1.7.0",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/gofrog:v1.1.1",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "53b5c82ff76628b33b04017e8c81fbc1875f5737",
          "md5": "3cb74476ca750cb267db738a4db2f534",
          "sha256": "5a46ccebeff510df3e2f6d3842ee79d3f68d0e7b1554cd6ee93390d68b6c6b34"
        },
        {
          "id": "gopkg.in/yaml.v3:v3.0.0-20200313102051-9f266ea9e77c",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/stretchr/testify:v1.7.0",
              "github.com/jfrog/gofrog:v1.1.1",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/stretchr/testify:v1.7.0",
              "github.com/jfrog/gofrog:v1.1.1",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/stretchr/testify:v1.7.0",
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "ec896ba2dc97dc3aa33066686b74259520428e00",
          "md5": "b8faa9934f8e54c43766ce7b4b2e0d49",
          "sha256": "acf19ccb4fca983b234a39ef032faf9ab70e759680673bb3dff077e77fee20fe"
        },
        {
          "id": "github.com/kr/pretty:v0.2.1",
          "type": "zip",
          "requestedBy": [
            [
              "gopkg.in/check.v1:v1.0.0-20201130134442-10cb98267c6c",
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "e808602a157cdd88fc8984f27895fffd3d15ce8c",
          "md5": "353d5783d72d7e5b4409747b0be33177",
          "sha256": "80af0452082052d1b3265d7cb8985d464d4be222c27e14658e95632c222761e5"
        },
        {
          "id": "github.com/bradleyjkemp/cupaloy/v2:v2.6.0",
          "type": "zip",
          "sha1": "079e9f3594bab1a396ab9fe2d3fc5f5de1e7282a",
          "md5": "0aba1848e0f4de1bd5dcabd9569bf8f8",
          "sha256": "362b2b0446926332be700b60629d8788f622969d861fbcff7e65ccb97ed07fb3"
        },
        {
          "id": "github.com/urfave/cli/v2:v2.3.0",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "0f882edb17acb1c544f6d53c5afa1d6d2add1308",
          "md5": "81a81c77ec9b2721e0229a66d5f77a83",
          "sha256": "bef25aedf2f3ac498094ec9cd216bca61ddf5f2eb7b1ecd850bbfb6053fe4103"
        },
        {
          "id": "github.com/minio/sha256-simd:v1.0.1-0.20210617151322-99e45fae3395",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "f091f68b7467e6dfb5ce28ae894b295525e59d47",
          "md5": "572ef4681740cfdacbbe601587609622",
          "sha256": "bb36b77f985b4ef963517202dbce3a9c72ffc7b90d70143ab4cd176981aa4c72"
        },
        {
          "id": "github.com/pkg/errors:v0.8.0",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/gofrog:v1.1.1",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "f539bd34de2d4ab21c2865065eebc072c37c1194",
          "md5": "4030db591c8aca36aec6773ca552d95f",
          "sha256": "e4fa69ba057356614edbc1da881a7d3ebb688505be49f65965686bcb859e2fae"
        },
        {
          "id": "github.com/davecgh/go-spew:v1.1.1",
          "type": "zip",
          "sha1": "0f9760bda0c6ccacac5e57f62d0f5ad9c7dab03f",
          "md5": "feef6644bd69286382139b28be3f0b91",
          "sha256": "6b44a843951f371b7010c754ecc3cabefe815d5ced1c5b9409fb2d697e8a890d"
        },
        {
          "id": "github.com/cpuguy83/go-md2man/v2:v2.0.0-20190314233015-f79a8a8ca69d",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/urfave/cli/v2:v2.3.0",
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "5586c962d5149ce9d73190ae61bab99ed56d4c7f",
          "md5": "ca2d6e511be9be839f06e049e710063e",
          "sha256": "38ea243c30ed1729d62ec8df91357ab040ac4967cc42d409b7600e0266f7e23c"
        },
        {
          "id": "github.com/!cyclone!d!x/cyclonedx-go:v0.4.0",
          "type": "zip",
          "sha1": "4fd24140c9d75be7361f204809ac509cfbec7d21",
          "md5": "2ac70bf2397c5bb980ccfd3dac6bd24d",
          "sha256": "329d65e011bde22c18a6210869b5ebe10cd943d53352b14d53d0b442007f279e"
        },
        {
          "id": "github.com/klauspost/cpuid/v2:v2.0.6",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/minio/sha256-simd:v1.0.1-0.20210617151322-99e45fae3395",
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "1ed6884c9ee6ecf98727186591ec597771bd9abe",
          "md5": "e5a4769c581330d21ea90f433cec2ad0",
          "sha256": "514cbd03b0ded074640a9034af2cbc87490167a6d622a8c4bf478e153d8366e2"
        },
        {
          "id": "github.com/kr/text:v0.2.0",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "7d227e9c9516bd2a9617dfec9b150df1cc8d2ef3",
          "md5": "52630c25195715aa3b747ed34c8c1536",
          "sha256": "368eb318f91a5b67be905c47032ab5c31a1d49a97848b1011a0d0a2122b30ba4"
        },
        {
          "id": "gopkg.in/check.v1:v1.0.0-20201130134442-10cb98267c6c",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "19bf400c2215e26dce7b3e966b0035d3c1dbdc87",
          "md5": "dcd82e15e290fa75348922f38492dae7",
          "sha256": "f555684e5c5dacc2850dddb345fef1b8f93f546b72685589789da6d2b062710e"
        },
        {
          "id": "github.com/buger/jsonparser:v1.1.1",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "e0c54d96564262a70bc7ed33fb3ee2b15596f68f",
          "md5": "7ab77d10951f73b96b9c19a6cca51bb1",
          "sha256": "be17ef1b44c22eac645eeac80f0e26cdfc70d77262e631358e00c2aa817eab8c"
        },
        {
          "id": "github.com/pmezard/go-difflib:v1.0.0",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/stretchr/testify:v1.7.0",
              "github.com/jfrog/gofrog:v1.1.1",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/stretchr/testify:v1.7.0",
              "github.com/jfrog/gofrog:v1.1.1",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/stretchr/testify:v1.7.0",
              "github.com/jfrog/build-info-go"
            ],
            [
              "github.com/cpuguy83/go-md2man/v2:v2.0.0-20190314233015-f79a8a8ca69d",
              "github.com/urfave/cli/v2:v2.3.0",
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "f200e2a5211b527ef2d2ff301718ccc4ad5c705b",
          "md5": "fb72df530a7f3fca56ccc192c9f30a58",
          "sha256": "de04cecc1a4b8d53e4357051026794bcbc54f2e6a260cfac508ce69d5d6457a0"
        },
        {
          "id": "github.com/shurcoo!l/sanitized_anchor_name:v1.0.0",
          "type": "zip",
          "sha1": "fd4810a945b887a2e0f0ebb760131e13dca566ae",
          "md5": "90b29aa5c53c3df1b2b80e4d7220b1e3",
          "sha256": "0af034323e0627a9e94367f87aa50ce29e5b165d54c8da2926cbaffd5834f757"
        },
        {
          "id": "github.com/russross/blackfriday/v2:v2.0.1",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/cpuguy83/go-md2man/v2:v2.0.0-20190314233015-f79a8a8ca69d",
              "github.com/urfave/cli/v2:v2.3.0",
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "afd8cfd78a268f5aaa7b86924145c333ea65c603",
          "md5": "8b04dcc4504ca8943c91a4b6cc59cda3",
          "sha256": "496079bbc8c4831cd0507213e059a925d2c22bd1ea9ada4dd85815d51b485228"
        },
        {
          "id": "github.com/jfrog/gofrog:v1.1.1",
          "type": "zip",
          "requestedBy": [
            [
              "github.com/jfrog/build-info-go"
            ]
          ],
          "sha1": "438ad3217d4ccbcb20bca8bfa5c1aa5aa704f9ed",
          "md5": "dc8cea2a1424c6abd4af2a74d2e680e2",
          "sha256": "137a603a124b5bfc14d13e17dbc8f50143aa64149cf0441b5ad10f59e08e72e4"
        }
      ]
    }
  ],
  "started": "2022-01-11T19:13:50.430+0200"
}

Using the build-info

What can we do with the information stored in the build-info? What value can it bring to us?

Build-info provides valuable access to everything that happened during the build. There are many useful ways to utilize this information. One main usage for build-info is directly related to software supply chain security.

Having a detailed manifest of all the dependencies and  artifacts, allows us to quickly verify that our artifacts are clean from security vulnerabilities. Every binary typically depends on thousands of other binaries, some of them are direct dependencies, and the majority of them are transitive (indirect). The build-info includes all of them. An automatic process of matching the list dependencies inside the build-info against a vast vulnerabilities database, can make sure no vulnerable dependencies are used by your build. This process can be run frequently, to ensure the issues are found as early as possible, allowing developers to take action before a release.

This is exactly how build-info is used at JFrog for every build. Every binary we release to the world goes through this security scanning. JFrog Xray fetches the build-info published by every build to Artifactory, and scans it for security vulnerabilities. In case vulnerabilities are found, the build process gets notified, and the build ends with a failure.

The following describes how to generate build-info for one of our open source projects in GitHub, Frogbot. For this example, we’ll use the binary created by the build-info-go project.

Follow these steps:

1. Download and install the Go programming language on your machine.
2. Run the following command to make sure go is installed successfully.
   go version
3. Download the “bi” executable. You should have a file named “bi” downloaded to your local machine.
4. Grant the bi binary execute permissions by running the following command:
   chmod +x bi
5. Ensure the binary works by running the following command:
   bi -v
6. Clone the Frogbot project by running the following git command:
   git clone https://github.com/jfrog/frogbot.git
7. ‘cd’ into the cloned frogbot directory and copy the “bi” executable into it.
8. Generate the build-info by running the following command:
   bi go build

Try out build-info generated for your builds, displayed right from within your terminal.