TL;DR The Descriptor data structure forms the backbone of the OCI spec's content-addressible storage (CAS). What is it?

The Open Container Initiative (OCI) is a combination of efforts to create industry standards for container runtimes, distributions, conformance, etc...

Most of the spec is derived from the Docker V2 spec, but has since added support for many non-image based usages of containers, and truly stands on its own.

I first started diving into the spec in early 2023 when Jeff gave the Zarf team its first independent feature to implement: zarf package publish.

"This is your problem now." ~ Jeff McCoy (early 2023) ~ Wayne Starr (early 2024) ~ Me (mid 2024)

Most of you wonderful readers have probably run the docker build ,docker pull and docker push commands.

Each one of these commands is an action on an image (a filesystem broken into chunks called layers {ok, its not "broken" per se but actually a differential layered filesystem that gets reconstructed back into a rootfs at runtime, but stop distracting me!}).

When docker pushed, each layer is uploaded to a registry, then a manifest is constructed that ties the layers together and tagged at an arbitrary reference string.

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:3b25b682ea82b2db3cc4fd48db818be788ee3f902ac7378090cf2624ec2442df",
    "size": 8714
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a480a496ba95a197d587aa1d9e0f545ca7dbd40495a4715342228db62b67c4ba",
      "size": 29126289
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f3ace1b8ce45351f711f841b07ecc15383939db71555b947a9ffef6fb168ab18",
      "size": 43798364
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:11d6fdd0e8a78c038b5c013368f76279d21e5ee239f18a1e20a3593414fa1095",
      "size": 629
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f1091da6fd5cd13c1004024fdc5661a0456be67716d81671d8d2e7f81c0dbc2e",
      "size": 955
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:40eea07b53d8dc814d92f772a7b2be5c1c3914b05e3edcb5f2489e805885a0a3",
      "size": 404
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:6476794e50f4265ce2cab9c2ef2444dae937d28280a742899c8770fbca18bfed",
      "size": 1210
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:70850b3ec6b2d92e9ccdff63bbd5d1aa0dec25087cb2507165f7538ffc7029df",
      "size": 1398
    }
  ],
}

But how does a registry and client communicate to each other what is each one of these layers?

We wouldn't want to have to re-upload a 40GB image (thanks for nothing Windows) everytime we change a line of code right?

This is where the Descriptor data structure comes into view, the "atom" that forms the backbone of the OCI spec's content-addressible storage (CAS).

The Descriptor

For the best documentation on Descriptors: https://github.com/opencontainers/image-spec/blob/v1.0.1/descriptor.md

In Go it's represented by the following type:

// https://github.com/opencontainers/image-spec/blob/main/specs-go/v1/descriptor.go#L19-L50
type Descriptor struct {
  MediaType string `json:"mediaType"`
  Digest digest.Digest `json:"digest"`
  Size int64 `json:"size"`
  // other properties omitted for brevity
  // might explore some of them in another post
}

Time to break this guy down into its main three components:

MediaType

"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",

MediaType is a MIME type and is a RFC7231 compliant string that communicates what "type" of data its describing.

In OCI land, the four most commonly seen are:

  • manifest: application/vnd.oci.image.manifest.v1+json (manifests are collections of descriptors)
  • manifest config: application/vnd.oci.image.config.v1+json
  • index: application/vnd.oci.image.index.v1+json (indexes are collections of manifests)
  • "standard" layer: application/vnd.oci.image.layer.v1.tar+gzip

but the flexibility of this string is what allows for OCI to be able to store aribitary content like:

  • Helm charts: application/vnd.cncf.helm.chart.content.v1.tar+gzip
  • Zarf packages: application/vnd.zarf.layer.v1.blob
  • Flux artifacts: application/vnd.cncf.flux.content.v1.tar+gzip

The MediaType is used to convey to clients what can be done with the described blob once it has been fetched (that is, parsed as JSON, unpacked w/ tar, etc...).

Size

"size": 1398,

Size (in bytes) serves as the first component in content verification (making sure the content being passed around has'nt been corrupted or tampered with).

Size allows for registries and conformant clients to support resumable push/pull and chunked uploads.

e.g.

I'm on a slow internet connection, or the registry I am pulling from is flaky (cough cough registry1).

By knowing the size of a layer, the OCI client can coordinate with the registry and keep pulling until the image is fully fetched. Pretty neat!

Digest

"digest": "sha256:70850b3ec6b2d92e9ccdff63bbd5d1aa0dec25087cb2507165f7538ffc7029df",

Digest is the final core component and is the most "expensive" (like a Starbucks Pumpkin Spice Latte with oat milk, whipped cream and a double shot of espresso).

Before any content can be pushed, it must be hashed for content verification as well as deduplication purposes. (i.e. it's what prevents the same content from needing to be uploaded twice!)

Bringing it all together

By leveraging the MediaType of content, implementations can perform easy content support checks.

By leveraging the Size of content, implementations can take advantage of more efficient pushing/pulling of content. Resuming pulls/pushes at the byte level.

By leveraging the Digest of content as an address:

  • Registries only need to store a single copy of a layer, even if it's used in a multitude of images.
  • Clients can perform differential pushes/pulls at the layer level, greatly speeding up CI times + reducing CPU cycles.

And what boils down to two short strings and a 64bit integer form the basis for a dead simple, yet extremely powerful distribution system.

Thanks for reading and I'll see you in the next post.