tldr; Refactoring and revamping component composition in Zarf, and what comes next.

History 101

Zarf has supported importing components from other zarf.yaml's for quite a while:

# taken from: https://docs.zarf.dev/ref/examples/composable-packages/
components:
  - name: local-games-path
    required: true
    description: "Example of a local composed package with a unique description for this component"
    import:
      # The local relative path to the folder containing this component's package definition
      path: ../dos-games
      # Example optional custom name to point to in the imported package (default is to use this component's name)
      name: baseline
    manifests:
      - name: multi-games
        files:
          - quake-service.yaml

Back in May of 2023 the team added Composability via OCI and introduced the idea of "skeleton" packages for this purpose.

Simply put: skeleton packages provide the ability to import components stored in an OCI repository instead of needing to re-define locally or with git submodules.

To accomplish this, I clobbered together some pretty ugly glue code to allow for this new behavior to exist within component composition, but that introduced some new and nasty bugs while also exasterbating many others.

In October of 2023, former Zarf team-lead Wayne ordered, directed me to kindly remedy these issues, as well as formulate a better strategy for composition as a whole.

The Old Way

At a high level component composition comprised of a recursive function (getChildComponent) that followed the composite pattern.

This function would:

  1. Keep track of "global" import history w/ a pathAncestry variable
  2. Validate a import definition exists and is valid in the given parent component
  3. Fetch the remote / local component and read its zarf.yaml
  4. Merge that package's ZarfVariables and ZarfConstants into the main package
  5. Verify the imported component's architecture (set w/ only.cluster.architecture) matched the main package's architecture
  6. Fix all of the component's filepaths to be relative to the main package (including actions!)
  7. Merge its values into the main package's component, following this strategy
  8. Recursively call itself until no more imports remained
  9. Perform migrations on the final composed component

One of the largest detractors from the old pattern was consistency and readability.

The parent <-> child relationship was confusing to visualize, and the recursive immediate composition made debugging annoying.

As well, this function was a bit of a "god" function, doing a lot of things at once, and was difficult to test (ie: it had no tests!). This lead to behaviors like flavor not being included in the composition lifecycle when it was first introduced.

The New Hotness

Instead of a recursive function, I opted for a doubly linked list approach.

While an array could easily have been used, I liked the visual representation of a linked list, as it made it easier to understand the relationship between components.

I called this new structure the ImportChain.

  1. An initial component is added to the chain, this also configures the chain's arch and flavor values that all imports must satisfy.
  2. This component is checked for imports, and if any are found, they are resolved and added to the chain (either by reading the local zarf.yaml or fetching the component's zarf.yaml from an OCI registry).
  3. This process continues until no more components are left to import.

The ImportChain for the default init package looks like this:

component "k3s" imports "k3s" in packages/distros/k3s, which imports "k3s" in common

component "zarf-injector" imports "zarf-injector" in packages/zarf-registry

component "zarf-seed-registry" imports "zarf-seed-registry" in packages/zarf-registry

component "zarf-registry" imports "zarf-registry" in packages/zarf-registry

component "zarf-agent" imports "zarf-agent" in packages/zarf-agent

component "logging" imports "logging" in packages/logging-pgl

component "git-server" imports "git-server" in packages/gitea

You can actually see this yourself if run zarf package create with --log-level debug!

Now that the chain is built, operations against each component can be performed in a more consistent manner.

  1. Each component is migrated according to migrations set forth in the deprecated Go package.
  2. Starting from the last component in the chain, each component is merged into the previous component according to Zarf's merge strategies.

This new approach has made the composition process much more predictable and easier to reason about.

Not featured:

  • How circular/self imports are prevented
  • How flavor/arch satisfaction is enforced
  • Why variables and constants are merged in the way they are
  • Why actions's dir needs special handling
  • How OCI (skeleton) imports are resolved and cached

Not Done Yet

While composition has greatly improved, there is an even larger potential refactor on the horizon.

This is because "skeleton" packages have secretly been broken since their inception.

sorry

The concept of "skeleton" packages is pretty simple: provide a way for components to be imported at create time from an OCI registry, thus enabling remote component composition and resulting in a more "DRY" zarf.yaml.

The creation of a skeleton package is also pretty simple (but very un-discoverable):

zarf package publish <dir>

"skeleton" packages are a special case when it comes to package creation. Unlike, regular packages, "skeleton" packages cannot be used as a standalone package, however they share the same internal structure and follow nearly identical creation steps.

This is where the problem lies: in order to create a "skeleton" package, you must first call component composition so that all imports are resolved and merged into the final package.

The entire design of ImportChain revolves around the principle that all node in the chain have a one:one relationship with each other. Each node is a component that satisifies the flavor and arch of the previous node.

In a "skeleton" package, all variants of a component must be present in the final package. The relationship between nodes becomes one:many, and the ImportChain is no longer a valid structure to represent this relationship.

A refactor like this will take some serious thought and design, and will likely involve a complete re-write of the composition process (probably from a linked list to a more DAG-like structure).

BUT that is only if we decide to keep "skeleton" packages around. They are a pretty niche feature, and I think there is potential that UDS bundles + more experienced package creators have superceded their usefulness. Musings for another day.