Releases

A release is one annotated vX.Y.Z tag on main. Pushing it runs the release lane, .github/workflows/release.yaml, which builds every artefact of the release, signs it with cosign keyless signing, verifies every signature, and creates the GitHub Release last. This runbook covers what a release publishes, how a maintainer cuts and rehearses one, how an operator verifies what they downloaded, and how to re-run the lane. Its last section covers the edge channel, which the edge lane, .github/workflows/edge.yaml, publishes from every push to main.

What a release publishes

Artefact	Where	Name
Container images, for linux/amd64 and linux/arm64	ghcr.io	ghcr.io/plexsphere/<image>:vX.Y.Z for plexsphere, plexsphere-signer, plexsphere-migrate, plexsphere-backup, plexsphere-bootstrap, plexctl and console
Helm charts	ghcr.io, as OCI artefacts	oci://ghcr.io/plexsphere/charts/<chart> for plexsphere, plexsphere-signer and spicedb, version X.Y.Z
plexctl binaries	GitHub Release assets	plexctl-<os>-<arch> for linux/amd64, linux/arm64, darwin/amd64, darwin/arm64 and windows/amd64 (.exe), each with a cosign bundle plexctl-<os>-<arch>.sigstore.json
Checksums	GitHub Release asset	SHA256SUMS over the plexctl binaries
SBOMs	GitHub Release assets	sbom-<image>-linux-<arch>.spdx.json, one SPDX document per image and platform
Licence and notes	GitHub Release	LICENSE, and the release notes as the Release body

Image tags carry the leading v; chart versions do not, because Helm versions charts in SemVer. A chart's appVersion is the image tag its workloads default to.

The images and charts are private packages for now: only members of the plexsphere organisation can pull them, after docker login ghcr.io and helm registry login ghcr.io with a token that can read packages. make publish mirrors the plexctl binaries, their bundles and SHA256SUMS to the public download page, get.plexsphere.com.

Signed build provenance and SBOM attestations are not published yet.

Cut a release

1. Add the version to the changelog in a pull request against main:

   make changelog RELEASE_VERSION=vX.Y.Z

   It inserts the section for vX.Y.Z above the newest one in CHANGELOG.md; Commit subjects and release notes explains which commits reach it.

2. Merge the pull request, then tag the merge commit on main with an annotated tag and push the tag:

   git fetch origin
   git tag -a vX.Y.Z -m vX.Y.Z origin/main
   git push origin vX.Y.Z

3. Watch the release run in the repository's Actions tab. It refuses a tag it must not release before it builds anything; see What the lane refuses.

4. When the run is green, mirror the plexctl downloads to the webspace:

   make publish

   make publish also republishes the docs site, while make publish-get mirrors only the downloads. Both take the newest Release unless RELEASE_VERSION names another; Docs preview and publish describes them.

Rehearse a release locally

make release builds the same artefact set on a laptop and publishes nothing:

make release RELEASE_VERSION=vX.Y.Z

It needs Docker with buildx, helm and git, and Go at the release /.go-version names. The first run creates a docker-container buildx builder named plexsphere-release, because Docker's default builder cannot write OCI image layouts. The result lands in bin/release/:

• images/<image>/: the OCI image layout of each image, for every platform;
• charts/<chart>-X.Y.Z.tgz: the packaged charts;
• assets/: exactly the GitHub Release assets, without the cosign bundles the lane adds when it signs;
• release-notes.md: the Release body.

Set RELEASE_PLATFORMS=linux/amd64 for a faster build of one platform. The nightly release-dry-run CI job runs the full build every night through make test-release.

Verify what you downloaded

The lane signs every image, chart and plexctl binary with cosign keyless signing, under the OIDC identity of the release workflow run for the tag. Install cosign, then check the identity and the issuer of a signature:

• identity: https://github.com/plexsphere/plexsphere/.github/workflows/release.yaml@refs/tags/vX.Y.Z
• issuer: https://token.actions.githubusercontent.com

An image, after docker login ghcr.io:

cosign verify ghcr.io/plexsphere/plexsphere:vX.Y.Z \
  --certificate-identity https://github.com/plexsphere/plexsphere/.github/workflows/release.yaml@refs/tags/vX.Y.Z \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

A chart, after the same login; cosign reads Docker's credentials:

cosign verify ghcr.io/plexsphere/charts/plexsphere:X.Y.Z \
  --certificate-identity https://github.com/plexsphere/plexsphere/.github/workflows/release.yaml@refs/tags/vX.Y.Z \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

A plexctl binary, with its bundle next to it:

cosign verify-blob --bundle plexctl-linux-amd64.sigstore.json \
  --certificate-identity https://github.com/plexsphere/plexsphere/.github/workflows/release.yaml@refs/tags/vX.Y.Z \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  plexctl-linux-amd64

Install plexctl covers the binary from both download channels.

Every release writes the workflow identity and the digests of its artefacts to Sigstore's public Rekor transparency log, even while the repository and its packages are private.

Re-run the lane

A failed run created no GitHub Release: the lane creates the Release last, after every signature verified. Fix the cause and run the lane again for the same tag, either with Re-run all jobs on the failed run or by dispatching the workflow from the tag:

gh workflow run release.yaml --repo plexsphere/plexsphere --ref vX.Y.Z

A dispatch from a branch is refused, because a branch name is not a release version.

The lane refuses a tag whose Release already exists. To publish a tag again, delete its Release first, then run the lane:

gh release delete vX.Y.Z --repo plexsphere/plexsphere --yes

The rebuild moves the image and chart tags to new digests, because every build stamps its build date. The digests of the earlier run stay signed in the transparency log.

What the lane refuses

scripts/release-lane.sh check stops the run before anything is built:

Message	Cause	Fix
release-lane: RELEASE_VERSION must be set	The job no longer passes the tag to the script	Restore the job's RELEASE_VERSION environment variable in release.yaml.
release-lane: GITHUB_REPOSITORY must be set	The script runs outside GitHub Actions	Run it from the workflow, which sets the variable.
release-lane: "<ref>" is not a release version like v1.2.3	The run is for a branch, or for a tag that is not vX.Y.Z, a pre-release such as v1.2.3-rc.1 included	Push, or dispatch from, a vX.Y.Z tag.
release-lane: fetching vX.Y.Z and main from origin failed	The tag is missing on the remote, or the runner cannot reach it	Push the tag, then re-run.
release-lane: tag vX.Y.Z is a lightweight tag; push an annotated tag (git tag -a vX.Y.Z)	The tag carries no tag object	Delete it with git push origin :refs/tags/vX.Y.Z, then tag again with git tag -a.
release-lane: the checkout is at <sha>, but tag vX.Y.Z points at <sha>	The tag moved after the run started	Re-run the lane for the tag.
release-lane: tag vX.Y.Z (<sha>) is not reachable from origin/main	The tagged commit is not on main	Tag a commit on main.
release-lane: a GitHub Release for vX.Y.Z already exists; delete it with "gh release delete vX.Y.Z" before re-running the lane	An earlier run published the tag	See Re-run the lane.
release-lane: cannot tell whether a Release for vX.Y.Z exists: <error>	The GitHub API refused the lookup	Re-run once the API answers.

The edge channel

Every push to main runs the edge lane, .github/workflows/edge.yaml. It builds the commit with make release, as the release lane does, pushes its images and charts to ghcr.io under an edge version, signs and verifies them with cosign keyless signing, and prunes old edge versions. It tags the images and pushes the charts only after every image verified, because a chart tag is what a cluster that follows the channel picks up, and the image tags are what that chart names. It creates no GitHub Release. A cluster that follows the channel runs every merge to main without anyone touching it.

What the channel publishes

Artefact	Where	Name
Container images, for linux/amd64 and linux/arm64	ghcr.io	ghcr.io/plexsphere/<image>:v0.0.0-edge.<stamp>.g<sha> for the seven images of a release
Helm charts	ghcr.io, as OCI artefacts	oci://ghcr.io/plexsphere/charts/<chart> for plexsphere, plexsphere-signer and spicedb, version 0.0.0-edge.<stamp>.g<sha>

The channel publishes no plexctl binaries, no SBOMs, no release notes and no GitHub Release: it is for a cluster, which pulls images and charts. make release still builds the rest, so the lane's build is the release build. The packages are private, like those of a release.

The edge version

An edge version is v0.0.0-edge.<stamp>.g<sha>, for example v0.0.0-edge.20261007124615.g7d738a9. <stamp> is the committer date of the published commit in UTC as fourteen digits (YYYYMMDDHHMMSS), and <sha> is the first seven hex digits of its commit id. The chart version is the same string without the leading v, and the appVersion of the plexsphere and plexsphere-signer charts is the edge version, so an edge chart defaults its images to the edge images of the same commit.

The version sorts the way a consumer needs it to:

• It is a SemVer pre-release of 0.0.0, so it sorts below every release, and a SemVer range without a pre-release comparator excludes it. A consumer of releases never receives an edge build.
• Within the channel the stamp compares numerically, so the newest edge version is the newest commit of main. Two pull requests cannot merge in the same second. A commit pushed straight to main with a committer date older than the commits before it sorts below them.
• A commit always gets the same version: a re-run publishes it again and moves its image tags to the new digests once the new images verified. A chart that an earlier run published and verified keeps its digest.

To build the edge version of your checkout's HEAD on a laptop, without publishing anything:

make release RELEASE_VERSION=$(scripts/release-lane.sh edge-version)

Select an edge version with Flux

A Flux OCIRepository selects the newest edge chart by SemVer range and a tag filter:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: plexsphere-edge
  namespace: plexsphere
spec:
  interval: 5m
  url: oci://ghcr.io/plexsphere/charts/plexsphere
  secretRef:
    name: plexsphere-registry
  ref:
    semver: ">=0.0.0-0"
    semverFilter: "^0\\.0\\.0-edge\\."
  layerSelector:
    mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
    operation: copy

The -0 suffix lets the range admit pre-releases, and the filter keeps release versions out of the candidates, so the newest edge chart wins. A HelmRelease whose spec.chartRef names the OCIRepository installs it, and the chart's appVersion carries the image tags. plexsphere-registry is a pull secret that can read the private packages. The Flux OCIRepository reference describes ref.semver and ref.semverFilter. The lab cluster consumes the channel this way, for both charts; the lab runbook describes its sources and how it updates.

Verify an edge image

The edge lane signs under the OIDC identity of the edge workflow run on main:

• identity: https://github.com/plexsphere/plexsphere/.github/workflows/edge.yaml@refs/heads/main
• issuer: https://token.actions.githubusercontent.com

An image, after docker login ghcr.io:

cosign verify ghcr.io/plexsphere/plexsphere:v0.0.0-edge.<stamp>.g<sha> \
  --certificate-identity https://github.com/plexsphere/plexsphere/.github/workflows/edge.yaml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

A chart verifies the same way under ghcr.io/plexsphere/charts/<chart>:0.0.0-edge.<stamp>.g<sha>. crane ls ghcr.io/plexsphere/plexsphere lists the edge versions a package holds.

Retention

After it publishes, the lane keeps the newest three edge versions of the plexsphere chart in every package and deletes the other edge versions, together with the platform manifests of their image indexes and the versions that hold their cosign signatures. An edge version the chart lacks, left by a run that failed at a chart push, is deleted even when it is newer, so every kept chart still finds its images. Release versions and the platform manifests a kept version shares are never deleted, and neither is an edge version that carries a tag besides its edge tag. A consumer more than three merges behind finds its version gone and moves to the newest one, which a Flux OCIRepository with the range above does on its next reconcile.

A run that failed at signing or verification left its images untagged. The prune step does not find them, so they count against the package quota until they are deleted by hand.

Package quota

The packages are private, so they count against the organisation's GitHub Packages quota: the Free plan includes 500 MB of private package storage and 1 GB of transfer a month, and three edge versions of seven images for two platforms are in the order of 500 MB. Set a spending limit for GitHub Packages before the first run; About billing for GitHub Packages lists the quotas.

Re-run the lane for a commit

Dispatching the workflow from main publishes the edge version of HEAD of main again:

gh workflow run edge.yaml --repo plexsphere/plexsphere --ref main

Re-run all jobs on a failed run publishes the edge version of that run's commit. Every rebuild stamps its build date, so the image tags move to new digests; the digests of the earlier run stay signed in the transparency log. The image tags move only after the new images verified, so a cluster that already runs the version's chart keeps pulling verified images while a re-run fails. helm push tags a chart before it is signed, so a re-run keeps a chart that an earlier run published and verified, and prints release-lane: keeping <reference>, which an earlier run published and verified. It pushes again only a chart that is missing or did not verify, after printing release-lane: <chart>:<version> does not verify; pushing it again for the latter. A re-run of an older commit is pruned again by its own prune step when three newer edge versions exist.

What the edge lane refuses

scripts/release-lane.sh stops the run with one of these messages:

Message	Cause	Fix
release-lane: the edge lane publishes main only, not <ref>	The workflow was dispatched from another branch	Dispatch it from main.
release-lane: cannot read HEAD: <error>	The job has no checkout with a commit	Restore the checkout step in edge.yaml.
release-lane: "<version>" is not an edge version like v0.0.0-edge.20260101000000.gabc1234	The version step did not set RELEASE_VERSION before the publish step	Restore the step order in edge.yaml.
release-lane: edge-publish runs only inside the edge workflow, which provides the OIDC token cosign signs with	The script runs outside GitHub Actions	Run it from the workflow; rehearse the build on a laptop with make release.
release-lane: bin/release is incomplete: missing images/<image>/index.json; run make release first	make release did not build every image	Fix the build, then re-run.
release-lane: pruning <package> failed: <error>	The GitHub Packages API or the registry refused the listing, a manifest read or a deletion	Re-run once the cause is fixed: the version was already published, and the next run prunes again.

The prune step deletes with the workflow's GITHUB_TOKEN: GitHub gives a repository admin access to the packages its workflows publish, and packages: write is the permission the deletion needs. Should GitHub refuse the deletion anyway, store a personal access token (classic) with the read:packages and delete:packages scopes as a repository secret and pass it to the prune step as GH_TOKEN instead of github.token.

The prune step also prints release-lane: no signature version for <package>@<digest>, skipping when the listing holds no signature for a deleted version; the run still succeeds.

Check list for the first run

The edge lane runs for real only on main, so its first run is checked by hand:

1. The GitHub Packages spending limit is set.
2. The edge run of the merge is green.
3. crane ls ghcr.io/plexsphere/plexsphere lists the run's v0.0.0-edge.<stamp>.g<sha> tag.
4. cosign verify of that image against the identity above succeeds.
5. The prune step printed release-lane: edge versions pruned from <package>: 0 for every package: nothing is deleted while a package holds three edge versions or fewer.
6. The run of the fourth merge deletes the version of the first: crane ls no longer lists it. A prune step that fails with pruning <package> failed and an HTTP 403 needs the token above.