Secure Container Image Promotion in CI/CD

Secure promotion moves one immutable image digest from build through candidate to production without rebuilding the application layer. This walkthrough on Rocky Linux 10.2 pushes the hardened secure-demo:hardened image from build secure minimal container images, runs a Trivy gate and archives a CycloneDX SBOM against that digest, copies across registry zones with digest-preserving skopeo, signs the production digest, and verifies the running workload matches the approved release record.

Build once

Environment-specific configuration belongs in ConfigMaps, Secrets, and Helm values—not in a second image build per stage.

Push build image
  ↓
Resolve BUILD_REF by digest
  ↓
Scan BUILD_REF
  ↓
Generate SBOM for BUILD_REF
  ↓
Copy BUILD_REF to candidate
  ↓
Verify candidate digest matches
  ↓
Copy candidate digest to production
  ↓
Verify production digest matches
  ↓
Sign production digest
  ↓
Verify production signature
  ↓
Deploy production digest

Rebuild only when source, dependencies, or base images change. Promotion copies bytes; it does not compile again.

Cosign signatures are separate registry objects associated with a repository and digest. Copying the image to another repository does not automatically move its Cosign signature. Sign after the production digest exists.

Separate registry trust zones

Model three repositories on one lab registry to mimic production boundaries:

127.0.0.1:5000/build/secure-demo:build-42
        ↓  (scan gate)
127.0.0.1:5000/candidate/secure-demo:rc-1
        ↓  (approval + copy)
127.0.0.1:5000/production/secure-demo:v1.0.0
        ↓  (sign digest)

Define who may:

Direct developer push to production bypasses every gate.

Record build identity

Create the release record when the build completes, then update it after each gate succeeds. Do not set scan_result, promotion status, or signature verification to complete before those commands have passed.

Initial record at build time:

{
  "release_id": "secure-demo-v1.0.0",
  "build_id": "build-42",
  "source_commit": "abc123def456",
  "builder_identity": "[email protected]",
  "base_image_digest": "sha256:597c2b4bc7f353100af9b8b06bb4f126c4a45f9d8175e25d4f01f965d5d94396",
  "built_at": "2026-07-28T12:30:00Z",
  "trivy_version": "0.72.0",
  "cosign_version": "3.1.2"
}

After scan, promotion, and signing succeed, merge these fields into the same release record. Keep release_id, source_commit, builder_identity, and the other build-time fields from the initial JSON:

{
  "release_id": "secure-demo-v1.0.0",
  "build_id": "build-42",
  "source_commit": "abc123def456",
  "builder_identity": "[email protected]",
  "base_image_digest": "sha256:597c2b4bc7f353100af9b8b06bb4f126c4a45f9d8175e25d4f01f965d5d94396",
  "built_at": "2026-07-28T12:30:00Z",
  "trivy_version": "0.72.0",
  "cosign_version": "3.1.2",
  "build_digest": "sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220",
  "candidate_digest": "sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220",
  "production_digest": "sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220",
  "scan_report": "trivy-build-42.json",
  "scan_report_sha256": "sha256:918c05f99fbe638aabd419b78be42a8a9ac2e0b08a94e6350b97eebcaab27811",
  "sbom": "sbom-build-42.cdx.json",
  "sbom_sha256": "sha256:b09f21bb66d27111314f9acf74610e710678fe06d1c958d7ef1c0e47b816d018",
  "scan_result": "pass",
  "signature_verified": true
}

Generate the archived file hashes:

sha256sum trivy-build-42.json sbom-build-42.cdx.json

918c05f99fbe638aabd419b78be42a8a9ac2e0b08a94e6350b97eebcaab27811  trivy-build-42.json
b09f21bb66d27111314f9acf74610e710678fe06d1c958d7ef1c0e47b816d018  sbom-build-42.cdx.json

Store this JSON beside scan reports, SBOM files, and signature verification logs.

Push the build artifact

Reuse the hardened application image from the secure-image guide—not the empty distroless base alone:

mkdir -p ~/promotion-lab
cd ~/promotion-lab

podman image exists secure-demo:hardened || {
  echo "Build secure-demo:hardened from the secure-image guide first"
  exit 1
}

Start the registry because this article assumes a fresh lab session:

podman run -d --name promotion-registry -p 127.0.0.1:5000:5000 quay.io/libpod/registry:2.8.2

until curl -fsS http://127.0.0.1:5000/v2/ >/dev/null; do
  sleep 1
done

Use one hostname consistently:

REGISTRY=127.0.0.1:5000
APP=secure-demo
BUILD_TAG=build-42
CANDIDATE_TAG=rc-1
PRODUCTION_TAG=v1.0.0

Push the build artifact:

podman tag secure-demo:hardened "${REGISTRY}/build/${APP}:${BUILD_TAG}"

podman push --tls-verify=false "${REGISTRY}/build/${APP}:${BUILD_TAG}"

Resolve its immutable registry digest:

BUILD_DIGEST=$(
  skopeo inspect \
    --tls-verify=false \
    "docker://${REGISTRY}/build/${APP}:${BUILD_TAG}" \
    --format '{{.Digest}}'
)

BUILD_REF="${REGISTRY}/build/${APP}@${BUILD_DIGEST}"

printf 'Build reference: %s\n' "$BUILD_REF"

Build reference: 127.0.0.1:5000/build/secure-demo@sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220

Use $BUILD_REF for every scan and SBOM step before the image enters candidate/.

Scan the build digest

Run the scan gate on the pushed build digest—not a mutable upstream tag:

trivy image --insecure --severity HIGH,CRITICAL --exit-code 1 "$BUILD_REF"

│ 127.0.0.1:5000/build/secure-demo@sha256:de3d3d... │   debian   │        0        │

Exit code 0 allows promotion. A non-zero exit blocks skopeo copy to the candidate repository.

Archive a machine-readable scan report for the same digest:

trivy image --insecure --format json --output trivy-build-42.json "$BUILD_REF"

Apply a written policy for:

• Severity threshold (HIGH,CRITICAL in this lab)
• Fixable-only reporting with --ignore-unfixed when appropriate
• Time-bound exceptions using .trivyignore expiration syntax or .trivyignore.yaml, with the owner tracked in the accompanying ticket or policy record
• Report retention (JSON or SARIF archived with build_digest)

Trivy accepts immutable image references and provides --insecure for test registries that do not use trusted TLS. See scan container images with Trivy for exit codes and ignore governance.

Generate the SBOM

Archive inventory for the exact digest under test:

trivy image --insecure --format cyclonedx --output sbom-build-42.cdx.json "$BUILD_REF"

Store sbom-build-42.cdx.json with build_digest and scanner version. Rescan the stored SBOM when the vulnerability database updates without pulling the image again—see generate container SBOMs with Trivy.

Promote without rebuilding

Copy manifest bytes between zones with skopeo. Use the scanned digest as the source and add --preserve-digests. Skopeo fails rather than silently accepting a changed destination digest.

Build to candidate:

skopeo copy \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  --preserve-digests \
  "docker://${BUILD_REF}" \
  "docker://${REGISTRY}/candidate/${APP}:${CANDIDATE_TAG}"

CANDIDATE_DIGEST=$(
  skopeo inspect \
    --tls-verify=false \
    "docker://${REGISTRY}/candidate/${APP}:${CANDIDATE_TAG}" \
    --format '{{.Digest}}'
)

test "$CANDIDATE_DIGEST" = "$BUILD_DIGEST" || {
  echo "Candidate digest mismatch"
  exit 1
}
echo "Candidate digest matches build digest"

CANDIDATE_REF="${REGISTRY}/candidate/${APP}@${CANDIDATE_DIGEST}"

Candidate to production:

skopeo copy \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  --preserve-digests \
  "docker://${CANDIDATE_REF}" \
  "docker://${REGISTRY}/production/${APP}:${PRODUCTION_TAG}"

PRODUCTION_DIGEST=$(
  skopeo inspect \
    --tls-verify=false \
    "docker://${REGISTRY}/production/${APP}:${PRODUCTION_TAG}" \
    --format '{{.Digest}}'
)

test "$PRODUCTION_DIGEST" = "$BUILD_DIGEST" || {
  echo "Production digest mismatch"
  exit 1
}
echo "Production digest matches build digest"

PRODUCTION_REF="${REGISTRY}/production/${APP}@${PRODUCTION_DIGEST}"

Candidate digest matches build digest
Production digest matches build digest

All three repositories now contain the scanned bytes. No docker build or podman build runs during promotion.

Multi-architecture promotion

For image indexes, copy every platform manifest:

skopeo copy \
  --all \
  --preserve-digests \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  "docker://${SOURCE_INDEX_REF}" \
  "docker://${DESTINATION_IMAGE}:${DESTINATION_TAG}"

--all copies the image index plus all platform manifests; without it, Skopeo defaults to the current system platform. Sign the index digest. Use cosign sign --recursive only when policy also requires signatures on every referenced platform manifest.

A runtime normally reports the selected platform-manifest digest in imageID. If the release record approves an image-index digest, record the platform digest as well before comparing it with the running container.

Sign the production digest

Sign only after the production digest exists and matches the scanned build digest:

COSIGN_KEY="$HOME/cosign-lab/cosign.key"
COSIGN_PUB="$HOME/cosign-lab/cosign.pub"

test -f "$COSIGN_KEY" || {
  echo "Missing $COSIGN_KEY"
  exit 1
}

test -f "$COSIGN_PUB" || {
  echo "Missing $COSIGN_PUB"
  exit 1
}

export COSIGN_PASSWORD=''

cosign sign --yes --key "$COSIGN_KEY" --allow-http-registry=true "$PRODUCTION_REF"

Signing artifact...
Pushing signature to: 127.0.0.1:5000/production/secure-demo

Cosign explicitly recommends signing an @sha256: reference. It distinguishes plain HTTP registries with --allow-http-registry; --allow-insecure-registry is intended for insecure TLS, such as expired or self-signed certificates.

The empty password matches the key-generation workflow used in the linked Cosign lab. Otherwise, set COSIGN_PASSWORD to the passphrase used when cosign.key was created.

Verify the same reference:

cosign verify --key "$COSIGN_PUB" --allow-http-registry=true "$PRODUCTION_REF"

The signatures were verified against the specified public key
..."docker-manifest-digest":"sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220"...

Keyless CI signing with OIDC identity and issuer verification is covered in sign and verify container images with Cosign.

Use immutable tags

Tags such as v1.0.0 and rc-1 are aliases for humans and Helm values. Policy should:

• Prevent tag overwrite in production/
• Restrict delete on promoted digests
• Block direct push to production/ outside the promotion job
• Deploy Kubernetes workloads by digest in manifests or record tag-to-digest mapping in the release JSON
• Reject deploy pipelines that resolve :latest at apply time without verifying digest

Add deployment gates

Before a cluster pulls a production image, verify:

Admission can reject a signed image if the Pod spec still runs privileged or uses hostNetwork.

Verify the running workload

The 127.0.0.1:5000 registry is used only for the workstation promotion demonstration. The following workload-drift check applies after the image has been copied to a production registry reachable by cluster nodes.

Preferred production setup: use a registry hostname or IP reachable from every Kubernetes node and configure trusted TLS. Replace $REGISTRY throughout with that address.

Lab deployment manifest using the production digest:

containers:
- name: secure-demo
  image: registry.example.com/production/secure-demo@sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220

After rollout, detect drift between what you approved and what runs:

kubectl get pod -l app=secure-demo -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'

secure-demo-7d4f8b9c6d-abcde   registry.example.com/production/secure-demo@sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220

Compare imageID digest to production_digest in the release record. Mismatch means the cluster pulled different bytes than the gated release—investigate a stale release record, manual image override, checking the wrong container status, or comparing an image-index digest with the selected platform-manifest digest.

Handle newly discovered vulnerabilities

Do not patch a digest that already shipped. The scan report and SBOM describe historical bytes.

1. Patch application source or bump the base image digest.
2. Build a new image under a new build_id.
3. Rescan with current Trivy database.
4. Regenerate SBOM for the new digest.
5. Resign with Cosign.
6. Repromote through candidate/ to production/.
7. Roll out and archive the new release record.

The old digest remains auditable as what ran before the fix.

Failure scenarios

Tag-move demonstration: after signing $PRODUCTION_REF, deliberately overwrite the production tag with a different image:

skopeo copy \
  --dest-tls-verify=false \
  docker://registry.k8s.io/pause:3.10 \
  "docker://${REGISTRY}/production/${APP}:${PRODUCTION_TAG}"

Tag-based verification against the moved tag may fail or may verify a different signed image:

cosign verify \
  --key "$COSIGN_PUB" \
  --allow-http-registry=true \
  "${REGISTRY}/production/${APP}:${PRODUCTION_TAG}"

Error: no signatures found

Digest-pinned verification still succeeds for the approved production digest:

cosign verify \
  --key "$COSIGN_PUB" \
  --allow-http-registry=true \
  "$PRODUCTION_REF"

Cosign recommends signing and verifying digest references specifically to prevent tag race and movement problems. Compare the resolved digest with the approved release record—not merely whether some signature for the current tag verifies.

Troubleshooting

What's Next

• kubectl logs, Events and describe with Examples
• Debug Kubernetes Pods with kubectl exec and kubectl debug
• Build Immutable Containers in Kubernetes

References

• CNCF Software Supply Chain Best Practices v2
• SLSA provenance
• OCI image specification

Summary

Secure promotion builds once, scans and archives artifacts against one digest, copies that digest through registry zones without rebuilding, signs the production digest, and verifies the signature before deployment. The lab pushed secure-demo:hardened to build/, scanned $BUILD_REF (sha256:de3d3d5630b993c8f6c8abb735d58901ba9b3dd2bfca1ecf4e2f2b80d24fc220 on this host), archived CycloneDX and JSON reports, promoted with digest-preserving skopeo copy to candidate/ and production/, signed $PRODUCTION_REF, and confirmed digest-pinned verification survives tag movement.

Pair this workflow with manifest scanning at deploy time and admission policy that enforces registry and digest rules in the cluster.