| Tested on | Rocky Linux 10.2 (Red Quartz) workstation |
|---|---|
| Package | podman 5.8.2kubectl 1.36.3registry:3 (container image) |
| Applies to | Ubuntu, Debian, Kali Linux, Linux Mint, Pop!_OS, Raspberry Pi OS, elementary OS, Zorin OS, Parrot OS, MX Linux, RHEL, Rocky Linux, AlmaLinux, Oracle Linux, CentOS Stream, Fedora Kubernetes cluster |
| Cert prep | CKAD · CKS |
| Lab environment | Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm |
| Privilege | Normal user for Podman and kubectl; sudo on cluster nodes to configure containerd registry trust for HTTP registries |
| Scope | Registry purpose, internet versus internal registry placement, optional registry authentication, direct versus authenticated cluster pulls, imagePullSecret creation methods, ServiceAccount attachment, imagePullPolicy, and first-line pull troubleshooting. Does not cover Harbor, cloud IAM, image signing, or full ImagePullBackOff remediation. |
| Related guides | Kubernetes secrets Deployments and rolling updates Kubernetes architecture |
Worker nodes pull images from a registry when Pods start. This guide covers registry access, cluster pull credentials, and referencing images from Pod or Deployment manifests.
Quick Reference
| Reader path | What you will do | Section |
|---|---|---|
| End-to-end lab | Run an internal registry, push an image, configure nodes, deploy with imagePullSecrets |
Part 1 through Part 3 |
imagePullSecrets only |
Registry already exists — create a pull Secret and attach it to a Pod or ServiceAccount | Part 3 |
| Pull without a Secret | Registry allows anonymous access (internet public repo or internal host without auth) | Part 2 — internal anonymous pull |
| Concepts first | Why a registry sits between your build and the cluster; placement vs authentication | Why registries and Understand placement and authentication |
| Troubleshooting | ImagePullBackOff, 401 unauthorized, HTTP trust on nodes |
Troubleshooting |
Lab defaults: authenticated registry at 192.168.56.108:5000 (labuser / labpass) and, for comparison, open registry at 192.168.56.108:5001. Substitute your own hostnames and credentials in production.
Why Kubernetes needs a container registry
Three facts explain why a registry sits in the path:
- A container image is a packaged filesystem and metadata
- Kubernetes schedules Pods; it does not host image storage
- The registry is the handoff between build tools on your laptop and pulls on worker nodes
Skipping the push step is the most common beginner mistake. Local Podman or Docker storage is not visible to remote nodes.
For the full build-and-tag workflow, see Build and push container images. This article focuses on registry setup and how the cluster authenticates when it pulls.
Two independent choices are often lumped together as “public vs private”:
| Term people use | What it usually means |
|---|---|
| Public registry host | On the internet and reachable broadly (registry.k8s.io, docker.io) |
| Internal registry host | On your LAN or VPC only (192.168.56.108:5000, registry.corp.local) |
| Anonymous pull | Registry serves layers without login |
| Authenticated pull | Registry returns 401 until username and password or token are supplied |
Kubernetes cares about the second pair when deciding whether imagePullSecrets are required:
- An internal registry can allow anonymous pulls on a trusted network
- A registry on the internet can still require credentials for each repository
| Registry example | Host placement | Anonymous pull? | imagePullSecrets on Kubernetes |
|---|---|---|---|
registry.k8s.io/pause |
Internet | Yes | No |
docker.io/library/nginx (public repo) |
Internet | Yes | No |
192.168.56.108:5001/web (no auth configured) |
Internal lab | Yes | No — but nodes must reach the host and trust HTTP/TLS |
192.168.56.108:5000/web (htpasswd enabled) |
Internal lab | No | Yes |
quay.io/myorg/private-app |
Internet | No | Yes |
Public registries already run on the internet. You do not install them for normal Kubernetes work — you reference a public image name in a manifest:
registry.k8s.io/pause:3.10
docker.io/library/nginx:alpineFor this path you do not need:
- A registry container on your lab host
- An
imagePullSecretwhen the repository allows anonymous pulls
The lab sections below run internal registries on your LAN so you can compare anonymous and authenticated pulls on the same host.
Part 1 — Set up and publish to an internal registry
Open registry without authentication (port 5001)
On a trusted lab network, you can run registry:3 without REGISTRY_AUTH:
- Anyone who can reach the host may push and pull
- That is convenient for learning
- It is unsafe on open networks — use authentication when the host is broadly reachable
sudo mkdir -p /opt/lab-registry-open/data
sudo podman run \
-d \
--name lab-registry-open \
-p 5001:5000 \
-v /opt/lab-registry-open/data:/var/lib/registry:Z docker.io/library/registry:3Anonymous catalog access should succeed. Registry health checks use the curl command with -w to print only the HTTP status code:
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5001/v2/_catalogSample output:
200200 means no login is required.
Authenticated registry with htpasswd (port 5000)
When the internal registry must enforce access control, enable htpasswd basic authentication. The rest of this article uses port 5000 with credentials.
Install htpasswd if needed. On Rocky or RHEL nodes, install the tools package with dnf command:
sudo dnf install -y httpd-toolsCreate storage, auth file, and a lab user:
sudo mkdir -p /opt/lab-registry/{data,auth}
sudo htpasswd -Bbn labuser labpass | sudo tee /opt/lab-registry/auth/htpasswd > /dev/nullStart the registry with Podman:
sudo podman run \
-d \
--name lab-registry \
-p 5000:5000 \
-v /opt/lab-registry/data:/var/lib/registry:Z \
-v /opt/lab-registry/auth:/auth:Z \
-e REGISTRY_AUTH=htpasswd \
-e REGISTRY_AUTH_HTPASSWD_REALM="Lab Registry" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd docker.io/library/registry:3On SELinux-enforcing RHEL-family hosts, append :Z to bind mounts so Podman can relabel host directories for container access.
Confirm the container is running:
sudo podman ps --filter name=lab-registry --format '{{.Names}} {{.Status}}'Sample output:
lab-registry Up 5 secondsAnonymous catalog access should fail; authenticated access should succeed:
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5000/v2/_catalogSample output:
401curl -s -u labuser:labpass http://127.0.0.1:5000/v2/_catalogSample output:
{"repositories":[]}401 without credentials confirms authentication is enabled. The empty list is normal before the first push.
Build and push the test image
Allow insecure pushes to the lab registries from your workstation:
sudo tee /etc/containers/registries.conf.d/999-lab-registry.conf <<'EOF'
[[registry]]
location = "192.168.56.108:5000"
insecure = true
[[registry]]
location = "192.168.56.108:5001"
insecure = true
EOFBuild a small image and publish it to both internal registries:
mkdir kubernetes-private-registry-demo && cd kubernetes-private-registry-demo<!-- index.html -->
<!DOCTYPE html>
<html><body><h1>Private registry demo v1.0.0</h1></body></html># Containerfile
FROM docker.io/library/nginx:alpine
COPY index.html /usr/share/nginx/html/index.htmlpodman build -t localhost/web:v1.0.0 .
podman tag localhost/web:v1.0.0 192.168.56.108:5001/web:v1.0.0
podman push --tls-verify=false 192.168.56.108:5001/web:v1.0.0Confirm the open-registry tag exists:
curl -s http://192.168.56.108:5001/v2/web/tags/listSample output:
{"name":"web","tags":["v1.0.0"]}Push the same image to the authenticated registry on port 5000:
podman tag localhost/web:v1.0.0 192.168.56.108:5000/web:v1.0.0
podman login 192.168.56.108:5000 -u labuser -p labpass --tls-verify=false
podman push 192.168.56.108:5000/web:v1.0.0 --tls-verify=falseConfirm the authenticated-registry tag exists:
curl -s -u labuser:labpass http://192.168.56.108:5000/v2/web/tags/listSample output:
{"name":"web","tags":["v1.0.0"]}Configure nodes for HTTP registries
Registry transport on nodes depends on how the host is exposed:
- Production — use HTTPS with a trusted certificate
- Lab — plain HTTP on a private network is common
- containerd default — HTTPS only; each node needs an explicit HTTP endpoint for port
5000,5001, or any internal registry hostname
On every node that pulls from an internal HTTP registry, ensure /etc/containerd/config.toml exposes the per-host certificate directory. In the containerd 2.x plugin tree, add or confirm:
[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"Do not duplicate this table if your file already defines config_path. Official containerd documentation places config_path in the main configuration.
Create a per-host directory under /etc/containerd/certs.d/ for each registry endpoint. Example for the authenticated lab registry on port 5000:
Create /etc/containerd/certs.d/192.168.56.108:5000/hosts.toml:
server = "http://192.168.56.108:5000"
[host."http://192.168.56.108:5000"]
capabilities = ["pull", "resolve", "push"]
skip_verify = trueRepeat the same pattern for 192.168.56.108:5001/hosts.toml when you run the no-authentication registry on port 5001.
Restart containerd and kubelet. On systemd nodes, use systemctl command to reload both daemons after registry trust changes:
sudo systemctl restart containerd kubeletConfirm containerd loaded the registry path:
sudo containerd config dump | grep -A2 'registry'Sample output:
[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/containerd/certs.d"Without the node configuration, pulls fail with http: server gave HTTP response to HTTPS client even when credentials are correct. Part 2 tests cluster pulls against the images you just published.
Part 2 — Test direct and authenticated image pulls
| Pull mode | When it applies | Kubernetes configuration |
|---|---|---|
| Direct pull | Registry allows anonymous access | image: reference only — no imagePullSecrets |
| Authenticated pull | Registry denies anonymous pulls and requires credentials | imagePullSecrets or ServiceAccount with pull Secrets |
Public image without imagePullSecrets
apiVersion: v1
kind: Pod
metadata:
name: public-pull
namespace: registry-access-demo
spec:
containers:
- name: pause
image: registry.k8s.io/pause:3.10kubectl create namespace registry-access-demo
kubectl apply -f public-pull.yamlkubectl get pod public-pull -n registry-access-demoSample output:
NAME READY STATUS RESTARTS AGE
public-pull 1/1 Running 0 8skubelet pulled registry.k8s.io/pause:3.10 without a pull Secret.
Internal anonymous pull (port 5001)
Deploy a Pod that references the image you pushed to port 5001:
apiVersion: v1
kind: Pod
metadata:
name: internal-open-pull
namespace: registry-access-demo
spec:
containers:
- name: web
image: 192.168.56.108:5001/web:v1.0.0kubectl apply -f internal-open-pull.yamlkubectl get pod internal-open-pull -n registry-access-demoSample output:
NAME READY STATUS RESTARTS AGE
internal-open-pull 1/1 Running 0 8sNo imagePullSecrets were required.
Authenticated pull without credentials (expected failure)
Deploy a Pod that references the image on port 5000 without pull credentials:
apiVersion: v1
kind: Pod
metadata:
name: private-pull-fail
namespace: registry-access-demo
spec:
containers:
- name: web
image: 192.168.56.108:5000/web:v1.0.0kubectl apply -f private-pull-fail.yamlSample events from kubectl describe pod private-pull-fail -n registry-access-demo:
Warning Failed kubelet Failed to pull image "192.168.56.108:5000/web:v1.0.0": ... authorization failed: no basic auth credentials
Warning Failed kubelet Error: ErrImagePull
Warning Failed kubelet Error: ImagePullBackOffThe registry is reachable, but kubelet has no username or password. Remove the failed Pod before Part 3:
kubectl delete pod private-pull-fail -n registry-access-demoPart 3 — Create and use imagePullSecrets
imagePullSecrets is a Pod-spec field that lists Secret names. Each pull Secret must satisfy:
- Type —
kubernetes.io/dockerconfigjson(recommended) or legacykubernetes.io/dockercfg - Data key —
.dockerconfigjson, a JSON document in the same shape as Docker or Podman auth files - Scope — namespace-scoped; kubelet reads it when pulling for Pods in that namespace
Pod spec.imagePullSecrets → Secret name (namespace-scoped)
↓
Secret type kubernetes.io/dockerconfigjson
↓
Key .dockerconfigjson holds registry host → auth entry
↓
kubelet passes credentials to containerd during pullThree common ways to create that Secret:
| Method | Command or source | Best for |
|---|---|---|
| Typed docker-registry Secret | kubectl create secret docker-registry |
Username + password or access token |
| Import auth file | kubectl create secret generic --from-file=.dockerconfigjson=… |
After podman login or docker login on the workstation |
| Manual manifest | kind: Secret with type: kubernetes.io/dockerconfigjson |
GitOps, operators, automation |
All methods produce the same Secret type. Pick based on where credentials already exist.
Option 1 — docker-registry Secret
kubectl create secret docker-registry builds the .dockerconfigjson content for you. Use it when you know the registry hostname, username, and secret string.
For the lab registry:
kubectl create secret docker-registry regcred \
--namespace=registry-access-demo \
--docker-server=192.168.56.108:5000 \
--docker-username=labuser \
--docker-password='labpass'Sample output:
secret/regcred createdConfirm the type:
kubectl get secret regcred -n registry-access-demoSample output:
NAME TYPE DATA AGE
regcred kubernetes.io/dockerconfigjson 1 1sThe --docker-password flag accepts passwords and tokens. The name is historical from Docker Hub:
| Registry | --docker-username |
--docker-password value |
|---|---|---|
Lab registry:3 + htpasswd |
labuser |
Password from htpasswd (labpass in this lab) |
| Docker Hub | Account name | Personal access token (preferred) or account password |
| Quay.io | Robot account name | Robot token |
| GitHub Container Registry | GitHub username | PAT with read:packages |
Cloud docs may label tokens ACCESS_TOKEN or REGISTRY_TOKEN. Kubernetes does not read those variable names — paste the token string into --docker-password.
Option 2 — import Podman or Docker auth file
If you already ran podman login, copy the auth file into a Secret instead of retyping credentials:
kubectl create secret generic regcred-from-file \
--namespace=registry-access-demo \
--from-file=.dockerconfigjson="${XDG_RUNTIME_DIR}/containers/auth.json" \
--type=kubernetes.io/dockerconfigjsonDocker default path:
kubectl create secret generic regcred-from-file \
--namespace=registry-access-demo \
--from-file=.dockerconfigjson="$HOME/.docker/config.json" \
--type=kubernetes.io/dockerconfigjsonThe file must contain an auths entry for the registry hostname in your image: field. It cannot rely on:
credsStorealonecredHelpersalone
If either is present without a direct auths entry, kubelet cannot use the file. Create the Secret with Option 1 instead.
Option 1 and Option 2 create interchangeable Secrets. Use one pull Secret name in the Pod (regcred or regcred-from-file).
Option 3 — manual Secret manifest
For GitOps pipelines, embed base64-encoded .dockerconfigjson in a manifest:
apiVersion: v1
kind: Secret
metadata:
name: regcred-manual
namespace: registry-access-demo
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: <base64-encoded-docker-config>Generate the JSON with kubectl create secret docker-registry --dry-run=client -o yaml and copy the data field, or encode an auth file with base64 -w0. Treat the manifest like a password file.
Attach to a Deployment
Reference the Secret from the Pod template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: private-web
namespace: registry-access-demo
spec:
replicas: 2
selector:
matchLabels:
app: private-web
template:
metadata:
labels:
app: private-web
spec:
imagePullSecrets:
- name: regcred
containers:
- name: web
image: 192.168.56.108:5000/web:v1.0.0
imagePullPolicy: IfNotPresentApply and verify:
kubectl apply -f private-web.yamlSample output:
deployment.apps/private-web createdkubectl rollout status deployment/private-web -n registry-access-demo --timeout=90sSample output:
deployment "private-web" successfully rolled outkubectl get pods -n registry-access-demo -l app=private-webSample output:
NAME READY STATUS RESTARTS AGE
private-web-5875c679f6-2mn56 1/1 Running 0 15s
private-web-5875c679f6-8p7r2 1/1 Running 0 15sBoth Pods pulled the private image after imagePullSecrets supplied credentials.
Attach to a ServiceAccount
Repeating imagePullSecrets in every Deployment is tedious. Attach the registry Secret to a ServiceAccount instead:
apiVersion: v1
kind: ServiceAccount
metadata:
name: private-registry
namespace: registry-access-demo
imagePullSecrets:
- name: regcredkubectl apply -f private-registry-sa.yamlReference the ServiceAccount from the Pod template and remove duplicate imagePullSecrets unless you intentionally override them:
spec:
template:
spec:
serviceAccountName: private-registryKubernetes copies ServiceAccount imagePullSecrets into new Pods that do not already define their own list. See Kubernetes ServiceAccounts for token and automount details.
A Pod can list multiple pull Secrets when it uses images from more than one private registry:
imagePullSecrets:
- name: quay-regcred
- name: internal-regcredimagePullPolicy, tags, and digests
imagePullPolicy tells kubelet whether to contact the registry when a container starts:
| Policy | Behaviour |
|---|---|
Always |
Resolve through the registry on every start |
IfNotPresent |
Skip the registry when the node already has the reference cached |
Never |
Never pull; the image must already exist locally |
Omitted fields default from the image reference:
:latestor no tag —Always- Any other explicit tag —
IfNotPresent - Digest reference —
IfNotPresent
Kubernetes sets this default when the object is first created. Changing the image reference later does not automatically change an existing imagePullPolicy field.
After the private-web Pods were running, the node recorded this digest:
192.168.56.108:5000/web@sha256:79ab0838785cba5ae5a2ac5ca38397c50ab66f3f2f2dce9d2b88648a62a8faa0Pin by digest when you need immutable rollouts:
image: 192.168.56.108:5000/web@sha256:79ab0838785cba5ae5a2ac5ca38397c50ab66f3f2f2dce9d2b88648a62a8faa0Troubleshoot pull failures
| Symptom | Likely cause | First check |
|---|---|---|
authorization failed: no basic auth credentials |
Authenticated registry but missing imagePullSecrets |
Secret referenced in Pod namespace |
unauthorized or 401 |
Wrong password or expired token | --docker-password value |
http: server gave HTTP response to HTTPS client |
HTTP registry without containerd hosts.toml |
Node registry config |
pull access denied |
Account lacks repository permission | Registry user or robot scope |
manifest unknown |
Tag does not exist | Registry tags list |
| Old image still runs | Mutable tag with IfNotPresent cache |
Digest pinning |
Read Pod events first:
kubectl describe pod <pod-name> -n registry-access-demokubectl get events -n registry-access-demo --sort-by=.lastTimestampWhat's Next
- Choose a Kubernetes Workload Resource
- Kubernetes Pods and Pod Lifecycle
- Differences and When to Use Each
References
Summary
Separate three concerns when you design registry access:
- Where images are stored — internet host versus internal host
- Whether authentication is enforced — anonymous pull versus credentials required
- How credentials are packaged —
kubectl create secret docker-registry, auth-file import, or manual Secret
Direct pulls work for any registry that allows anonymous access, including an internal HTTP registry on a trusted network. When anonymous pulls are denied and the registry requires credentials, create a kubernetes.io/dockerconfigjson Secret in the Pod namespace and reference it through imagePullSecrets or a ServiceAccount.

