| Tested on | Rocky Linux 10.2 (Red Quartz) workstation |
|---|---|
| Package | kubectl 1.36.3istioctl 1.30.3 |
| Applies to | Ubuntu, Debian, Kali Linux, Linux Mint, Pop!_OS, Raspberry Pi OS, elementary OS, Zorin OS, Parrot OS, MX Linux, RHEL, Rocky Linux, AlmaLinux, Oracle Linux, CentOS Stream, Fedora Kubernetes cluster |
| Cert prep | CKS |
| Lab environment | Multi-node cluster with Istio sidecar injection enabled — install Kubernetes with kubeadm plus an existing Istio control plane |
| Privilege | cluster-admin kubectl to create namespaces, workloads, and security policies |
| Scope | Istio workload identity overview, PeerAuthentication versus DestinationRule roles, mesh enrollment checks, PERMISSIVE and STRICT policies at workload and namespace scope, auto mTLS and conflicting DestinationRule behavior, certificate and proxy verification, ambient-mode contrast notes, PERMISSIVE-to-STRICT migration steps, and troubleshooting. Assumes Istio is already installed. Does not cover full mesh administration, traffic shifting, AuthorizationPolicy, Ingress Gateway, multicluster mesh, CA replacement, or full ambient migration. |
| Related guides | Kubernetes Services Multi-tenancy and workload isolation Kubernetes security architecture |
Istio mutual TLS (mTLS) gives each enrolled workload an X.509 identity issued by the mesh certificate authority. PeerAuthentication decides whether inbound connections must present that identity. This walkthrough uses sidecar mode on a kubeadm lab running Istio 1.30.3: a mesh client and server in one namespace, a plaintext client outside the mesh, and policies that move from PERMISSIVE to STRICT.
Understand Istio identity and mTLS
Mesh security binds to workload identity, not Pod IP addresses:
- Each injected Pod receives an Envoy sidecar (sidecar mode) or is handled by a node ztunnel (ambient mode).
- Istio issues and rotates short-lived workload certificates through the control plane.
- Sidecars terminate TLS between proxies. Applications normally send plaintext traffic within the Pod network namespace. The sidecar intercepts outbound traffic, establishes mTLS to the destination proxy, and forwards decrypted traffic from the destination proxy to the application container. The application does not normally address its sidecar through
localhost. - Identity is carried in SPIFFE-style credentials, so IP rotation does not break authentication policy.
PeerAuthentication answers one question for inbound traffic: must the caller present a mesh-issued client certificate?
PeerAuthentication vs DestinationRule
Do not treat these resources as interchangeable:
| Resource | Controls | Typical question |
|---|---|---|
PeerAuthentication |
Inbound mTLS acceptance on the server | Must callers present mesh TLS? |
DestinationRule |
Outbound client TLS behavior | How should this client dial the destination? |
| Auto mTLS | Default outbound upgrade when no conflicting rule exists | Does the client sidecar use ISTIO_MUTUAL automatically? |
A common failure pattern is STRICT PeerAuthentication on the server paired with a DestinationRule that sets tls.mode: DISABLE on the client path. The server demands mTLS while the client is told to send plaintext.
Deploy a minimal test mesh
This article assumes istiod is already running. Verify the control plane before you create test workloads:
kubectl get pods -n istio-system -l app=istiodNAME READY STATUS RESTARTS AGE
istiod-7d4f9b6877-8whrg 1/1 Running 0 32mConfirm the istioctl client matches the control plane before you create test workloads:
istioctl versionclient version: 1.30.3
control plane version: 1.30.3Create two namespaces: one with sidecar injection, one without:
kubectl create namespace mtls-lab
kubectl label namespace mtls-lab istio-injection=enabled --overwrite
kubectl create namespace mtls-plainThe mtls-lab label tells Istio to inject Envoy sidecars into new Pods. mtls-plain stays outside the mesh for plaintext client tests.
Deploy an nginx server and a curl-based mesh client in mtls-lab:
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: server
namespace: mtls-lab
spec:
replicas: 2
selector:
matchLabels:
app: server
template:
metadata:
labels:
app: server
spec:
containers:
- name: nginx
image: registry.k8s.io/e2e-test-images/nginx:1.14-2
ports:
- name: http
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: server
namespace: mtls-lab
spec:
selector:
app: server
ports:
- name: http
port: 80
targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mesh-client
namespace: mtls-lab
spec:
replicas: 1
selector:
matchLabels:
app: mesh-client
template:
metadata:
labels:
app: mesh-client
spec:
containers:
- name: curl
image: curlimages/curl:8.11.1
command: ["sleep", "3600"]
EOFDeploy a plaintext client in mtls-plain:
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: plain-client
namespace: mtls-plain
spec:
replicas: 1
selector:
matchLabels:
app: plain-client
template:
metadata:
labels:
app: plain-client
spec:
containers:
- name: curl
image: curlimages/curl:8.11.1
command: ["sleep", "3600"]
EOFName the Service port http so Istio protocol discovery stays clean. Unnamed ports trigger analyzer warning IST0118.
Wait for all three Deployments to become ready before you test traffic:
kubectl rollout status deployment/server \
-n mtls-lab --timeout=120skubectl rollout status deployment/mesh-client \
-n mtls-lab --timeout=120skubectl rollout status deployment/plain-client \
-n mtls-plain --timeout=120sVerify workload enrollment
Confirm sidecars injected only where expected:
kubectl get pods -n mtls-lab -o wideNAME READY STATUS RESTARTS AGE IP NODE
mesh-client-657556c56d-n48hf 2/2 Running 0 3m 192.168.1.248 worker01
server-c778c4cbd-mmzsm 2/2 Running 0 3m 192.168.1.76 worker01
server-c778c4cbd-pppft 2/2 Running 0 3m 192.168.1.165 worker012/2 READY means application container plus istio-proxy. The plain client should stay 1/1:
kubectl get pods -n mtls-plain -o wideNAME READY STATUS RESTARTS AGE IP NODE
plain-client-5dc4667697-24znc 1/1 Running 0 3m 192.168.1.224 worker01Record the server ClusterIP for tests:
SERVER_IP=$(kubectl get service server -n mtls-lab \
-o jsonpath='{.spec.clusterIP}')
test -n "$SERVER_IP"
echo "Server ClusterIP: $SERVER_IP"Server ClusterIP: 10.101.203.197If cluster DNS is slow in your lab, use the ClusterIP directly. DNS resolution issues are separate from mTLS policy.
Define reusable bounded checks before you apply policies. Each helper retries for up to 60 seconds and exits with a non-zero status when the expected behavior never appears:
wait_for_http_code() {
expected_code=$1
namespace=$2
deployment=$3
container=$4
for attempt in $(seq 1 30); do
actual_code=$(
kubectl exec -n "$namespace" "deploy/$deployment" \
-c "$container" -- \
curl --connect-timeout 3 -sS \
-o /dev/null -w '%{http_code}' \
"http://$SERVER_IP/" 2>/dev/null || true
)
if [ "$actual_code" = "$expected_code" ]; then
echo "Observed expected HTTP code: $expected_code"
return 0
fi
sleep 2
done
echo "Expected HTTP $expected_code, last result was: ${actual_code:-no response}" >&2
return 1
}
wait_for_plaintext_rejection() {
for attempt in $(seq 1 30); do
if ! kubectl exec -n mtls-plain deploy/plain-client \
-c curl -- \
curl --connect-timeout 3 -fsS \
"http://$SERVER_IP/" >/dev/null 2>&1; then
echo "Plaintext client rejected"
return 0
fi
sleep 2
done
echo "Plaintext client was still accepted after 60 seconds" >&2
return 1
}Baseline connectivity before any PeerAuthentication:
kubectl exec -n mtls-lab deploy/mesh-client -c curl -- \
curl -s -o /dev/null -w 'http_code=%{http_code}\n' "http://$SERVER_IP/"http_code=200kubectl exec -n mtls-plain deploy/plain-client -c curl -- \
curl -s -o /dev/null -w 'http_code=%{http_code}\n' "http://$SERVER_IP/"http_code=200Both clients reach the server before policy hardening.
Start with PERMISSIVE
PERMISSIVE accepts plaintext and mTLS on the inbound listener. Use it while migrating callers onto the mesh.
Apply a namespace default:
kubectl apply -f - <<'EOF'
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: mtls-lab
spec:
mtls:
mode: PERMISSIVE
EOFConfirm the policy:
kubectl get peerauthentication -n mtls-labNAME MODE AGE
default PERMISSIVE 10sConfirm all expected proxies are connected, then use the bounded traffic checks to wait for the new behavior:
istioctl proxy-statusNAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES
mesh-client-657556c56d-n48hf.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-mmzsm.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-pppft.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)The displayed proxy-status output proves the expected proxies are registered and subscribed. It does not itself assert that a specific policy change has reached every proxy. Istio documents SYNCED, STALE, and NOT SENT as the relevant xDS acknowledgement states; the behavioral retries below provide the definitive result for this lab.
wait_for_http_code 200 mtls-lab mesh-client curl
wait_for_http_code 200 mtls-plain plain-client curlObserved expected HTTP code: 200
Observed expected HTTP code: 200Under PERMISSIVE, the non-mesh client still succeeds.
Inspect the mesh client certificate chain:
istioctl proxy-config secret deploy/mesh-client -n mtls-labRESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true e18b7784094c3bd55e94b7cfffd035de 2026-07-29T07:27:04Z 2026-07-28T07:25:04Z
ROOTCA CA ACTIVE true fea5e88dad4468fc3aac536e62946b47 2036-07-25T06:58:07Z 2026-07-28T06:58:07ZThe default cert chain is the workload identity Envoy presents on outbound mesh connections. Use proxy-config secret and deliberate policy tests to confirm mTLS behavior.
Apply workload-level STRICT
Lock down only the server Deployment with a selector:
kubectl apply -f - <<'EOF'
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: server-strict
namespace: mtls-lab
spec:
selector:
matchLabels:
app: server
mtls:
mode: STRICT
EOFList policies in the namespace:
kubectl get peerauthentication -n mtls-labNAME MODE AGE
default PERMISSIVE 3s
server-strict STRICT 1sThe server Pods inherit STRICT from server-strict. Other workloads in the namespace still follow the PERMISSIVE default.
Confirm all expected proxies are connected, then use the bounded traffic checks to wait for the new behavior:
istioctl proxy-statusNAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES
mesh-client-657556c56d-n48hf.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-mmzsm.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-pppft.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)wait_for_http_code 200 mtls-lab mesh-client curl
wait_for_plaintext_rejectionObserved expected HTTP code: 200
Plaintext client rejectedIstio automatically configures mesh clients for mTLS, while PeerAuthentication changes what the server accepts. Both updates must reach the proxies before the test result is reliable. A single immediate curl can still succeed if Envoy has not yet applied STRICT on the server listener; the bounded helpers fail explicitly when propagation does not complete within 60 seconds.
Inspect the failure directly:
kubectl exec -n mtls-plain deploy/plain-client -c curl -- \
curl -s -o /dev/null -w 'http_code=%{http_code}\n' "http://$SERVER_IP/"curl: (56) Recv failure: Connection reset by peer
http_code=000
command terminated with exit code 56connection reset by peer is the expected symptom when plaintext HTTP meets a STRICT mTLS listener.
Apply namespace-level STRICT
Remove the workload-specific policy and tighten the whole namespace:
kubectl delete peerauthentication server-strict -n mtls-lab
kubectl apply -f - <<'EOF'
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: default
namespace: mtls-lab
spec:
mtls:
mode: STRICT
EOFPolicy inheritance in practice:
- Mesh-wide policies live in the root namespace (
istio-systemby default) and apply unless overridden. - Namespace
defaultwithout a selector applies to every workload in that namespace. - Workload selectors narrow scope to matching Pod labels and override broader defaults for those Pods.
Verify namespace STRICT:
kubectl get peerauthentication default -n mtls-lab -o yaml | grep -A2 'mode:'mode: STRICTConfirm all expected proxies are connected, then use the bounded traffic checks to wait for the new behavior:
istioctl proxy-statusNAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES
mesh-client-657556c56d-n48hf.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-mmzsm.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-pppft.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)wait_for_http_code 200 mtls-lab mesh-client curl
wait_for_plaintext_rejectionObserved expected HTTP code: 200
Plaintext client rejectedPlain client with verbose output:
kubectl exec -n mtls-plain deploy/plain-client -c curl -- \
curl -sv "http://$SERVER_IP/" 2>&1 | tail -6> GET / HTTP/1.1
> Host: 10.101.203.197
> User-Agent: curl/8.11.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* closing connection #0
command terminated with exit code 56Every destination workload in mtls-lab now requires mesh mTLS on inbound connections. Callers outside the mesh cannot satisfy that requirement.
Explain DestinationRule interaction
With no custom rules, Istio auto mTLS upgrades outbound connections between enrolled clients and STRICT servers.
List DestinationRules:
kubectl get destinationrule -ANo resources foundAuto mTLS is active when no conflicting rule exists.
Create a conflicting DISABLE rule to see client-side breakage:
kubectl apply -f - <<'EOF'
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: server-disable-mtls
namespace: mtls-lab
spec:
host: server.mtls-lab.svc.cluster.local
trafficPolicy:
tls:
mode: DISABLE
EOFConfirm all expected proxies are connected, then use the bounded traffic checks to wait for the new behavior. The mesh client should fail even though PeerAuthentication is still STRICT:
istioctl proxy-statusNAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES
mesh-client-657556c56d-n48hf.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-mmzsm.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-pppft.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)wait_for_http_code 503 mtls-lab mesh-client curlObserved expected HTTP code: 503Delete the conflicting rule to restore mesh traffic:
kubectl delete destinationrule server-disable-mtls -n mtls-labistioctl proxy-statusNAME CLUSTER ISTIOD VERSION SUBSCRIBED TYPES
mesh-client-657556c56d-n48hf.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-mmzsm.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)
server-c778c4cbd-pppft.mtls-lab Kubernetes istiod-7d4f9b6877-8whrg 1.30.3 4 (CDS,LDS,EDS,RDS)wait_for_http_code 200 mtls-lab mesh-client curlObserved expected HTTP code: 200Outbound DISABLE forces plaintext from the client proxy while the server still demands mTLS. Align PeerAuthentication and DestinationRule on every client path before you declare the migration complete.
Verify encryption and identity
Application HTTP 200 is necessary but not sufficient. Collect proxy-level evidence:
- Active workload certificate from
istioctl proxy-config secret - Mesh request succeeds under STRICT
- Plaintext request to the same Service fails
- Conflicting
DestinationRulebreaks the mesh request - Removing the conflict restores HTTP 200
Read server sidecar logs for certificate issuance:
kubectl logs -n mtls-lab deploy/server -c istio-proxy | grep 'workload certificate' | tail -1info cache generated new workload certificate resourceName=default latency=7.483742927s ttl=23h59m59.21183335sRun analyzer warnings before production rollout:
istioctl analyze -n mtls-labFix named Service ports and duplicate injection webhooks if reported. Overlapping istio-sidecar-injector and revision-tag webhooks can inject sidecars twice on some clusters.
Controlled failure remains the strongest negative test: plaintext clients must fail under STRICT while mesh clients succeed with the same Service IP.
Ambient-mode notes
This lab used classic sidecar injection (2/2 Pods). Istio ambient mode shifts the datapath:
- A per-node ztunnel provides transparent L4 mTLS and identity without an injected application sidecar.
- Optional waypoint proxies add L7 policy when you need it.
PeerAuthenticationSTRICT still rejects non-mesh plaintext paths; ambient is designed so bypassing the secure overlay is harder.DISABLEis not supported the same way in ambient; validate port-level exceptions against ambient docs before copying sidecar YAML.- Sidecar-specific
DestinationRuleTLS subtleties may not map one-to-one; test outbound behavior per destination in ambient clusters.
Treat ambient as a separate enrollment and verification path. The PERMISSIVE-to-STRICT policy concepts transfer, but the datapath inspection commands differ.
Migrate PERMISSIVE to STRICT
Use this sequence on production meshes:
- Inventory east-west callers for each Service.
- Enroll intended clients with injection or ambient labels.
- Confirm auto mTLS with
proxy-config secreton each client Deployment. - Apply namespace
PERMISSIVEand watch for plaintext stragglers. - Move critical servers to workload-level STRICT first.
- Fix plaintext clients that still need access (inject sidecars or route through a gateway).
- Expand namespace
defaultto STRICT. - Monitor connection resets, 503 responses, and proxy logs during the window.
Skipping enrollment before namespace STRICT produces the same connection reset by peer failures you tested with plain-client.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Pod stays 1/1 in injected namespace |
Missing istio-injection=enabled label or webhook failure |
Label namespace; check istiod and injector Pods |
Mesh client 503 under STRICT |
DestinationRule with tls.mode: DISABLE |
Remove or change conflicting rule; prefer auto mTLS |
| Plain client still works under STRICT | Policy selector too narrow or wrong namespace | Confirm kubectl get peerauthentication -A; widen to namespace default STRICT |
| Plain client fails during PERMISSIVE | Accidental STRICT selector on server | Delete workload-level STRICT until migration completes |
istioctl authn tls-check missing |
Removed in recent Istio releases | Use istioctl proxy-config secret and test traffic |
Analyzer IST0118 on Service |
Unnamed port | Name ports http, grpc, or protocol-specific names |
| Headless Service or direct Pod IP behaves differently | Client bypasses Service-based routing | Test through ClusterIP and document direct-IP paths separately |
What's Next
- Build Secure Minimal Container Images
- Scan Container Images for Vulnerabilities with Trivy
- Generate and Scan Container SBOMs with Trivy
References
- Istio security concepts
- PeerAuthentication reference
- Mutual TLS migration task
- Istio ambient overview
Summary
You configured Istio mTLS through PeerAuthentication, not by changing application code. Mesh workloads received rotated certificates in Envoy, while PeerAuthentication defined whether inbound connections had to present them. PERMISSIVE let both mesh and plaintext clients reach the server; STRICT at workload and namespace scope blocked the plain client with connection reset by peer while the mesh client kept returning HTTP 200.
The DestinationRule exercise showed why inbound and outbound policy must align: tls.mode: DISABLE on the client turned mesh traffic into HTTP 503 even with STRICT server policy. Verify with istioctl proxy-config secret, sidecar certificate logs, istioctl proxy-status, and deliberate plaintext failure tests rather than trusting status codes or response headers alone.
For underlay encryption between nodes regardless of mesh enrollment, see Cilium WireGuard. For L4 allow lists around mesh traffic, pair mTLS with NetworkPolicy once enrollment is stable.

