Published on

Solving Certificate Chain Errors with AKS and Istio Gateway

Authors
  • avatar
    Name
    Alexander Arana Escobedo
    Twitter

Intro

I have a public endpoint configured as a public listener on Azure Application Gateway (AGW). The backend pool points to our Istio Gateway, which is set up to use single TLS. When checking the backend health, I encountered the following error message:

The intermediate certificate is missing from the backend server chain. Please ensure that the certificate chain is complete and correctly presented by the backend server.

ado-output-secure-string

This error may indicate that the Kubernetes secret specified in the credentialName field of your Istio Gateway YAML is missing part of the certificate chain.

To resolve this issue, make sure the secret contains the full certificate chain: the root certificate, intermediate certificate(s), and the server (or client) certificate.

I solved this using the script below, which fetches the certificate from Azure Key Vault, extracts and prepares the certificate chain and private key, creates a Kubernetes TLS secret, and restarts Istio to apply the updated secret:

# Set the environment
ENV=$1

SECRET_NAME=${2:- "istio-credentials"}  # Use 'istio-credentials' as default if no argument is provided

# Define variables for Key Vault access and Kubernetes integration
CERT_NAME=""
KV_NAME=""
K8S_NAMESPACE="aks-istio-ingress"
SUBSCRIPTION_NAME=""

# Set the subscription
az account set --subscription $SUBSCRIPTION_NAME

echo "[*] Create a certificate temporary folder"
FOLDER_NAME="certs_temp"
mkdir $FOLDER_NAME

echo "[*] Navigate into the certs_temp folder"
cd $FOLDER_NAME

echo "[*] Download secret from Key Vault"
az keyvault secret download \
    --vault-name "$KV_NAME" \
    --name "$CERT_NAME" \
    --file certificate.pfx \
    --encoding base64

echo "[*] Extract the client/server certificate"
openssl pkcs12 -in certificate.pfx -clcerts -nokeys -out client.crt -passin pass:

echo "[*] Extract the CA certificate"
openssl pkcs12 -in certificate.pfx -cacerts -nokeys -out ca.crt -passin pass:

echo "[*] Combine client and CA certificates into a full chain"
cat client.crt ca.crt > fullchain.crt

echo "[*] Extract the Encrypted Private Key"
openssl pkcs12 -in certificate.pfx -nocerts -nodes -out encrypted.key -passin pass:

echo "[*] Decrypt the Private Key"
openssl rsa -in encrypted.key -out decrypted.key

echo "[*] Get deployment name"
DEPLOYMENT_NAME=$(kubectl get deployments \
    --namespace $K8S_NAMESPACE \
    --output jsonpath="{.items[*].metadata.name}")

echo "[*] Deleting existing secret '$SECRET_NAME' (if it exists)..."
kubectl delete secret "$SECRET_NAME" \
  --namespace $K8S_NAMESPACE \
  --ignore-not-found

echo "[*] Create secret $SECRET_NAME on cluster"
kubectl create secret tls $SECRET_NAME \
   --cert=fullchain.crt \
   --key=decrypted.key  \
   --namespace $K8S_NAMESPACE

# Restarting Istio pods ensures they load the latest TLS secrets stored in Kubernetes, preventing certificate mismatches.
echo "[*] Rollout deployment $DEPLOYMENT_NAME"
kubectl rollout restart deployments/$DEPLOYMENT_NAME \
    --namespace $K8S_NAMESPACE

echo "[*] Navigate back to the parent directory"
cd ..

echo "[*] Remove temporary folder"
rm -rf $FOLDER_NAM

There are a few ways to troubleshoot this, but what helped me was using the commands listed in the Tip section below 👇

💡 Tip: Three Handy OpenSSL Commands to Check Your TLS Certificates

To verify and inspect how your public certificate chain appears to clients connecting to your service. This helps ensure that the server correctly provides the full certificate chain, including the server, intermediate, and root certificates:

openssl s_client -connect <HOST_NAME>:443 -showcerts

To view the certificates inside a .pfx file without exposing the private key. This is useful for checking which certificates are included and viewing details like who issued them and when they expire:

openssl pkcs12 -in certificate.pfx -nokeys -info

To inspect the full details of all certificates stored in a PEM file in a readable format. This is helpful for confirming what’s included in your certificate chain:

openssl crl2pkcs7 -nocrl -certfile tls_from_kv.pem | openssl pkcs7 -print_certs -text -noout

I hope this post helps you manage your TLS certificates and Istio Gateway setup in AKS more confidently! If you have any questions, don’t hesitate to reach out! 🙏

Alexander Arana.E

Discuss on X