Published on

XXXX

Authors
  • avatar
    Name
    Alexander Arana Escobedo
    Twitter

Intro

I have a public endpoint configured as a public listener on Azure Application Gateway (AGW). The backend pool points to our Istio Gateway, which is set up to use single TLS. When checking the backend health, I encountered the following error message:

The intermediate certificate is missing from the backend server chain. Please ensure that the certificate chain is complete and correctly presented by the backend server.

ado-output-secure-string

This error indicates that the Kubernetes secret referenced in your Istio Gateway YAML file (specifically, the secret you’ve specified in the credentialName field) doesn’t contain the complete certificate chain. To resolve this issue, ensure that the secret includes the complete certificate chain: the root certificate, intermediate certificate(s), and the server (or client) certificate.

# Set the environment
ENV=$1

SECRET_NAME=${2:- "istio-credentials"}  # Use 'istio-credentials' as default if no argument is provided

# Define variables for Key Vault access and Kubernetes integration
CERT_NAME=""
KV_NAME=""
K8S_NAMESPACE="aks-istio-ingress"
SUBSCRIPTION_NAME=""

# Set the subscription
az account set --subscription $SUBSCRIPTION_NAME

# Set the common Key Vault prefix
KV_PREFIX="kv-mgmt-$ENV-we"

echo "[*] Create a certificate temporary folder"
FOLDER_NAME="certs_temp"
mkdir $FOLDER_NAME

echo "[*] Navigate into the certs_temp folder"
cd $FOLDER_NAME

echo "[*] Download secret from Key Vault"
az keyvault secret download \
    --vault-name "$KV_NAME" \
    --name "$CERT_NAME" \
    --file certificate.pfx \
    --encoding base64

echo "[*] Extract the client/server certificate"
openssl pkcs12 -in certificate.pfx -clcerts -nokeys -out client.crt -passin pass:

echo "[*] Extract the CA certificate"
openssl pkcs12 -in certificate.pfx -cacerts -nokeys -out ca.crt -passin pass:

echo "[*] Combine client and CA certificates into a full chain"
cat client.crt ca.crt > fullchain.crt

echo "[*] Extract the Encrypted Private Key"
openssl pkcs12 -in certificate.pfx -nocerts -nodes -out encrypted.key -passin pass:

echo "[*] Decrypt the Private Key"
openssl rsa -in encrypted.key -out decrypted.key

echo "[*] Get deployment name"
DEPLOYMENT_NAME=$(kubectl get deployments \
    --namespace $K8S_NAMESPACE \
    --output jsonpath="{.items[*].metadata.name}")

echo "[*] Deleting existing secret '$SECRET_NAME' (if it exists)..."
kubectl delete secret "$SECRET_NAME" \
  --namespace $K8S_NAMESPACE \
  --ignore-not-found

echo "[*] Create secret $SECRET_NAME on cluster"
kubectl create secret tls $SECRET_NAME \
   --cert=fullchain.crt \
   --key=decrypted.key  \
   --namespace $K8S_NAMESPACE

# Restarting Istio pods ensures they load the latest TLS secrets stored in Kubernetes, preventing certificate mismatches.
echo "[*] Rollout deployment $DEPLOYMENT_NAME"
kubectl rollout restart deployments/$DEPLOYMENT_NAME \
    --namespace $K8S_NAMESPACE

echo "[*] Navigate back to the parent directory"
cd ..

echo "[*] Remove temporary folder"
rm -rf $FOLDER_NAM

💡 Tip: Validate Kustomize Locally

You can run and validate your Kustomize overlays locally using the following command:

kubectl kustomize overlays/test

or for production:

kubectl kustomize overlays/production

This command will output the final rendered YAML so you can verify that patches and deletions are applied as expected—without needing to deploy anything to your cluster. It’s a great way to catch mistakes early and ensure your Kustomize logic behaves the way you intend.

I hope this post helps you manage your Kustomize environments more effectively! If you have any questions, feel free to reach out! 🙏

Alexander Arana.E

Discuss on X