VMware Aria Automation Provisioning Fails with "Cloud-init phone_home module callback had expired" After Certificate Rotation
search cancel

VMware Aria Automation Provisioning Fails with "Cloud-init phone_home module callback had expired" After Certificate Rotation

book

Article ID: 417899

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Provisioning Virtual Machines (VMs) using VMware Aria Automation  fails when the cloud-init script attempts to complete the phone_home callback back to the vRA instance.

The specific error message observed in vRA is:

"Cloud-init phone_home module callback had expired."

  • This issue affects all provisioning workflows utilizing the cloud-init callback mechanism across multiple environments
  • This issues is confirmed to be related to a recently rotated Root Certificate Authority (CA) certificate in Aria Automation 
  • The core problem is a TLS/SSL handshake failure because the newly provisioned VM does not trust the vRA server's certificate.

Environment

VMware Aria Automation 8.x

Cause

The underlying cause is that the provisioned Virtual Machines do not trust the new Root CA certificate of the vRA instance. A recent certificate rotation introduced a new Root CA that is not present in the trusted certificate store of the VM templates used for provisioning. This prevents the cloud-init module on the new VM from establishing a secure, trusted connection to vRA, causing the callback to fail and expire.

Resolution

Modify the cloud-init section of the affected vRA blueprints to inject the new Root CA certificate into the VM's trust store during initial provisioning.

  1. Extract the Aria Automation Root CA Certificate Run the following command on a host that trusts the new certificate (or the vRA appliance) to extract the Root CA certificate and save it to a file (e.g., vra-root-ca.pem):

     
    aria_host="<vra-fqdn>"; aria_ssl_port="443"; openssl s_client -showcerts -connect ${aria_host}:${aria_ssl_port} </dev/null 2>/dev/null | \
    awk '/-----BEGIN CERTIFICATE-----/ {flag=1; cert=""} 
         flag {cert = cert $0 "\n"} 
         /-----END CERTIFICATE-----/ {flag=0; last=cert} 
         END {printf "%s", last}' > vra-root-ca.pem
    

    Note: The awk command isolates the last certificate in the chain, which is typically the Root CA.

  2. Modify the vRA Blueprint's cloud-init Script Edit the affected blueprints and add a ca-certs section to the cloud-init configuration. Paste the full content of the PEM-encoded Root CA certificate obtained in Step 1 into the trusted array.

    YAML
     
    #cloud-config
    ca-certs:
      remove_defaults: false
      trusted:
      - |
        -----BEGIN CERTIFICATE-----
        # PASTE THE FULL PEM ENCODED CERTIFICATE CONTENT FROM vra-root-ca.pem HERE
        MI... (Example Certificate Content)
        -----END CERTIFICATE-----
    
    # Ensure existing bootcmd and other cloud-init configurations follow this section.
    bootcmd:
      # ... existing commands