Error: "Internal error occurred. Failed to attach disk" [VCFMS-VM-012] seen during "Perform Component Upgrade" stage of upgrade to VCFA 9.1

Products

• VCF Automation

Issue/Introduction

• VCF Automation upgrade to 9.1 fails at "Perform Component Upgrade" stage.
  
  • Step vmsp_upgrade_vcfa failed.
• The following particular error is seen:
  • Internal error occurred. Failed to attach disk [<Disk_Name>] <Disk_Path>.vmdk to node <VCFA_Node>. Please retry the operation or contact support if the issue persists. [VCFMS-VM-012]
• The logs for the upgrade step show these errors. Command:
  • kubectl logs vcfa-upgrade-80-entrypoint-<tag>-execute-script-### -n prelude
• Errors:
  • ERROR : <long_tag> Script exited unexpectedly at line 134 with exit code 1
  • ERROR : <long_tag> get message_count returned empty, defaulting to 0
  • ERROR : <long_tag> Failed to attach disk "[<Disk_Name>] <Disk_Path>.vmdk" to "<VCFA_Node>".

Environment

VCF Automation 8.x or 9.x, upgrading to 9.1

Cause

A disk mount is timing out

Resolution

Once the Create Automation Cluster task is complete, ssh into one of the VCFA control plane nodes and perform the following actions:

1. Run the following commands:
   export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get releasetemplates -A
   
   
2. Make these two edits to override a timeout on the disk mount:
    
   1.  kubectl edit releasetemplates argo-templates-v9.1.1865 -n vmsp-platform
    
   remove below lines: :-
       driftDetection:
      mode: <% ternary "enabled" "warn" (eq ( .Values.flags.driftDetection | default
        "true") "true") %><%/* type=boolean; description=specify if drift detection
        is enabled; default=true */%>
    
   
   2.  kubectl edit clusterworkflowtemplate vcenter-virtualmachine
    
   Below this section:
   log "Mount Disk INFO [${TRACKING_ID}]: Mounting disk..."
get_action_status mount-disk
export
    
   Add this sleep statement after exports:
    
   sleep 300