Intermittent error "InvalidIdentityValues" in BOSH Azure CPI when creating TKGI clusters

Products

• Operations Manager

Issue/Introduction

Intermittent error "InvalidIdentityValues" in BOSH Azure CPI when creating TKGI clusters: 

Task 99 | 05:45:46 | Error: Unknown CPI error 'Bosh::AzureCloud::AzureError' with message 'http_put - http code: 400
x-ms-client-request-id:
x-ms-request-id: XXX
x-ms-correlation-request-id: YYY
x-ms-routing-request-id: AAA:BBB
Error message: {"error":{"code":"InvalidIdentityValues","message":"Invalid value for the identities '/subscriptions/aaa/resourceGroups/bbb/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ccc'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."}}' in 'set_vm_metadata' CPI method (CPI request ID: 'cpi-ddd')

Resolution

When updating a VM using the virtualMachines API (in a verison > '2021-04-01'). Azure assumes that `userAssignedIdentities` will never contain a identity url key that points to anything other than an empty hash (since the parameters in this hash are considered "read only").
However, R&D think sometimes it works. This can be very confusing and cause issues during deploys.
Version 45.0.0 of the Azure CPI makes changes such that these values are cleared out before an update request is made. Customer can upgrade to Ops Manager 2.10.59+ which has the fix.