Reconciliation of migrated VMs in VCFA fails with `Retry exhausted (EBS-Connect endpoint/node availability)

Products

• VCF Automation

Issue/Introduction

On environments that use EBS-Connect (Event Broker Service integration in provisioning-service), reconciliation of migrated VMs fails, with errors of the form:

* `Retry exhausted ...` during reconciliation of migrated VMs.
* EBS-Connect logs (`[ebs-connect]`) showing endpoints being dropped / marked unavailable during periods of load.

Environment

VCF Automation 9.x

Cause

The EBS-Connect integration in provisioning-service tracks a set of vCenter endpoints. Per endpoint, it tolerates only a small number of consecutive upstream-service "unavailable" results before dropping the endpoint — controlled by `ebs.connect.upstream.service.unavailability.threshold` (default `2`). It also runs a periodic node-availability check (defaults: initial delay `PT1M`, interval `PT3M`). During large migrations, transient upstream unavailability crosses the low threshold, endpoints are dropped, and the reconciliation retries exhaust.

Resolution

Broadcom engineering is currently working on the permanent fix. Use the workaround below to relax the availability checks so the transient unavailability no longer exhausts reconciliation retries.

Workaround:

Apply an `ebs-connect-availability` profile that raises the per-endpoint unavailability threshold and pushes out the node-availability check cadence. The profile is shipped as a self-contained script that handles both application and removal.

> **Note:** Application or removal of the profile causes a redeploy of VCFA services (typically ~15 minutes). The script asks for confirmation before triggering the operation.

> **Tuning:** The defaults (`--unavailability-threshold 250`, `--availability-check-period P180D`) are the recommended values and are applied when you run `apply` with no flags. Adjust them only if guided to by engineering.

Procedure:

1. Take a backup of VCFA via Fleet LCM.

2. Log in to one of the VMs of VCFA and become root:

ssh vmware-system-user@<VCFA-Platform-FQDN>
sudo -i

3. Install the patch scripts on the node:

mkdir -p ~/ebs-connect-availability-profile && cd ~/ebs-connect-availability-profile

base64 -d <<< '/Td6WFoAAAFpIt42BMC6JICgASEBHAAAAAAAAOgChbLgT/8SMl0AMpiKyQKBgHzJpAcEPmnKUk78yrxtIb7Dd+fI7ML+aAGxFn+aPHPxyXcOhAAqYzMX3Poq5CoUnB9326ZtB7uP7o9Bs3LyqfysWXej4gcPkMD2sTJA7ZW5+ggi38ufgZFevtI0d95A0hXerTrMpyGbWCMnnhq2deYJWLJKChzxFElUMViyEhNJGuMM+2kyahK+p6HcSTMAXbhoVBN7iJo8TNXRVbJakrPWy/5JVkAY1f9zsRXg3QOJL4+Obw0afZdaCLEhw7V8SmRtavJiq43yZICVHW8MNZuGPtg7v323U+FSPYUndJhoO/E7PS1M6CMpCRiwy03RvG0fSZhCnRYcC1wEClwocmFzcrI2zYqUYvaURxx2wnDQYQfADUZ9uq/kOR0S6AJiXOqXmJnh6D1ek+iNTWG07ijdTFAujz9vbJgGS9yOc5512E6FK5TrAbJ77b/Ubx2q6U9Qxahca00INhuj0K/EGUmLueZ3CJWJ2BTwfhT2iV5hT+QdlK6EZqLpWayGU6lJRMFGd8TB5PDG+W0yL0SzBXS7our9LdnhvrZJkOe8BAkVBUex89JY/bPe8ks8AEi/Evd4eLalFlHo2vuSw+pYVGgUSz48675OvM0hZo3t6SzNR+7+Ll+7v7uABwtFylsj55rhS0zuHuNPtbGdjB54Mtw5HsOQCfIWZFFKsxj06T/L6xhKPn12YpannThghsy83qnM32DUWMVa3ktrCHAiqgZaIMJIpbYP2oxBss4Y9vmqX1XdhbQy9MkFNyofbJh4GeDtWz8o1U9vyjrKaQVapz8+q86QaHihFp0vE06lu++QfrgiyiqlZlmdi88zZzDZa6Q8ajNnYK+38BJpxVhrpBZZc7Hxa+l+3W9P4t+K/FlgJZewPYfBDOBpjYyXHWW97gZyQ11my7egtFJI5yLrkYkcAMoFj+Eh/b2fBOQ7b59qz1D86fKpN9yO3SWBgU9Ln5t4uWWyBxQ9qWzLvMhjxo54I3bJRSVgbGcNS80JvNwTUVD6PNq0alYv6sHnxYEi2Mtg/uqA3sbs5vPsFl6TP7GlRh+qitYo/dV8T78gZeVmdWOd1F0LsYAooC64VMHxpSaMsuyjNkWx2cUvTZ6I6KbHm50KJenuogU1UHUy9sEsM93JRsOME1nhYGXJpwFhISOIpHmiZo5fuhZHPCDFnUYQO9GFIDBYqixFzRvfCJ1DRimbAygH+tQVlqYn4oJRpXdI4/a9aFkDJgjzCFrM+iXl2HgE4gu0cL3CQXYmsl+IY9u58JBCpfXgdxRCGkihnSPkAuZYxg/Wlkn+TFEOLYYMfKt4kSRJ9uPh7kIAuhHIRHekE342L6c9NNgCA0Kmn6MOulYcZ9vB2K4GP/Rn6zfjdtHjuFT/zytq/8ZY2ZEbgGegxyqWyRhYdTwArtErv0jbUV7IEB4pV+rjmbgokE3bla1sKHs8Xi+DUkeE8h4kaYgWdv/zsabmr8wcZw44nkr22WcmETa56C88cNYJncGs/iON7wrKUuFQcTs34ZmIolY1uu/JKpIHTWR4G8gy1EkRKmcHBrSLLFGcI04Ewx0VRqYD8m9Pmu8n/6sK177r5O0PzpVEpKC8LTUAIB+qG+mB62VacWRxq/VJv4SLOj4BaGle/TA6v94NbyTRXgH+a4ipAXnJRpb6Wi4i4ZTp3LdAU4Wb6VCej9T2jFXPPrtEYdK/wGm0i2Zrp74sZTZLUmitcyU6pwfS41MgwRUJyGzTEDOvnY5k5cF/LLwd5gG3etuwus7sE0yaHKinJ1AS4/l6683ofxDr0yCA4T43ZNJx7Xq6Jyv7ZOcbYz3O3NWbzyVFpkxoeA/0wRLZtDdYs5F5joOeCxNZ1vAURh13g0/9u5xLsAtNTyvtTnD1+tXK6/k+0YTdG478iq86yfylaxQi8QKhjY9AJWLc4USYJyuzT5UFewXWQfOArYIP3lcHmSr911Wjrh2tg/37PjmCyxIx6xjKc2vI0o2Ay4I73AYdIInjq/1OxQIktSGsPWE9Mmedw0cZTxLhtRBp1Fmj1XpAjL625NptYMkRwbEmAcR5E6xblW0TlNz0M9Ix/RGTvBAt4fH8jAzGFH0k2xZqw4jc2KQnfV5YSAktGkC6E1hIK34sh6yEvM6digUXjUcgEbFgi8atqS7GIgSpIvEYDErAlFQP9QJYZr5b+vPGrbGWHz/4qnB4TTJk4I96bRKLLSn33UCBaei68dcIyceUG/ZYzfDXCd0oFbVHpPzfoe0hWPCVQCPZYQBKKn9lmo6yypQFvcbFaQ+6tWlijv0bM7T/tR3Qh5EKLyTloEQz7imgcGqOF2eWggGT/W/DKvzqdACFZhK0IgLgYAs96M7eDC18H4NyAifXSDzfVt0b7m1gRMrjA01xpu2H1qhkdcBFbBFtwngrOrXyBwSZFY/32L3cIWH70/K7MEKMKCZEoTse5bYknp9NZj0t2ThTB++j/WzAhz5PDRXkRkE4JzeXhIr7/m3DvOWzknjR496oH7l7PS6hPkMLyeLm1qYuW/LzxpVeKbNbMl/xPXoQNLsf+/NvAafIk49V8NgID1BS+RTs/4AGZvWhzgFKLQIBYr2srpU8UYMxwUxDiDAb0qv27FunpjOL/Lv2InfRojTjbcZ10f3UxcMe3ar8YQsYHQH1OlEeja6SUw9Xv+4rV9IQ17embIzsUbPRtFGIDPm3dL1t95rdq2t8TW58rFdb1zT5k2P/EylBtKT4UZhjIBG6KKLkzdinv4fAyF7gC3o2atUeWeyxm4pEFOdKG/yaQCXn+b8VE+PSBzWfki+Igiq1f3yddbV9FhYSvgU+6hii467QtxPr0hwabMU80qMImcg2K3DcPQ8GxS8+VCxB4LlRoY3ZvcGJPkFtC/Jk6BfMFl3w0ahqkCNZNUCD1B2qUBnYUqMprF1mOucd0zNEFNoyJ6PKcKbwnEXxfUgBYuIMxhHyAiCmWwvXfAHtzfdy30gQ/ONN1cv+ASmSIBjQMrHdGSYWp2OpxBIFPeh+3qD/0c6+3bj5kXqt3VLBA+8heA3BC+0fnQM1cYG7gOWMUATEITrHKv31eszb+cDZFRiQBs+mGGbsRG2rbtb1CGMJt83Tl5mfI4TM1NaxMhhg4+WHE39gcPrReu+SiIXxEe8Hm7hb24RWL9fbDP1OBZSnLIwpAsJ6BWya1Agv/uoZVU4O3eMts3fc1Pirnjy/EyN5xC3sw7SnnFO+sYMSXPk/JgPrNdTOO2vW5yD/xz+ZPkx6gRjmEEkafFR4lnr7+b7JTXYL4aOiyuFtplKm14M5aY6ZdZ631xtJ/BE5n1+PQxI3MExJrZmhiRDhA6vEv/ndD2Hlmj8+kHSb0SuTUb7rPkZE6+iMbuC134wpFxdqzJ3fC2wMcs2oIrrUWShTMsOWLB8cBNP7nK8u0J16AWlF6yj3J6BQXl83B4YZeoGGb7es/V2ljym/uOPA1EUimUuo06B8mPcno5IaERDqxvLO5oPRDZMrJ755Zku+rvRbY2RHHfn2PbMuReKPpTHq2hB9269PW5HQ0lbUqZm0zlm+Klcws8+5Z/Fw1NgmQHOoaxkz9A/fbFWW+TEvKdAkNK0CZskEIp0AWSNzP7OBwXIxsQDaGDJHApta97Fa9MjikwdlWVyux3y5eTDYMnl7Dx0H1o0YO0q/eOKackXTKoWJDduIuvS2ZpijkExFCCbN4LcN0E9yR/i1sN9ppTipEyxMSUF8kRMf59Ag9H4htgeT6nnWbj7RmM6FNItzUg/WVnsI89z0hPdUPFc9jyFjTfWdGdXkgg4FDN9JBKoF8h4s6xxRoRQbzyLw56zx2fBrP60RyMH6zYLCJPMZOArIv86eE2HjfeUM6ltUDnNTKOpIXZkw3xPza3EhI14OcFm02dvGTr1fu5il8tbRAlm370NoPSqcGypeP0mHypycUFZoHRqQJ1UrDv5VMCuRZvV9TpysO6ndvI5S/rW3EgnGf7Zh34xzVZIFEwKrIgQrakFtkCD6fucXixR7YZBZ+DJ+dx/iB4POnMrXWmGM649fSVN50MO7trjoNLWR+nmk/pgiXvoQnZV6wVcLy9yzrC6nHY6DuYWzgeVJ6EYenLkYJIJDMwvzRSjuF325kK+9cbioO/lD9GTuDtxgjR420PbeM7HwiL1IeDomdRaBM8Mp11EK+gl8pGlMx3dgFxGXpIsiIDAj5AyU+Yp9GORcB8xdSohHRiBH/Av7QSyom4/zIxuzdfkzGmIuizb6M9NKWo61iXHaZfx6JIQTJ76YV+kY15+647SP2FUVNfGkdg1RwqsRAj7qUWhWPc7BFMi0VePQuaxInbpFjK4rzXflVC6nR/MociKWt5AvFUQo6DCG9aum+90mZwPYm1WpxO4MFHpxmbmUe1kXJBN0POBwr083Utk30R7s6+32Cgk7mYL8B6rIlLXjtS1VxZnTUUfZmR5jZGpjw4XU29WRQT6cNGNDPy1SA6zCA4n07v4K2HkTsVE16GDLnNcNUz7jMRRTHKawd4xwkEpHyUd/cZzQCG9CFlKQpsuY/qfsIHT5vBlcqIGAuyU7vxhQvMvJ29x83emZrcppzNEGkPPu9iydWQUA4l7OnM98xdIydpoV8GaejwDXPWXkPo8E67gaaBP7tJZryEmszJAKLDA9HWRTdz01Fs3YXbU8CGkVn8LW9pLyAMroH4Nmk9JqeOwCsUlp+y5BDMXpi4coIj4HfEyNQ2YqiFkLFDmVhAcjrLvCJsrWfdSzNInOdIx40RrX6xZMsS9uLrXI564E6FzB4nmjNp6PQcJJT+osly0LWFmaGj7iHxQ3sbmuySgP/tWHcYXfchTVrXJnIgeFf/8Glt8vZLdUixGY71DE7l4xGPhndKciV4e7xRgOat68oKrAZtGxpVp/ob6JUsRMrNX8QUBqlFYGK3pm4ZZlIFUqdjvINyPeLauP0oxWJPr3PP1pyRoSn6VNXZcE6mSFptGZEeGtbM3KfHZoUmd2ZcIADTY8Mp7em8omRY1rBATvwLp2iSpr7NQ9R/BkAFDWRGphDumiPzLrHqmZd7TFIJGVB1rr54gH8rCZEyWkMKAEq0eSQYY7fNNhb4tHjdXPKnRoyo4hJU+p0ZqfrtiCnjQiGO4SbTpLc2LcH9JR4k91YD792kSy+8jSsL8+SNS8L6/tEz7TI7yTLhki+9klMoZMKMgt6vv+X4RlBP0gA1aP5g/DZa6s4EQPg+qGcInT9u4+sV4bKkZcGmKN5xyEftQkqWnAm49/41+fGvQOCPH5FU0hb7ivTcguDXKVONce/i/fOBbwPKDPsaaiiNlX1Rn1IoXWojZFV0uDoMXDSu01lCPUClNQ+BtlW0GD/zfn2huWaLk//Uru1adei4P6AArRhQ4HqZp9WdPadmZwYZag5+UA2ErFsw7ydRW/sbmG75PBGuEwC5CBMOW91oUWodPduu3Xp6OLd2EUaBgbQNDnPBpavZ2Dd2LqN/EE1gTmK8VD2Db36x50Ly+92uGrc4EdBjBqm7TqlRzTTxDr4TcbPdlqOeDjYUGMzBwHXfZg+KkxiYqjWDlulteoLTnjO/cT+GpMWmeNy9TRVzDTULfDSA2QepT2VPJGuHP9Uw9fCYSglzeQutyzP4tbGaHR3gY0wVs5bO5EcpraTXuqoxxDqDkfqahtV+QLC63EAr7NcZC1UQILAHS0EuesBUWDZBd2d9Btur8k2N5HIDk11aSBKo5bXeSpWt3DcxRWspDoCqRu2SEfMBumaP7sdtfD8Al6AcK0cXVd3W2qcroAmyQYsnGVuDVuEkaWg2GkOsw31oHsGb4aELXgE2gvorcWiuTWGNBCflYpLIzt8glalZm3yisfOByuvXdjHRig+OFTsXFIELqjONP7UpcdtmqwTUzuCCP7nVTq+e5NXAXtcYK+3pz5r9KbA55aSeGHmtviQTWzXNjXoTgQi+GyGOEuYU/tKdM3X+MTl7O34faYhvAMHFAURaga0EZuonaIT91d4W6Q7RTHaD1DY8fHKXegOmw57m1KsORgyMPzr4NOjZ47t2tfY3dBNl5cjBK+gIhF0jNICfZqijM6f+nvq80zKt2NwFejkMiczPiv5VFlgOduyNhVhuUXmsWIkDxch6VT5gWgeFdQbcyIJ53T2Hgy5Gg4nVcov/fiXQLqB8zCY86EnPTa7Kk2xKTjmg3eHZX1csihr5DSf3qB6BE69U1N6FStsvWI8AAAAAAVPIbFAAHSJICgAQAnOtiFPjANiwIAAAAAAVla' | xz -d | tar -x

4. Apply the override. Running with no flags applies the recommended values (`--unavailability-threshold 250`, `--availability-check-period P180D`):

./ebs-connect-availability.sh apply

The script:
* verifies that `kubectl` works against the appliance cluster,
* checks whether `provisioning-service-app` is already properly configured,
* asks for confirmation before redeploying services,
* uploads the `ebs-connect-availability` profile, triggers reconciliation,
* waits for reconciliation and health,
* verifies the new settings are in effect on `provisioning-service-app`.

To revert the above changes:

1. Follow steps 1-3 from the procedure above (if the script is not already available on your node).
2. **Remove** the override (revert to service defaults):

./ebs-connect-availability.sh remove

The script verifies the profile is present, asks for confirmation, deletes the profile, triggers reconciliation, monitors it, and waits for health to return.

Additional Information

Logs:

Each run creates a working directory in the current working directory named `ebs-connect-availability-profile-<timestamp>` containing the run log (`patch.log`) and, on apply, the staged profile bundle. Attach this directory to the SR if support is needed.