Note: If you have applied the workaround from this KB article prior to December 17, 2021, you will need to re-run through to ensure you have the latest fixes as outlined by Apache.
Note: KB is only applicable for versions up to 7.6 HF 24.
Note: The following procedure requires downtime of the system due to service restarts.
SSH into each vRA appliance node and run the following steps:
service vco-configurator stop
base64 -d <<< "bG9nX21lc3NhZ2VfbG9nNGooKSB7CiAgZWNobyAiWyQoZGF0ZSAtLXV0YyAiKyVGVCVULiUzTloiKV0gJDEiIHwgdGVlIC1hIC92YXIvbG9nL3Ztd2FyZS92Y28vYXBwLXNlcnZlci92Y29fbG9nNGpfY3ZlLmxvZwp9Cgpsb2dfZXJyb3JfbG9nNGooKSB7CiAgbG9nX21lc3NhZ2VfbG9nNGogIkVSUk9SOiAkMSIKICBleGl0IDEKfQoKc2V0X2phdmFfb3B0KCkgewogIGxvY2FsIGZpbGU9IiQxIgogIGxvY2FsIGJha3VwX3N1ZmZpeD0iJChkYXRlIC0tdXRjICsiJVklbSVkJUglTSIpIgogIAoKICBpZiBncmVwIC1xICdEbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlJyAkZmlsZQogIHRoZW4gCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiVGhlIGphdmEgcHJvcGVydHkgbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIGlzIGFscmVhZHkgc2V0IGluICRmaWxlLiIKICBlbHNlCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiQ3JlYXRpbmcgYmFjayB1cCBmb3Igc2V0ZW52IGZpbGUgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCIKICAgIGNwIC1mICIkZmlsZSIgIiRmaWxlLiRiYWt1cF9zdWZmaXgiCiAgICBsb2dfbWVzc2FnZV9sb2c0aiAiQWRkaW5nIC1EbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIHRvIEpWTV9PUFRTIGluICRmaWxlIgogICAgcmVzPSQoYXdrICdGTlI9PU5SeyBpZiAoL15KVk1fT1BUUz0vKSBwPU5SOyBuZXh0fSAxOyBGTlI9PXB7IHByaW50ICJKVk1fT1BUUz1cIiRKVk1fT1BUUyAtRGxvZzRqMi5mb3JtYXRNc2dOb0xvb2t1cHM9dHJ1ZVwiIiB9JyAkZmlsZSAkZmlsZSkgfHwgbG9nX2Vycm9yX2xvZzRqICJGYWlsZWQgdG8gZWRpdCAkZmlsZSIKICAgIGVjaG8gIiRyZXMiID4gJGZpbGUKICBmaQp9Cgp1cGRhdGVfdnJvX3RvbWNhdF9zdGFydCgpIHsKICBsb2NhbCBmaWxlPSIkMSIKCiAgaWYgZ3JlcCAtcSAncGF0Y2hfYWxsX3BsdWdpbnNfdjIgPicgJGZpbGUKICB0aGVuIAogICAgbG9nX21lc3NhZ2VfbG9nNGogIk5vdGhpbmcgdG8gZG8uIEtCIGFscmVhZHkgYXBwbGllZCEiCiAgZWxzZQogICAgbG9jYWwgcmVzCiAgICBsb2NhbCBiYWt1cF9zdWZmaXg9IiQoZGF0ZSAtLXV0YyArIiVZJW0lZCVIJU0iKSIKICAgIGxvZ19tZXNzYWdlX2xvZzRqICJDcmVhdGluZyBiYWNrIHVwIGZvciB0b21jYXQgc3RhcnR1cCBjb25maWcgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCIKICAgIGNwIC1mICIkZmlsZSIgIiRmaWxlLiRiYWt1cF9zdWZmaXgiCgogICAgbG9nX21lc3NhZ2VfbG9nNGogIk1vZGlmeWluZyB2Uk8gdG9tY2F0IHN0YXJ0dXAgY29uZmlnIC0gJGZpbGUiCgogICAgaWYgZ3JlcCAtcSAnI2xvZzRqX2N2ZV93b3JrYXJvdW5kJyAkZmlsZQogICAgdGhlbgogICAgICBzZWQgLWkgJ3MvI2xvZzRqX2N2ZV93b3JrYXJvdW5kL1xuI2xvZzRqX2N2ZV93b3JrYXJvdW5kL2cnICRmaWxlCiAgICAgIHNlZCAtaSAnL2xvZ19tZXNzYWdlX2xvZzRqMl9jdmUgIlBhdGNoaW5nIGRvbmUuIiQve247cy8uKi99XG4jbG9nNGpfY3ZlX3dvcmthcm91bmRfZW5kL30nICRmaWxlCiAgICAgIHNlZCAtaSAnL3ZSTyBzZXJ2ZXIgc2VydmljZSBkaWQgbm90IHN0YXJ0IHdpdGhpbiB0aGUgZXhwZWN0ZWQgcGVyaW9kLiokL3tuO247bjtzLy4qL31cbiNsb2c0al9jdmVfd29ya2Fyb3VuZF9lbmQvfScgJGZpbGUKICAgICAgc2VkIC1pICcvI2xvZzRqX2N2ZV93b3JrYXJvdW5kLywvI2xvZzRqX2N2ZV93b3JrYXJvdW5kX2VuZC9jXGRlbGV0ZV9tYXJrZXJcJyAkZmlsZQogICAgICBwZXJsIC1pIC0wcGUgJ3MvKC4qKVxuLipkZWxldGVfbWFya2VyXG4vXDEvZzsnICRmaWxlCgogICAgICBzZWQgLWkgJy9sb2c0al9jdmVfd29ya2Fyb3VuZCA+L2QnICRmaWxlIAogICAgICBzZWQgLWkgJy9wYXRjaF9hbGxfcGx1Z2lucyA+L2QnICRmaWxlCiAgICBmaQogICAgc2VkIC1pICcvYmFzZTY0IC1kIDw8PC9kJyAkZmlsZSAKCiAgICBpZiAhIGdyZXAgLXFFICdhY3Rpb249InN0YXJ0InxTdGFydGluZyB0Y1NlcnZlcicgIiRmaWxlIgogICAgdGhlbgogICAgICBsb2dfZXJyb3JfbG9nNGogIlVuYWJsZSB0byBhcHBseSBwYXRjaDogVW5leHBlY3RlZCBmb3JtYXQgb2YgdlJPIHRvbWNhdCBzdGFydHVwIGNvbmZpZyAtICRmaWxlLiIKICAgICAgZXhpdCAxCiAgICBmaQoKICAgIGxvY2FsIHV0aWw9IiQoZGlybmFtZSAiJGZpbGUiKS9jdmVfdXRpbC5zaCIKICAgIGJhc2U2NCAtZCA8PDwgIkkyeHZaelJxWDJOMlpWOTNiM0pyWVhKdmRXNWtDbXh2WjE5dFpYTnpZV2RsWDJ4dlp6UnFNbDlqZG1WZmRqSW9LU0I3Q2dsbFkyaHZJQ0piSkNoa1lYUmxJQzB0ZFhSaklDSXJKVVpVSlZRdUpUTk9XaUlwWFNBa01TSUtmUW9LQ25CaGRHTm9YM0JzZFdkcGJsOTJNaWdwSUhzS0NXeHZZMkZzSUhCc2RXZHBibDl3WVhSb1BTSWtNU0lLQ1d4dlkyRnNJSEJzZFdkcGJsOXVZVzFsUFNJa0tHSmhjMlZ1WVcxbElDSWtjR3gxWjJsdVgzQmhkR2dpS1NJS0NXeHZZMkZzSUhSbGJYQmZjR0YwYUQwaUwzUnRjQzhrY0d4MVoybHVYMjVoYldVaUNnbHNiMk5oYkNCaVlXTnJkWEJ6WDNCaGRHZzlJaTkxYzNJdmJHbGlMM1pqYnk5aVlXTnJkWEJ6SWdvS0NXMXJaR2x5SUMxd0lDSWtZbUZqYTNWd2MxOXdZWFJvSWdvS0NYSnRJQzF5WmlBaUpIUmxiWEJmY0dGMGFDSUtDWFZ1ZW1sd0lDMXhJQ0lrY0d4MVoybHVYM0JoZEdnaUlDMWtJQ0lrZEdWdGNGOXdZWFJvSWlCOGZDQmxlR2wwSURFS0Nna2pJRU5vWldOcklHbG1JSFJvWlhKbElHbHpJR0VnYm1WbFpDQjBieUIxY0dSaGRHVWdkR2hsSUhCc2RXZHBiZ29KYVdZZ1ptbHVaQ0FpSkhSbGJYQmZjR0YwYUNJZ0xYaGtaWFlnTFhSNWNHVWdaaUF0Ym1GdFpTQW5iRzluTkdvdFkyOXlaUzB5S21waGNpY2dMV1Y0WldNZ0wzVnpjaTlpYVc0dmVtbHdJQzF6WmlBaWUzMGlJRnc3SUh3Z1ozSmxjQ0J2Y21jdllYQmhZMmhsTDJ4dloyZHBibWN2Ykc5bk5Hb3ZZMjl5WlM5c2IyOXJkWEF2U201a2FVeHZiMnQxY0M1amJHRnpjenNLQ1hSb1pXNEtDUWxzYjJkZmJXVnpjMkZuWlY5c2IyYzBhakpmWTNabFgzWXlJQ0pOYjJScFpubHBibWNnY0d4MVoybHVPaUFrY0d4MVoybHVYMjVoYldVaUNna0piWFlnSWlSd2JIVm5hVzVmY0dGMGFDSWdJaVJpWVdOcmRYQnpYM0JoZEdnaUNna0pabWx1WkNBaUpIUmxiWEJmY0dGMGFDSWdMWGhrWlhZZ0xYUjVjR1VnWmlBdGJtRnRaU0FuYkc5bk5Hb3RZMjl5WlMweUttcGhjaWNnTFdWNFpXTWdjMmdnTFdNZ0p5OTFjM0l2WW1sdUwzcHBjQ0F0Y1NBdFpDQWllMzBpSUc5eVp5OWhjR0ZqYUdVdmJHOW5aMmx1Wnk5c2IyYzBhaTlqYjNKbEwyeHZiMnQxY0M5S2JtUnBURzl2YTNWd0xtTnNZWE56T3lCMGIzVmphQ0F0ZENBeU1ESXhNREV3TVRBd01EQWdJbnQ5SWljZ1hEc0tDUWtvWTJRZ0lpUjBaVzF3WDNCaGRHZ2lJRHNnZW1sd0lDMXhJQzF5SUMxWUlDMUVJQ0lrY0d4MVoybHVYM0JoZEdnaUlDb3BJSHg4SUdWNGFYUWdNUW9LQ1FraklFWnBlQ0J5YVdkb2RITUtDUWxqYUc5M2JpQjJZMjg2ZG1OdklDSWtjR3gxWjJsdVgzQmhkR2dpQ2drSlkyaHRiMlFnTURZME5DQWlKSEJzZFdkcGJsOXdZWFJvSWdvSkNXeHZaMTl0WlhOellXZGxYMnh2WnpScU1sOWpkbVZmZGpJZ0lsTjFZMk5sYzNObWRXeHNlU0J3WVhSamFHVmtJSEJzZFdkcGJqb2dKSEJzZFdkcGJsOXVZVzFsSWdvSlpXeHpaUW9KQ1d4dloxOXRaWE56WVdkbFgyeHZaelJxTWw5amRtVmZkaklnSWs1dmRHaHBibWNnZEc4Z1pHOGdabTl5SUhCc2RXZHBiam9nSkhCc2RXZHBibDl1WVcxbElnb0pabWtLQ1hKdElDMXlaaUFpSkhSbGJYQmZjR0YwYUNJS2ZRb0tjR0YwWTJoZllXeHNYM0JzZFdkcGJuTmZkaklvS1NCN0NnbG1iM0lnWm1sc1pTQnBiaUF2ZFhOeUwyeHBZaTkyWTI4dllYQndMWE5sY25abGNpOXdiSFZuYVc1ekx5b3VaR0Z5Q2dsa2J3b0pDWEJoZEdOb1gzQnNkV2RwYmw5Mk1pQWlKR1pwYkdVaUlIeDhJR3h2WjE5dFpYTnpZV2RsWDJ4dlp6UnFNbDlqZG1WZmRqSWdJa1ZTVWs5U09pQkdZV2xzWldRZ2RHOGdjR0YwWTJnZ2NHeDFaMmx1T2lBa0tHSmhjMlZ1WVcxbElDUm1hV3hsS1NJS0NXUnZibVVLQ2dsc2IyZGZiV1Z6YzJGblpWOXNiMmMwYWpKZlkzWmxYM1l5SUNKUVlYUmphR2x1WnlCa2IyNWxMaUlLZlFvPSIgPiAiJHV0aWwiCgogICAgc2VkIC1pICdzLGV2YWwgZXhlYywnInNvdXJjZSAkdXRpbCInXG4mLCcgIiRmaWxlIgogICAgZ3JlcCAtcSAic291cmNlICR1dGlsIiAiJGZpbGUiIHx8IHsgbG9nX2Vycm9yX2xvZzRqICJGYWlsZWQgdG8gZWRpdCAkZmlsZS4gQmFja3VwIGNhbiBiZSBmb3VuZCBpbiAkZmlsZS4kYmFrdXBfc3VmZml4IjsgZXhpdCAxOyB9CgogICAgc2VkIC1yIC1pICcvYWN0aW9uPSJzdGFydCJ8U3RhcnRpbmcgdGNTZXJ2ZXIvYSBcXHRcdFx0cGF0Y2hfYWxsX3BsdWdpbnNfdjIgPj4gL3Zhci9sb2cvdm13YXJlL3Zjby9hcHAtc2VydmVyL3Zjb19sb2c0al9jdmUubG9nJyAkZmlsZQogICAgZ3JlcCAtcSAncGF0Y2hfYWxsX3BsdWdpbnNfdjIgPicgIiRmaWxlIiB8fCB7IGxvZ19lcnJvcl9sb2c0aiAiRmFpbGVkIHRvIGVkaXQgJGZpbGUuIEJhY2t1cCBjYW4gYmUgZm91bmQgaW4gJGZpbGUuJGJha3VwX3N1ZmZpeCI7IGV4aXQgMTsgfQoKICAgIGxvZ19tZXNzYWdlX2xvZzRqICJTdWNjZXNzZnVsbHkgbW9kaWZpZWQgdGhlIHZSTyB0b21jYXQgc3RhcnR1cCBjb25maWcgLSAkZmlsZSIKICBmaQp9CgoKKHNldF9qYXZhX29wdCAiL3Vzci9saWIvdmNvL2NvbmZpZ3VyYXRpb24vYmluL3NldGVudi5zaCIgJiYgc2V0X2phdmFfb3B0ICIvdXNyL2xpYi92Y28vYXBwLXNlcnZlci9iaW4vc2V0ZW52LnNoIiAmJiB1cGRhdGVfdnJvX3RvbWNhdF9zdGFydCAiL3Zhci9saWIvdmNvL2FwcC1zZXJ2ZXIvYmluL2luaXQuZC5zaCIpIHx8IGxvZ19lcnJvcl9sb2c0aiAiRmFpbGVkIHRvIGFwcGx5IHRoZSBsb2c0aiBDVkUgd29ya2Fyb3VuZCBmb3IgdlJPLiBGb3IgbW9yZSBkZXRhaWxzIHNlZSAvdmFyL2xvZy92bXdhcmUvdmNvL2FwcC1zZXJ2ZXIvdmNvX2xvZzRqX2N2ZS5sb2cuIg==" | sh -
/usr/lib/vco/tools/configuration-cli/bin/vro-configure-inner.sh controlcenter-update
base64 -d <<< "bG9nX21lc3NhZ2UoKSB7CiAgZWNobyAiWyQoZGF0ZSAtLXV0YyAiKyVGVCVULiUzTloiKV0gJDEiIHwgdGVlIC1hICAgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nCn0KCmxvZ19lcnJvcigpIHsKICBsb2dfbWVzc2FnZSAiRVJST1I6ICQxIgogIGV4aXQgMQp9CgpzZXRfamF2YV9vcHQoKSB7CiAgbG9jYWwgZmlsZT0iJDEiCgogIGlmIGdyZXAgLXEgJ0Rsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUnICRmaWxlCiAgdGhlbiAKICAgIGxvZ19tZXNzYWdlICJUaGUgamF2YSBwcm9wZXJ0eSBsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUgaXMgYWxyZWFkeSBzZXQgaW4gJGZpbGUuIgogIGVsc2UKICAgIGxvZ19tZXNzYWdlICJBZGRpbmcgLURsb2c0ajIuZm9ybWF0TXNnTm9Mb29rdXBzPXRydWUgdG8gVkNBQ19PUFRTIGluICRmaWxlIgogICAgZWNobyAnVkNBQ19PUFRTPSIkVkNBQ19PUFRTIC1EbG9nNGoyLmZvcm1hdE1zZ05vTG9va3Vwcz10cnVlIicgPj4gJGZpbGUgfHwgbG9nX2Vycm9yICJGYWlsZWQgdG8gZWRpdCAkZmlsZSIKICBmaQp9CgpkZWxldGVfam5kaV9jbGFzcygpIHsKICBsb2dfbWVzc2FnZSAiRGVsZXRpbmcgYWxsIEpuZGlMb29rdXAuY2xhc3MgZmlsZXMgZm91bmQgZm9yIGxvZzRqIDIueCB2ZXJzaW9ucyIKICBmaW5kIC8gLXhkZXYgLXR5cGUgZiAtbmFtZSAnbG9nNGotY29yZS0yKmphcicgLWV4ZWMgL3Vzci9iaW4vemlwIC1xIC1kICJ7fSIgb3JnL2FwYWNoZS9sb2dnaW5nL2xvZzRqL2NvcmUvbG9va3VwL0puZGlMb29rdXAuY2xhc3MgXDsgfCB0ZWUgLWEgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nCn0KCihzZXRfamF2YV9vcHQgIi9ldGMvdmNhYy9zZXRlbnYtdXNlciIgJiYgZGVsZXRlX2puZGlfY2xhc3MgKSB8fCBsb2dfZXJyb3IgIkZhaWxlZCB0byBhcHBseSB0aGUgbG9nNGogQ1ZFIHdvcmthcm91bmQgZm9yIHZSQS4gRm9yIG1vcmUgZGV0YWlscyBzZWUgL3Zhci9sb2cvdm13YXJlL3ZjYWMvdmNhY19sb2c0al9jdmUubG9nLiI=" | sh -
service horizon-workspace restart && base64 -d <<< "IyEvYmluL2Jhc2gKZWNobyAnV2FpdGluZyBmb3IgaG9yaXpvbiBzZXJ2aWNlIHRvIHN0YXJ0Li4uJwpmb3IgaSBpbiB7MS4uMTIwfQpkbwogICBzdGF0dXNfY29kZT1gY3VybCAtLW1heC10aW1lIDMwIC1vIC9kZXYvbnVsbCAtcyAtdyAiJXtodHRwX2NvZGV9XG4iICdodHRwOi8vbG9jYWxob3N0OjgwODAvU0FBUy9BUEkvMS4wL1JFU1Qvc3lzdGVtL2hlYWx0aCdgCiAgIFtbICIke3N0YXR1c19jb2RlfSIgPT0gIjIiKiBdXSAmJiBicmVhawogICBlY2hvICJIb3Jpem9uIGlzIHN0aWxsIHN0YXJ0aW5nLi4uIgogICBzbGVlcCA1CmRvbmUKCmlmIFtbICIke3N0YXR1c19jb2RlfSIgPT0gIjIiKiBdXTsKdGhlbgogICBlY2hvICdIb3Jpem9uIHNlcnZpY2Ugc3RhcnRlZCBzdWNjZXNzZnVsbHkhJwplbHNlCiAgIGVjaG8gJ0hvcml6b24gc2VydmljZSBkaWQgbm90IHN0YXJ0IHdpdGhpbiB0aGUgZXhwZWN0ZWQgcGVyaW9kLiBDaGVjayBzdGF0dXMgb2YgaG9yaXpvbi13b3Jrc3BhY2Ugc2VydmljZSBhbmQgbG9ncyBpbiAvdmFyL2xvZy92bXdhcmUvaG9yaXpvbi8gZm9yIG1vcmUgZGV0YWlscy4nCiAgIGV4aXQgMQpmaQo=" | sh -
service elasticsearch restart
service vco-server status | grep PID && service vco-server restart
service vco-configurator start
service vcac-server restart
To validate that the workaround has succeeded, take the following steps on all nodes:
ps aux | grep -i java | grep Dlog4j2.formatMsgNoLookups=true
find / -xdev -type f -name 'log4j-core-2*jar' -exec sh -c '/usr/bin/unzip -l "{}" | grep -q org/apache/logging/log4j/core/lookup/JndiLookup.class && echo Found in file: {}' \;
The scripting was successful if the output is empty or contains only log4j version 2.17 and above.
Note: If the above validation detects anything in the /var/lib/vco/migration-cli/ location - this directory can be safely removed, as it is a non-essential cli tool used only during vRO migration.
VMware Response to Apache Log4j Remote Code Execution Vulnerabilities (VMSA-2021-0028.9)
Change log:
Impact/Risks:
Do not run the instructions in the Workaround section of this article if you have recently applied Patch 25 from Cumulative Update for vRealize Automation 7.6, as this Cumulative update contains permanent fixes for CVE-2021-44228.