Upgrading of OpsMan to 3.0.41 breaks CPI functionality to Azure

Products

• VMware Tanzu Platform

Issue/Introduction

In Azure CanadaCentral, Upgrading of OpsMan to 3.0.41 breaks CPI functionality to Azure

OpsMan version in question = 3.0.41 

In upgrading OpsMan to version 3.0.41 in customer's Lab Azure CanadaCentral environment, the OpsMan VM is unable to communicate via the CPI to Azure. In performing this upgrade in customer's other Lab environment, which is running out of Azure EastUS2, the upgrade succeeds and communication to Azure is as expected - it does appear that this issue affects CanadaCentral only.

After upgrading OpsMan to 3.0.41, the attempt to upgrade the Bosh Director to 3.0.41 as well as upgrading the stemcell produces the following error:

 

 CPI 'create_stemcell' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"Failed to upload page blob: inspect:#\u003cArgumentError: proxy must be :ENV or a URI::HTTP\u003e\n backtrace:/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/net-http-persistent-4.0.5/lib/net/http/persistent.rb:812:in `proxy='\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday-net_http_persistent-1.2.0/lib/faraday/adapter/net_http_persistent.rb:26:in `net_http_connection'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday-net_http-1.0.2/lib/faraday/adapter/net_http.rb:43:in `build_connection'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday-1.10.4/lib/faraday/adapter.rb:47:in `connection'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday-net_http-1.0.2/lib/faraday/adapter/net_http.rb:63:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday_middleware-1.2.1/lib/faraday_middleware/response/follow_redirects.rb:79:in `perform_with_redirection'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday_middleware-1.2.1/lib/faraday_middleware/response/follow_redirects.rb:67:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday-1.10.4/lib/faraday/rack_builder.rb:154:in `build_response'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/faraday-1.10.4/lib/faraday/connection.rb:516:in `run_request'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/http_response_helper.rb:27:in `set_up_response'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/http_request.rb:150:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/retry_policy.rb:48:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/retry_policy.rb:48:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/http_request.rb:111:in `block in with_filter'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/signer_filter.rb:28:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/signer_filter.rb:28:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/http/http_request.rb:111:in `block in with_filter'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/service.rb:36:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/filtered_service.rb:34:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/core/signed_service.rb:41:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-common-2.0.4/lib/azure/storage/common/service/storage_service.rb:60:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-blob-2.0.3/lib/azure/storage/blob/blob_service.rb:179:in `call'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/azure-storage-blob-2.0.3/lib/azure/storage/blob/page.rb:109:in `create_page_blob'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:392:in `_create_page_blob'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:83:in `block in create_page_blob'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:330:in `block in _initialize_blob_client'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:318:in `synchronize'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:318:in `_initialize_blob_client'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/storage/blob_manager.rb:75:in `create_page_blob'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager.rb:48:in `block in create_stemcell'\n/var/vcap/packages/azure-cpi-ruby-3.3/lib/ruby/3.3.0/tmpdir.rb:99:in `mktmpdir'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager.rb:41:in `create_stemcell'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/stemcell/stemcell_manager2.rb:20:in `create_stemcell'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:88:in `block (2 levels) in create_stemcell'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/telemetry/telemetry_manager.rb:71:in `monitor'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:84:in `block in create_stemcell'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/bosh_common-2.0.0/lib/common/thread_formatter.rb:50:in `with_thread_name'\n/var/vcap/packages/bosh_azure_cpi/lib/cloud/azure/cloud.rb:80:in `create_stemcell'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/bosh_cpi-2.6.0/lib/bosh/cpi/cli.rb:90:in `public_send'\n/var/vcap/packages/bosh_azure_cpi/gem_home/ruby/3.3.0/gems/bosh_cpi-2.6.0/lib/bosh/cpi/cli.rb:90:in `run'\n/var/vcap/packages/bosh_azure_cpi/bin/azure_cpi:36:in `\u003ctop (required)\u003e'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/cli/exec.rb:59:in `load'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/cli/exec.rb:59:in `kernel_load'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/cli/exec.rb:23:in `run'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/cli.rb:452:in `exec'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/vendor/thor/lib/thor.rb:538:in `dispatch'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/cli.rb:35:in `dispatch'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/cli.rb:29:in `start'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/exe/bundle:28:in `block in \u003ctop (required)\u003e'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'\n/usr/local/lib/ruby/gems/3.3.0/gems/bundler-2.6.8/exe/bundle:20:in `\u003ctop (required)\u003e'\n/var/vcap/packages/azure-cpi-ruby-3.3/bin/bundle:25:in `load'\n/var/vcap/packages/azure-cpi-ruby-3.3/bin/bundle:25:in `\u003cmain\u003e'","ok_to_retry":false}

Resolution

The issue is now resolved. The reason for this is because the affected environment's proxy settings were set to "" instead of null, which may have been caused by setting the proxy and unsetting it in the past. It looks like a change between OpsMan 3.0.38 and 3.0.41 made it become more strict about the proxy values.

curl using the POST method to update these values to null, which resolved our issue successfully. 

Command used: om -e interpolated_env.yml curl -p /api/v0/settings/proxy_settings -x PUT -d '{"proxy_settings":{"http_proxy":null,"https_proxy":null,"no_proxy":null}}'