Certificate overview for VMware Aria Operations
search cancel

Certificate overview for VMware Aria Operations

book

Article ID: 427283

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

There are four main categories related to certificates in VMware Aria Operations

  • Internal (self-signed) certificates
  • Custom certificates
  • Adapter and authentication source certificates
  • Dashboard/Reports - certificate data collected from endpoints

This article aim to explain common queries in relation to the above certificate categories, with links to common KBs that may help answer frequent queries that we see in relation to VMware Aria Operations and certificates

Environment

VMware Aria Operations 8.x

Resolution

Quick-links to topics:

 

Internal certificates:

  • When the first Aria Operations node is deployed, a self-signed CA certificate is generated on the first (primary) node. This self-signed CA certificate is copied to additional nodes that are added to the cluster as part of the node addition process.
  • All analytics nodes will generate a web certificate that is signed against the self-signed CA certificate generated on the first node.
  • The self-signed CA and web certificate will be valid for 5 years from the time the certificate was generated.
  • The self-signed CA and web certificate will be automatically renewed on upgrade to a new build, regardless of whether you upgrade to a maintenance or hotfix build. (ex. 8.18.0 to 8.18 HF1, or 8.18.0 to 8.18.3)
  • The self-signed CA and web certificate will be automatically renewed on upgrade to a new build, even with a custom certificate installed.
  • Since version 8.10 (known as vRealize Operations) the internal certificates are automatically renewed on upgrade. Prior to version 8.10, internal certificate needed to be renewed manually when expired.
  • To avoid expiring internal certificates, ensure that you keep Aria Operations up to date with the latest fixes and security patches by applying the latest maintenance or hotfix release.
  • The self-signed certificates will always be used for internal communication between services internally on each node, and between individual nodes. It is currently not possible to replace the certificate for internal services with a custom certificate. This include services like:
    • Gemfire (ports 10000-10010, 20000-20010)
    • Gemfire Locator (port 6061)
    • Postgres (ports 5432 and 5433)
  • Only the web certificate (port 443) can be replaced with a custom web certificate signed by a certificate authority.
  • For security scans triggering on self-signed certificates for internal services, ensure that you do not allow traffic on those ports outside the network where Aria Operations is deployed. Also ensure that you do not block these ports between the analytics nodes, especially for clusters with continuous availability enabled. You can enable firewall hardening on the Aria Operations cluster via the Admin UI to block access to those ports from nodes outside the cluster. 

Related KBs:
Replace expired internal certificate in Aria Operations - Note that the procedure in this article is not for use with Aria/vRealize Operations higher than version 8.10.

 

Custom certificates:

  • Any certificate that is signed by an external certificate authority is considered a custom certificate from an Aria Operations perspective, regardless whether the web certificate is signed against a self-signed CA certificate though other products from VMware, or 3rd party products.
  • The custom certificate only serves the web UI on port 443, for both Product UI and Admin UI.
  • The custom certificate cannot be used for internal services as mentioned in the internal certificates section above. This is a product limitation.
  • The custom certificate must be applied though the Admin UI, and will be applied to all analytics nodes in the cluster.
  • When one or more certificates in the chain has expired, you will not be able to replace the custom certificate without first reloading the default certificate.
  • When one or more certificates in the chain has expired, you will not be able to bring the cluster online from an offline state through admin UI, without first reloading default certificate.  
  • Aria Operations will continue to operate even when a certificate in the custom certificate chain has expired. Restarting cluster from Admin UI, or individual services form CLI will result in services failing to start.
  • Some certificate authorities enforce use of it's own key when signing a certificate signing request (CSR) rather than the key that were generated with the CSR. The KB for manually validating PEM file below can help you identify this.

Related KBs:
Configure a Certificate For Use With VCF Operations - Details the steps to generate a certificate signing request, and assemble the PEM file in the correct order for use with Aria Operations.
Reload the default certificate in Aria Operations - Instruction on reloading the default internal certificate for use as web certificate. Needed when one or more certificates in the custom certificate chain is expired.
Using the Custom Certificate Tool in VMware Aria Operations - Instruction on using built-in tool (8.x) to validate a custom certificate chain.
Manually validating custom certificate chain PEM file - Detailed KB on manually validating a custom certificate chain. Product agnostic.

 

Adapter and authentication source certificates:

  • Adapter and authentication source certificates are certificates that are installed on endpoints where we collect data through adapters, or authentication sources that is configured in Aria Operations, such as AD or LDAP.
  • Endpoint certificates are stored on Aria Operations to ensure secure communication between Aria Operations and the endpoint.
  • The endpoint certificates are found under Administration -> Control Panel -> Trusted Certificates in the Product UI. For versions older that 8.18, the navigation to Trusted Certificates will be slightly different.
  • When a certificate is replaced for an adapter endpoint. The data collection stops, as trust has to be re-established through Validate Connection for the affected adapter instance.
  • When a certificate is replaced for an authentication source. Users may not be able to log in with that authentication source, and automatic synchronization will fail, until trust is re-established through Test Connection for the affected authentication source.
  • When a certificate is replaced for any endpoint, the new certificate is added to Trusted Certificates after importing via validate/test connection. The old certificate will remain in Trusted Certificates until manually deleted.
  • When a certificate listed under Trusted Certificates expires, a banner will display on the home page, indicating that there is an expired certificate. This also includes certificates that has already been replaced for any endpoint.
  • It's generally safe to delete certificates under Trusted Certificates, particularly when there is no object listed in the 'Trusted Objects' column. Note that manually imported certificates may not have any assigned trusted object. For certificates that has a trusted object assigned, you will get a warning before deleting. You can validate/test connection for an endpoint to import current certificate from the endpoint. It's advisable to take snapshots of the Aria Operations analytics nodes prior to deleting certificates.

Related KBs:
How to renew an adapter certificate or clear the Expired Certificates banner in Aria Operations - Details steps to import new certificate when endpoint certificate has been renewed, and clean up expired trusted certificates.
Failed to setup SSO source, reason: Auth Exception occurred : 'Solution user detail' certificate is invalid - 'Solution user detail' certificate is invalid when configuring SSO Authentication source
AD/LDAP credentials do not always work in Aria Operations and the integration must be revalidated or refreshed to import the certificate - AD backed by multiple DCs presenting different certificates
vCenter adapter integration failure in Aria Operations - vCenter user account name case-sensitive, and format of username
Remove VMware Infrastructure Health adapter instance through API - How to remove VMware Infrastructure Health adapter via API. Adapter is automatically recreated on service restart.

 

Dashboard/Reports - certificate data collected from endpoints:

  • Aria Operations collect information on certificates used by other components, such as vCenter, NSX, and other VMware appliances. 
  • The certificate data is collected through the out-of-the-box VMware Infrastructure Health adapter.
  • The VMware Infrastructure Health adapter is installed automatically, and cannot be added/removed from integrations pages manually.
  • The collected certificate data used for dashboards/reports are mostly used in relation to certificate validity period.
  • Additional certificate data are stored under the Certificates [VMware Infrastructure health] object type. Search: Certificates

Related KBs:
Diagnostics for VMware Cloud Foundation - Certificates “No data available” - Identified problem with VMware Infrastructure Health adapter in releases from 8.18.0 up to and including 8.18.2. Resolved in 8.18.3.
Diagnostics certificates panel in Aria Operations showing expired certificates that are not in use - Workaround for unused expired certificates displaying in diagnostics.

 

Analytics node definition: Primary, Primary Replica, and Data nodes are considered analytics nodes. Cloud proxies are not considered here even though they collect data for Aria Operations.
Endpoint definition: This refers to any endpoint that Aria Operations interact with through data collection, other integrations, or authentication sources that requires trust between Aria Operations and the endpoint.

Additional Information

Internal and Custom certificates are stored in directory /storage/vcops/user/conf/ssl.

The following directory listing contains comments with regards to the most important certificate related files in this directory:

root@<HOSTNAME> [ /storage/vcops/user/conf/ssl ]# ls -al
total 296
drwxr-xr-x 2 admin    admin   4096 Feb  2 11:42 .
drwxr-xr-x 4 admin    admin   4096 Feb  2 11:29 ..
-r--r--r-- 1 admin    admin   1419 Feb  2 11:29 cacert.pem                          # Internal (self-signed) CA certificate
-r-------- 1 admin    admin   1675 Feb  2 11:29 cakey.pem                           # Internal (self-signed) CA key
-rw-r----- 1 admin    admin      0 Dec 15 10:03 certs-generated-by-casa
-rw-r----- 1 admin    admin      6 Dec 15 11:06 cert.type                           # Indicates CUSTOM or DEFAULT certificate
-r--r--r-- 1 admin    admin   1289 Dec 15 10:08 cluster_cert.pem                    # Used by internal processes only
-r-------- 1 admin    admin   1704 Dec 15 10:08 cluster_key.pem                     # Used by internal processes only
-rw-r----- 1 admin    admin   2534 Feb  2 11:29 cluster.truststore
-r--r--r-- 1 admin    admin   2232 Dec 15 11:06 customCert.pem                      # Custom web certificate
-r--r--r-- 1 admin    admin   1302 Dec 15 11:06 customChain.pem                     # Custom chain (intermediate and root)
-r-------- 1 admin    admin   1679 Dec 15 11:06 customKey.pem                       # Custom web key
-rw-r----- 1 admin    admin 182463 Feb  2 11:44 jre_and_tcserver.truststore
-r--r--r-- 1 admin    admin   1249 Feb  2 11:29 postgres_vcops_cert.pem             # Internal PGSQL certificate (PG_DATA_DB)
-r-------- 1 admin    admin   1675 Feb  2 11:29 postgres_vcops_key.pem              # Internal PGSQL key (PG_DATA_DB)
-r--r--r-- 1 postgres root    1253 Feb  2 11:29 postgres_vcopsrepl_cert.pem         # Internal PGSQL certificate (PG_REPL_DB)
-r-------- 1 postgres root    1675 Feb  2 11:29 postgres_vcopsrepl_key.pem          # Internal PGSQL key (PG_REPL_DB)
-rw-rw---- 1 admin    admin   2684 Nov 24 18:31 secure-communications.properties
-rw-rw---- 1 admin    admin   1589 Nov 24 18:31 security.properties
-r--r--r-- 1 admin    admin   1342 Feb  2 11:29 slice_#_cert.pem                    # Internal (self-signed) web certificate
-r-------- 1 admin    admin   2510 Feb  2 11:29 slice_#_cert.pfx
-r-------- 1 admin    admin   1679 Feb  2 11:29 slice_#_key.pem                     # Internal (self-signed) web key
-rw-r----- 1 admin    admin    104 Feb  2 11:38 storePass.properties
-rw-r----- 1 admin    admin   2305 Feb  2 11:29 tcserver.keystore
-rw-r----- 1 admin    admin   5334 Feb  2 11:38 tcserver.truststore
-rw------- 1 admin    admin   5213 Dec 15 11:06 uploaded_cert.pem                   # Full custom chain as uploaded via Admin UI
lrwxrwxrwx 1 admin    admin     43 Dec 15 11:06 web_cert.pem -> /storage/vcops/user/conf/ssl/customCert.pem    # Symbolic link to web cert
lrwxrwxrwx 1 admin    admin     44 Dec 15 11:06 web_chain.pem -> /storage/vcops/user/conf/ssl/customChain.pem  # Symbolic link to cert chain
lrwxrwxrwx 1 admin    admin     42 Dec 15 11:06 web_key.pem -> /storage/vcops/user/conf/ssl/customKey.pem      # Symbolic link to cert key

Note: Do not manually change the symbolic links for web_cert.pem, web_chain.pem, and web_key.pem. Always use command as per Reload the default certificate in Aria Operations to change between default and custom certificate when this is required.

Powered by Wolken Software