Enable FIPS causes failed to fetch system information

Products

VCF Automation

Issue/Introduction

After enabling FIPS they system is rebooted and FIPS is enabled. However, they system information and network information does not show anything,

Environment

Life Cycle Manager: 8.18.0.0 (PATCH1)

Product Support Pack : 5

Build : 24029603

Cause

This could be due to multiple reasons

1. Postgres DB doesn't have enough storage to accommodate new entries
2. Engine is not picking the request due to overload
3. There could be stuck requests in engine for long time which is blocking newer requests to be picked up.

Resolution

1. Postgres DB doesn't have enough storage to accommodate new entries

To see the current table sizes use this query

Go to the psql prompt:

 
psql -h localhost -d vrlcm -U postgres

To displat the table sizes sorted by size:

 
SELECT 
	table_name, 
	  pg_size_pretty( pg_total_relation_size(quote_ident(table_name))), 
	  pg_total_relation_size(quote_ident(table_name))
	FROM 
	  information_schema.tables
	WHERE 
	  table_schema = 'public'
	ORDER BY 
	  pg_total_relation_size(quote_ident(table_name)) DESC;

Ideally the table sizes shouldn't be huge, i.e should be under 300 Megabytes. Even this size is for long running setups with lot of requests executed. If the size is huge for tables like 'vm_engine_requests' or 'vm_engine_events' or 'vm_rs_requests' this is most likely a case of table bloat and we can use the commands below to do a 'VACUUM FULL verbose analyze' to defragment the DB data.

Since storage partition might be filled-up, we need to manually increase the disk size and see if that help. But this could be due to toasts tables, suggesting to vacuum the table first.

check the db partition /storage used space and available space

Filesystem                      Size  Used Avail Use% Mounted on

devtmpfs                        2.9G     0  2.9G   0% /dev

tmpfs                           3.0G   12K  3.0G   1% /dev/shm

tmpfs                           3.0G  1.1M  3.0G   1% /run

tmpfs                           3.0G     0  3.0G   0% /sys/fs/cgroup

/dev/sda4                       8.7G  2.6G  5.7G  32% /

/dev/sda2                       120M   39M   76M  34% /boot

tmpfs                           3.0G   27M  2.9G   1% /tmp

/dev/mapper/stroage_vg-storage  9.8G  9.7G  9.2G   99% /storage

/dev/mapper/data_vg-data         49G  5.7G   41G  13% /data

tmpfs                           595M     0  595M   0% /run/user/0

```
Login to vRSLCM using ssh and type below commands to connect to postgres,
```
```
su - postgres
cd /opt/vmware/vpostgres/11/bin
./psql -d vrlcm
```
PS: Incase if postgres login is not working due to space issue, please increase the disk size for "/storage" partition and reboot vRSLCM

If the size of the /storage is mostly used like more than ~70%, please try below,

VACUUM FULL verbose analyze vm_rs_request;

If step-2 didn't help reducing the used size in /storage, increase the disk size from vCenter and reboot vRSLCM.

2. Engine is not picking the request due to overload and stuck requests

Incase solution-1 didn't work, This could be due to engine is not picking up the request to due to already processing requests (overloaded) or genuine unprocessed but stuck requests.

Connect to postgres DB
Run the postgres query select currentState,status from vm_engine_event where status='IN_PROGRESS';
Check if there is any request with status "IN_PROGRESS" If this number is greater than or equal to 25 incase of < vRSLCM 8.3 release and greater than or equal to 50 incase of vRSLCM 8.4 onwards. Please contact support if this is the case