Watch4Net, ViPR SRM: Import properties on frontend taking too long to run
Article ID: 324755
Updated On:
Products
VMware
Issue/Introduction
Symptoms:
The M&R Frontend runs a scheduled task called import-properties , which creates an indexed database (property-store) of metrics and properties used during report generation. This database is normally found in /opt/APG/Databases/APG-Property-Store .
As the number of metrics and properties increases, it takes longer to generate the property-store. The conditional execution logic of the task, by default, will schedule the import-properties task to run once a day at midnight if the average time to generate is more than 1 hour (conditional execution logic can be changed in import-properties.task).
This has a negative impact on the timeliness of data in reports. If the property-store is recreated each day at midnight and takes 12 hours, a device added to collection on Tuesday morning will not appear in reports until Wednesday afternoon.
Dell EMC engineering recommends that the import-properties task can be reconfigured to run more quickly, so that changes to metrics and properties are reflected in reports sooner.
The M&R Frontend runs a scheduled task called import-properties , which creates an indexed database (property-store) of metrics and properties used during report generation. This database is normally found in /opt/APG/Databases/APG-Property-Store .
As the number of metrics and properties increases, it takes longer to generate the property-store. The conditional execution logic of the task, by default, will schedule the import-properties task to run once a day at midnight if the average time to generate is more than 1 hour (conditional execution logic can be changed in import-properties.task).
This has a negative impact on the timeliness of data in reports. If the property-store is recreated each day at midnight and takes 12 hours, a device added to collection on Tuesday morning will not appear in reports until Wednesday afternoon.
Dell EMC engineering recommends that the import-properties task can be reconfigured to run more quickly, so that changes to metrics and properties are reflected in reports sooner.
Environment
VMware Smart Assurance - Watch4Net/M&R
Resolution
There are two steps in property-store generation, each can be optimized by parallel processing.
An import-properties task can be installed and run on each backend. This will export the metrics and properties for the database into a file that can be fetched by the frontend, without the frontend needing to run a query for each database.
Each import-properties task should appear in the GUI and should run without errors. The BE system task scheduler log file should have messages like the below:
The extracted data is in the following location on the BE system by default:
The import-properties task running on the frontend does not require any changes, if the extracted data is found on the backend it will be fetched, with messages like the following in the FE system taskRun log file:
Improvements to the frontend task run time should be visible in the GUI under Scheduled Tasks .
Optimizing Indexing
After the metrics and properties for each backend are fetched they are indexed and written out to the APG-Property-Store directory. The size of this directory can be obtained as follows:
As a rule, this directory can be split up into smaller directories, each being around 5gb in size. In the procedure below, it is assumed that the monolithic cache DB is being split into 6: (DB1..DB6)
4.Once the import-properties tasks have completed, on the frontend:
a.Stop the tomcat service (manage-modules.sh service tomcat stop)
b.Delete the APG-Property-Store DB by removing the directory /opt/APG/Databases/APG-Property-Store/Default/data/DB
c.Run the import-properties task from the GUI. This may take several hours, so there will be an extended pause in the procedure
d.Check that the directory /opt/APG/Databases/APG-Property-Store/Default/data has subdirectories DB1..DB6.
e.Start the tomcat service (manage-modules.sh service tomcat start)
f.Login with the Web GUI and confirm that reports work as before.
5.If there is a need to back out
a.Stop tomcat on frontend
b.Restore the resources table from master database from the saved version.
c.Delete the (DB1..DB6) directories and run the import-properties task, that should recreate the DB directory
d.Start tomcat on front end and check reports
6.Enable the import-properties tasks on front end. Check the time to run.
- Data extraction and fetching. The metrics and properties from each database are exported and retrieved by the front end.
- Indexing. The fetched data is indexed to create the property-store used by the frontend.
An import-properties task can be installed and run on each backend. This will export the metrics and properties for the database into a file that can be fetched by the frontend, without the frontend needing to run a query for each database.
| Install one property-Store package per backend So, you ll have one property-store on the PBE and 4 property-store on each ABE This is distribute the task execution in smaller/local chunks and FE will pull a pre-generated version instead of generating itself. PBE manage-modules.sh install property-store Default ABE manage-modules.sh install property-store apg1 manage-modules.sh install property-store apg2 manage-modules.sh install property-store apg3 manage-modules.sh install property-store apg4 Remarks On the ABE, it will ask you each time where is the backend associated to the property-store. Example # manage-modules.sh install property-store apg4 Required dependencies, in processing order: [1] java '8.0.51' v8.0.51 [2] module-manager '1.8u1' v1.8u1 [3] jdbc-drivers 'Default' v2.6 [4] I property-store 'apg4' (none) => v2.1u1 > 3 not modified, 1 to install > 7.5 MB space required / 89.7 GB available ? Enter the step to modify, 'yes' to accept them, or 'no' to cancel the operation [yes] > y Starting installation of APG-Property-Store v2.1u1 from property-store-2.1u1-linux-x64... * Gathering information... * 'APG-Property-Store v2.1u1' will be registered with instance name 'apg4'. * It will be installed in '/opt/APG/Databases/APG-Property-Store/apg4'. * Unpacking files... [1] Backends/APG-Backend/apg1/conf/mysql.xml [2] Backends/APG-Backend/apg2/conf/mysql.xml [3] Backends/APG-Backend/apg3/conf/mysql.xml [4] Backends/APG-Backend/apg4/conf/mysql.xml ? Which file should the 'import-property.task' be configured with ? [4] > 4 * Installing files... 100% * 42 files have been installed. * Finalizing installation... ? Task file import-properties.task added by module property-store apg4. Enable it? (yes/no/All/None) [y] > y Installation complete. |
Each import-properties task should appear in the GUI and should run without errors. The BE system task scheduler log file should have messages like the below:
| /opt/APG/Tools/Task-Scheduler/Default/logs [root]# tail scheduler-0-0.log INFO -- [2017-01-30 10:00:00 CST] -- : Task import-properties-Default -- Starting task import-properties-Default. INFO -- [2017-01-30 10:00:01 CST] -- : Task import-properties-Default -- Successful completion of task import-properties-Default with status 0. |
The extracted data is in the following location on the BE system by default:
| /opt/APG/Databases/APG-Property-Store/Default/tmp [root]# ll total 4 drwxr-xr-x 3 apg apg 4096 Feb 16 16:11 _NONE_ -rw-r--r-- 1 apg apg 0 Jan 27 01:00 _NONE_.lock |
The import-properties task running on the frontend does not require any changes, if the extracted data is found on the backend it will be fetched, with messages like the following in the FE system taskRun log file:
| 170216 01:02:38 : Feb 16, 2017 1:02:38 AM com.watch4net.propstore.apg.tasks.FetchPropertiesTask compute 170216 01:02:38 : INFO: Fetching data table 'data_property' from partition 'APG-apg1' into '/opt/APG/Databases/APG-Property-Store/Default/tmp/DB/APG-DB-nsm123vp141b-1'... 170216 01:02:39 : Feb 16, 2017 1:02:39 AM com.watch4net.propstore.apg.tasks.FetchPropertiesTask completed |
Improvements to the frontend task run time should be visible in the GUI under Scheduled Tasks .
Optimizing Indexing
After the metrics and properties for each backend are fetched they are indexed and written out to the APG-Property-Store directory. The size of this directory can be obtained as follows:
| [root2@nsmdenvp138b DB]# pwd /opt/APG/Databases/APG-Property-Store/Default/data/DB [root2@nsmdenvp138b DB]# du -m 17009 ./1487206984103 16996 ./1487120564721 34004 . |
As a rule, this directory can be split up into smaller directories, each being around 5gb in size. In the procedure below, it is assumed that the monolithic cache DB is being split into 6: (DB1..DB6)
- Disable the import-properties tasks on the frontend, stop the task if it is running.
- Before starting procedure: save copy of resources table from master database on PBE
| /opt/APG/Databases/MySQL/Default/bin/mysqldump defaults-file=/opt/APG/Databases/MySQL/Default/my.cnf --user=root -p master resources > /root/resources.sql |
- On the PBE, run manage-resources.sh update for each APG database, changing the setting:cachegrp so that instead of being DB in every case, it uses a value from (DB1..DB6). After running the commands to update the resources, the import-properties tasks should be run for each backend.
| To get list of resources: /opt/APG/bin/manage-resources.sh list | grep APG "dba/APG-DB", dba/APG-DB-nsm123vp141b-1 , Etc. Update cachegrp for each APG DB: /opt/APG/bin/manage-resources.sh update dba/APG-DB "{'settings':{'cachegrp':'DB1'}}" /opt/APG/bin/manage-resources.sh update dba/APG-DB-nsm123vp141b-1 "{'settings':{'cachegrp':'DB2'}}" Etc. |
4.Once the import-properties tasks have completed, on the frontend:
a.Stop the tomcat service (manage-modules.sh service tomcat stop)
b.Delete the APG-Property-Store DB by removing the directory /opt/APG/Databases/APG-Property-Store/Default/data/DB
c.Run the import-properties task from the GUI. This may take several hours, so there will be an extended pause in the procedure
d.Check that the directory /opt/APG/Databases/APG-Property-Store/Default/data has subdirectories DB1..DB6.
e.Start the tomcat service (manage-modules.sh service tomcat start)
f.Login with the Web GUI and confirm that reports work as before.
5.If there is a need to back out
a.Stop tomcat on frontend
b.Restore the resources table from master database from the saved version.
| /opt/APG/Databases/MySQL/Default/bin/mysql --defaults-file=/opt/APG/Databases/MySQL/Default/my.cnf --user=root -p master < /root/resources.sql |
c.Delete the (DB1..DB6) directories and run the import-properties task, that should recreate the DB directory
d.Start tomcat on front end and check reports
6.Enable the import-properties tasks on front end. Check the time to run.
Feedback
Yes
No
Powered by
