Publishing to XLSX files in TDM Portal do not complete, and seem to hang

Products

• CA Test Data Manager (Data Finder / Grid Tools)

Issue/Introduction

We are trying to generate and publish data in file format (XLSX). But the job is not getting completed and shows as “Running” status for days. This was tested for 1 policy.

Other file formats like CSV and XLS 97-2003 workbook are working fine, and output file is generated as expected. However, the publishing to XLSX is slow and at times hangs, and never completes.

Environment

TDM Portal 4.11.2005.0

Resolution

Publishing to an XLXS file is very memory intensive. See. Publishing Performance Examples.

Publish to XLSX

Publish to XLSX is memory intensive compared to publish to CSV. Using the default CA TDM Portal configuration, the publish hit a wall around 300,000 counts when the CPU usage went high and stayed high. In fact, performance started degrading around 260,000 counts.

This behavior is caused by the GC (Garbage collector) going overdrive when trying to clean up some memory to make sure that the Portal application does not crash with an out-of-memory exception.

The following graph outlines the impact of garbage collection on the Portal:

Tip: Before starting a high-volume publish to XLSX, make sure you increase the memory that is allocated to the Portal.

 

Edit the YASJW config file called wrapper.conf located under CA\CA Test Data Manager Portal\service\conf.
Set either maxmemory (wrapper.java.maxmemory) or maxmemory.percent (wrapper.java.maxmemory.relative) to the desired memory configuration for your system.

With maxmemory.percent, the maximum allocated memory is calculated from the number that was set, times the physical memory.

NOTE: wrapper.java.maxmemory.relative is used to tell the system what percentage (%) of the memory on the server you want to make available to the Java Virtual Machine, whereas wrapper.java.maxmemory is used to define the maximum amount/size (in MB) of memory to allocate the JVM. Both of these properties are commented out by default. Uncomment the one you want to use, and set accordingly for your system's configuration. Restarting the CA Test Data Manager Portal service is required after any changes are made, since the wrapper.conf is read into memory at the time the service is started. 

 

There was an issue found, which can cause the publishing to an XLSX file to hang. This has been resolved in TDMWeb-4.11.2016.0.zip, which is available for download from the Test Data Manager (TDM) Support Patches page.