Handling large lists in UDPs

Products

CA Harvest Software Change Manager - OpenMake Meister CA Harvest Software Change Manager

Issue/Introduction

When passing a large list of files to a UDP, the UDP fails with an error.

E03020132: Could not find UDP program...

Recreating the Problem

The following sample scenario recreates the problem:

Create a Harvest project and repository. Attach the repository to the project.
Check in more than 20,000 files.

Create a sample Perl program (C:\Scripts\ARG_Test.pl) with the following code:

#!/usr/bin/perl
use strict;
use warnings;
# This example script reads parameters from the command line
# and writes them to a text file named C:\Scripts\CMDARGS.TXT

# Open a file for output
open(my $ARGUMENTS, '>', 'C:\Scripts\CMDARGS.TXT');

# Loop through list of arguments, printing each to the file
foreach my $arg ( @ARGV )
{
  print $ARGUMENTS " $arg\n";
}

# Close the output file
close ($ARGUMENTS);
exit(0);

Create a pre-link UDP to the check out process that passes the [version] variable. This UDP takes all versions and writes them to a file. Set up your UDP Properties dialog as follows:

Name: ARG Test
Program: perl C:\Scripts\ARG_Test.pl [version]
Type: Server
No Prompt: (unchecked)
Secure Input: (checked)
Allow Additional Parameters: (checked)
Input: (empty)
Output: display
Errors: display

Select 20,000 files and try to check them out. This results in the failure message:

---------- Begin  <Check Out for Browse>  Process ---------------
E03020132: Could not find UDP program: perl C:\Scripts\ARG_Test.pl   \SampleRepository\Folder-f\Folder-f\Folder-c\File-f.txt;0   \SampleRepository\Folder-c\Folder-i\Folder-c\File-j.txt;0 ...
S10060050: The PreLinked process(es) failed for parent process: Check Out for Browse.
E03060019: Process Execute failed. Process Name: Check Out for Browse.
---------- End  <Check Out for Browse>  Process ---------------

Environment

Harvest Software Change Manager all versions

Cause

Processing large lists with a User-Defined Process (UDP) can be a challenging task depending on which operating system you use. Harvest SCM offers different ways to handle the input into a UDP so that your process results will not be affected by the operating system limitations to the command line. This article effectively describes how to invoke and write UDPs that can handle all kinds of large lists.

Using "Harvest System Variables" in UDPs

The use of UDPs as pre-link or post-link processes is very common in Harvest SCM (referred to simply as Harvest). Additionally, stand-alone UDPs are widely used to enhance the native capabilities of Harvest and to meet the expectations and needs of the users.

The power of Harvest is to create generic UDPs that later, during execution, are adapted to the specific context that the UDP is running in. To make UDPs generic, Harvest allows you to pass Harvest System Variables as parameters in an argument list to the command or script being run by the UDP. This allows the parameters to provide specific values depending on the context set in Harvest. For example, if a UDP uses the [package] variable, this variable is replaced at execution time with the name of the package that is selected. When promoting a package, the user may want to send a notification email to managers or run a script automatically when a package is promoted to the next state.

Most of the Harvest System Variables return a single value, for example, [broker] or [project]. Other variables return a list of values, for example, [version], [file], and [package], depending on the selection by the user. For example, if a user selects 10 packages to promote to the next state, the [package] variable will be replaced with the names all 10 packages.

The most problematic variables are [version] and [file]. These variables can be replaced with a very lengthy list. A check out pre-link UDP that uses the [version] variable to process the list of versions will become very lengthy if a user executes a complete check out of the repository attached to a project. Problems are more likely to occur on big projects, such as Java projects where the files are small but numerous and the view paths are very long (Java Package naming convention).

Argument List Limitations

For any script or command a UDP might execute, the maximum length (in characters) of the argument list varies depending on the operating system. Harvest interacts with the operating system to execute the UDP's command or script and is dependent on the operating system capabilities and limitations in this regard.

The UNICODE_STRING structure limits the maximum argument list length for the CreateProcess function to 32,767 characters. CreateProcess is the core function for creating processes, so if you are using Win32 that is the only limitation concern. If using the CMD.EXE command processor, you are additionally subject to an 8,192 character command line length limitation imposed by CMD.EXE. If you want to pass more than 32,767 characters of information to a process, you need to use a method other than the command line. Some options will be discussed later in this document.

UNIX has a larger limitation than the Unicode limitation, but it is not unlimited. Limitations on UNIX can vary and the maximum is 4 MB. In the case of a large Harvest project with numerous versions and items, this 4 MB limitation is insufficient. Some users execute UDPs with version lists that may reach 20 MB. All UNIX and Linux operating systems can be adapted, but all of them share the same problem that occurs on very lengthy lists.

For example, Sun UNIX limits the length of the command line. In the file,
/usr/include/limits.h, the limitations are defined. See the line:

#define MAX_INPUT       512     /* max size of a char input buffer */

By changing these values, the Sun UNIX administrator may allow longer lists, but an UNLIMITED setting is preferred.

Despite all this, the use of arguments is not the best method for handling very lengthy lists.

Differences Between [version], "[version]" and ["version"]

The variable [version] is used once in the UDP setup that is described in the above section Recreating the Problem. The variable ["version"] is used in the UDP setup that is described in the next section Handling Very Lengthy Lists in UDPs. By using both variables, you may better understand how to handle these variables and the difference between them. The following table lists and describes the version variables:

["version"]	Quotes located in brackets expand the system variable list members quoted individually: ["version"] evaluates to "version name 1" "version name 2" ...
"[version]"	Quotes located outside the brackets expand the system variable list members as one quoted list: "[version]" evaluates to "version name 1 version name 2 ..."
[version]	With no quotes located outside the brackets, the system variable list members expands as a simple list: [version] evaluates to version name 1 version name 2 ...

Resolution

Handling Very Lengthy Lists in UDPs

The Harvest UDP dialog includes an Input field.

Harvest System Variables can be used in this field and Harvest will parse the variables and pass their data to the UDP's command or script via the <STDIN> stream rather than as command line parameters.

The following sample scenario fixes the problem of lengthy lists:

Create a sample Perl program (C:\Scripts\STDIN_Test.pl) with the following code:

#!/usr/bin/perl
use strict;
use warnings;
# This example script reads data passed in via the STDIN stream
# and writes all lines to a text file named C:\Scripts\STDINPUT.TXT

# Open a file for output
open(my $ARGUMENTS, '>', 'C:\Scripts\STDINPUT.TXT');

# Loop through all lines of <stdin> input and add them to an array
my @STIN;
while (<STDIN>)
{
  next if /^\s*$/;  # ignore blank lines
  s/^\t//;          # trim the leading tab
  chomp;            # trim the trailing EOL
                    # version, file or package name is now in $_ 
  push(@STIN, $_);  # push the $_ into an Array
}

# Loop through the array and print each line to the file
foreach my $stin ( @STIN )
{
  print $ARGUMENTS " $stin\n";
}

# Close the file
close ($ARGUMENTS);
exit(0);

Change the pre-link UDP configuration to call the new script:

Name: STDIN Test
Program: perl C:\Scripts\STDIN_Test.pl
Type: Server
No Prompt: (unchecked)
Secure Input: (checked)
Allow Additional Parameters: (checked)
Input: ["version"]
Output: display
Errors: display

Select all the files in the repository and try to check them out. Harvest succeeds with the message:

---------- Begin  <Check Out for Browse>  Process ---------------
I00060052: STDIN Test execution was successful.
I00020110: File \SampleRepository\File-a.txt;0 checked out to example-broker\\C:\Temp\File-a.txt.
I00020110: File \SampleRepository\File-b.txt;0 checked out to example-broker\\C:\Temp\File-b.txt.
I00020110: File \SampleRepository\File-c.txt;0 checked out to example-broker\\C:\Temp\File-c.txt.
...
I00060080: Check out summary: Total: 11,110; Success: 11,110; Failed: 0; Not Processed: 0.
---------- End  <Check Out for Browse>  Process ---------------

And resulting STDINPUT.TXT file looks like this:

 "\SampleRepository\Folder-f\Folder-f\Folder-c\File-f.txt;0"
 "\SampleRepository\Folder-c\Folder-i\Folder-c\File-j.txt;0"
 "\SampleRepository\Folder-a\Folder-i\Folder-i\File-a.txt;0"
 "\SampleRepository\Folder-j\Folder-c\Folder-a\File-d.txt;0"
 "\SampleRepository\Folder-a\Folder-h\Folder-c\File-b.txt;0"

Sample Java Code to Handle STDIN

The same idea applies with any programming or scripting language that accepts input from the "STDIN" input stream. Here is a Java example:

package stdinJavaTest;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
import java.util.Scanner;

// This is an example Java program that accepts input from STDIN and writes the same to a text file

public class stdinJavaTestMain {

   public static void main(final String[] args) {

      // Set the path and name of the output file
      Path file = Paths.get("C:\\Scripts\\output.txt");

      // Read from Stdin
      try (Scanner scanner = new Scanner(System.in)) {
         while (scanner.hasNextLine()) {
            String line = scanner.nextLine();
            try {

               // Write to the file
               Files.writeString(file, line + System.lineSeparator(), StandardOpenOption.CREATE);

            } catch (IOException e) {
               e.printStackTrace();
            }
         }
      }
   }
}

Additional Information

The above scenarios, Perl and Java program are offered as-is, as an example only to show what is possible. Beyond further explanations of what these scenarios and programs do, support or enhancements to the above examples will not be provided by the support team. You should use these as a starting point to design your own custom solution that meets your requirements.