Tips on maintaining Endevor for Administrators

Products

• Endevor

Issue/Introduction

As an Endevor Administrator there is much more tasks than maintaining Endevor and just apply PTFS.

This document will help administrator get started. 

Environment

All supported Releases

Resolution

One of the many functions as the Endevor administrator, is to ensure that the installation of Endevor is current with maintenance. 

For more information on maintaining the product see the documentation section on Install maintenance. 
It is suggested that a check for maintenance be done monthly and at a minimum of quarterly. 

A - Monitor Space Utilization

As an Endevor Administrator, one of the responsibilities is to monitor the Endevor files on a regular basis to ensure that there is adequate space.  
The information below should help to monitor & maintain those files. 

 

1 - Control the status of Endevor files

The Master Control Files and Package data set are VSAM files, and should be maintained using the standard IBM VSAM maintenance utility IDCAMS.
To obtain file utilization statistics, run the utility with the LISTCAT command.  

Monitor the Endevor libraries regularly, as explained in the following:

1. PDS or PDS/E libraries

   1. Use ISPF/PDF,  Option 3 (Utilities) and/or Option 2 (Data sets) to display space utilization statistics.

   2. PDS or PDS/E libraries include the Endevor listing libraries and processor load library, and may include the base and delta libraries and the source output library.

   
   
2. ELIB data sets

Run the BC1PNLST utility program. ELIB data sets can be used for the base, delta, and/or listing libraries.
For more information about the BC1PNLST utility, see Setting Up ELIB Data Sets.

2 - Expand or Compress a File

If a file becomes full, compress or expand the file, as appropriate. This section explains how to do this for the Master Control File, Package data sets, PDS or PDS/E libraries, and ELIB data sets.

1. Expand or Compress the Master Control File
   
   For the Master Control File (MCF), run the job supplied as member BC1JRMCF in your installation JCL library - CSIQJCL.
   Use this job to expand the files, to periodically clean up control interval splits, or both.
   This job processes the Stage 1 and Stage 2 files.
   When expanding an MCF, you can modify the procedure as necessary.
   
   
2. Expand or Compress Package Data Sets
   
   For package data sets, run the job supplied as member BC1JRPKG in your installation JCL library - CSIQJCL.
   Use this job to expand the data set, to periodically clean up control interval splits, or both.
   When expanding the data set, you can modify the job as necessary.
   
   
3. Reorganize Element Catalog and Eindex Files
   
   For Element Catalog and Eindex files, run the job supplied as member BC1JRCAT in your installation JCL library - CSIQJCL.
   Use this job to reorganize the Element Catalog and Eindex files.
   
   
4. Expand or Compress PDS or PDS/E Libraries
   
   For partitioned data set libraries, run a batch job using the IBM IEBCOPY utility, such as that shown next, to perform the compression.
   The first step of the job should back up the data set in the event of a subsequent problem.
   

   //BACKUP EXEC PGM=IEBCOPY
   //SYSPRINT DDSYSOUT=*
   //DFILE DDDSN=library-dsn,DISP=OLD
   //TFILE DDDSN=seq-backup-dsn,DISP=(NEW,CATLG),UNIT=TAPE
   //SYSIN DD*
    COPY INDD=DFILE,OUTDD=TFILE
   /*
   //*
   //COMPRESS EXEC PGM=IEBCOPY,COND=(0,NE,BACKUP)
   //SYSPRINT DDSYSOUT=*
   //DFILE DDDSN=library-dsn,DISP=OLD
   //SYSIN DD*
    COPY INDD=DFILE,OUTDD=DFILE
   /*
   

5. Expand or Compress ELIB Data Sets
   
   ELIB data sets are automatically maintained each time a member is stored or updated and therefore, compression is unnecessary.
   In addition, ELIB data sets can automatically expand into secondary extents.
   For more information about changing the secondary expansion quantity or the directory size, see Setting up ELIB Data Sets.

B - Backup Endevor files and Inventory.

Everybody understands the need to take backups of critical data, and likewise, everybody agrees that Endevor data is critical data.
Taking backups however is not enough. Backups of Endevor data should also allow restoring to a specific point in time in the shortest time possible and with minimal loss of data.
in case disaster recovery situation  

1 - FULL and INCREMENTAL UNLOAD

Endevor recommends using the Endevor UNLOAD utility to take full and incremental backups of its datasets.
The UNLOAD utility “knows” the internal structures of the Endevor datasets and use an enqueue at the (Endevor-) system level to guarantee that the data that gets backed up is not only restorable but also useable. 
It may be a good idea to do a full UNLOAD once per week or every two weeks and doing incremental backups every night. This way, the full backup can be used to recover from a disk crash; while the incremental backups can be used to recover the latest changes since the full unload. 

The enqueue on the system that is being unloaded can be penalizing because modifications are not possible in this system while the UNLOAD is running, but it is the only way to guarantee useable backup data.

UNLOAD also allows to back-up the package file, but you should note that it does not backup the ACMQ files. These files need to be backed-up manually using a simple IDCAMS job that runs in a few seconds. Make sure you specify DISP=OLD on the ROOT and XREF datasets, otherwise you may get the ROOT and XREF content out of sync. See below for a sample ACMQ backup step.  

 

2 - Other utilities combined with INCREMENTAL UNLOAD

The full UNLOAD can be replaced by any other backup utility (DFSMShsm  component of DFSMS for example), followed by an UNLOAD CHECKPOINT.

The unload checkpoint merely sets a flag in Endevor indicating that a full backup exists and at what date/time it was taken. Its purpose is to set a starting date/timestamp for the incremental backups.

When using other utilities, Endevor does not itself guarantee the integrity of the backed-up data, so this must be done externally.
This means the other utility programs should use some sort of enqueue mechanism to ensure the data to remain in sync while the backup process is executing.

What is usually noticed is that people rely on DFSMShsm’s capability to take secure backups, but it must be observed that a secure backup as assumed by this feature of DFSMShsm is actually not secure for Endevor at all. 
DFSMShsm's definition of secure does not go beyond the backup of a simple data set, which leaves the possibility to back-up a base file (securely) and  its corresponding delta file (or MCF/CATALOG..) securely too, but this doesn't avoid the risk of Endevor changes to be made between these separate data set backups.
It is obvious that a method of backup performed without synchronization consideration can not guarantee that the data will be useable for a recovery.

The only safe way to obtain restorable and useable data when using other utilities for full backups is to lock the full access to ENDEVOR while these backups are taking place in order to guaranty the synchronization of the Endevor inventory.
The comparison of a general lock with the locking of a single SYSTEM like Endevor’s UNLOAD utility clearly gives the advantage to the Endevor solution. 

 

1. Endevor Recommended Backups
   
   

   Weekly and daily backups of Endevor files/data are strongly recommended. Below are the steps that are recommended by the Endevor Support Team.

   Perform a weekly DFDSS Backup of every file related to Endevor (Package file, MCF's, Element Catalog, ACMROOT, ACMXREF, GCF, base and delta files, Output libraries, etc...). Have this done in a single job on either Saturday, Sunday night or any other day or night the condition beeing that there is no activities in Endevor. Support recommends doing the weekly backup with a DISP=OLD on the entry MCFs in order to guarantee that nobody can update ENDEVOR while the weekly DFDSS backups are running. This should secure DFDSS backup quality and guarantee restorability.  
   
   Note: 
   When using LServ, in order to be able to backup Endevor files, files must be temporarily removed from L-Serv, or L-Serv must be shut down.
   For more information, see "how to Disable Point in Time Recovery Journaling ?" explanation available from the  Point in Time Recovery chapter "Perform Periodic Backups of Endevor".

   After the DFDSS backups have been successfully completed, run an Endevor UNLOAD CHECKPOINT ONLY.

               UNLOAD FULL
                 FROM ENV ENV1
                             SYS  *
               CHECKPOINT ONLY.

   For more information on the Unload Utility please reference UNLOAD/RELOAD/VALIDATE.

   NOTE:
   UNLOAD jobs (incremental or full) use an ENQ on the SYSTEM level and will thus exclude updates of the ENDEVOR SYSTEM and its contents during these backups. 
   This will guarantee absolute coherence of the backed-up entities.

   Once the UNLOAD FULL CHECKPOINT is completed, Daily incremental unloads can be performed. It is recommended to run 1 job for each environment. 
   Optionally a DFDSS incremental backup can be performed on the Endevor output libraries, Package file, ACMROOT and ACMXREF files. Below is a sample of the SCL that can be used for the Endevor Incremental Unload:

               UNLOAD INCREMENTAL
                 TO DDNAME dddddddd
                 FROM ENV ENV1
                 SYS *   .

   This procedure will allow to have good backups useable for any restore from if anything ever happens to your Endevor data. Another advantage, is that if an element was deleted by accident a transfer from the unload file can be completed to recover.

   It is strongly recommend that the Unload/Reload Utilities be used for backups for ease of recovery. However, many sites currently do full DFDSS backups which if an UNLOAD/CHECKPOINT is done in conjunction with these, your recovery will be fine. 
   
   

2. ACMQ backups
   

   
   As stated previously, ACMQ backups need to be restorable and useable as well and since UNLOAD does not backup ACMQ data, it must be done independently. This is very easy and extremely fast. 
   The following sample of JCL using the IDCAMS utility should run in a few seconds, and when combined in a run immediately following a full UNLOAD it should ensure a useable ACMQ backup.
   Note: Don’t overlook the DISP=OLD on your ROOT/XREF specifications:

   //COPY    EXEC PGM=IDCAMS                                       
   //IN1      DD DISP=OLD,DSN=uprfx.uqual.ACMROOT               
   //IN2      DD DISP=OLD,DSN=upfrx.uqual.ACMXREF                
   //OUT1     DD DISP=(NEW,CATLG,DELETE),
   //            DSN=uprfx.uqual.ROOT.backup,  
   //            UNIT=SYSDA,SPACE=(CYL,(30,30),RLSE),             
   //            LRECL=4096,RECFM=FB,BLKSIZE=0                    
   //OUT2     DD DISP=(NEW,CATLG,DELETE),
   //            DSN=iprfx.xref.BACKUP,  
   //            UNIT=SYSDA,SPACE=(CYL,(30,30),RLSE),             
   //            LRECL=4096,RECFM=FB,BLKSIZE=0                    
   //SYSPRINT DD SYSOUT=*                                         
   //SYSIN    DD *                                                
    REPRO INFILE(IN1) OUTFILE(OUT1)                                
    REPRO INFILE(IN2) OUTFILE(OUT2)         
   /*

   For more information on other activities that Endevor Administrators should perform take a look at the Roadmap for Administrators Tasks or contact the Broadcom Endevor Support Team. 

   
   
3. GCF backups
   
   The Global Control File required for the Dynamic Environments feature is not yet included in the Unload, Reload, and Validate utility.
   Use a backup utility or the IDCAMS utility REPRO command for GCF file disaster recovery until this functionality is implemented.
   
   
   
4. Point in Time Recovery - PITR
   
   Endevor’s Point in Time Recovery (PITR) extends the capability of the Unload/Reload/Validate utility to include activity since the last backup.
   The Endevor Unload/Reload/Validate utility provides a point of backup recovery mechanism. PITR extends this capability to the recovery of Endevor
   activity that has occurred since the last backup.
   PITR lets you recover changes that are made since the last backup to the Element Catalog (ECF), Element Catalog Index (EIX), Global Control File (GCF), Package Control File (PCF), Master Control File (MCF), and base and delta libraries.
   For further information, see Point in Time Recovery