How gp_log_collector automates the collection of VMware Tanzu Greenplum Logs

Products

• VMware Tanzu Greenplum

Issue/Introduction

This article applies to the following:

• VMware Tanzu Greenplum Database (GPDB) v5.x.
• VMware Tanzu Greenplum Database (GPDB) v6.x.
• VMware Tanzu Greenplum Database (GPDB) v7.x.

Note:

The "gpmt" is delivered with Greenplum 5.28 and above and 6.16 and above.

In Greenplum 7.x: The tool has been renamed to "gpsupport". Use "gpsupport" instead fo "gpmt" in the examples given in this KB.

Resolution

gp_log_collector is a tool within the Greenplum Magic Tool (GPMT) meta-tool.

It uses ssh, mkdir, scp, and tar to collect GPDB logs on the Greenplum cluster.

The purpose of this tool is to make supporting GPDB effortless by automating the collection of coordinator and segment log files and configurations.

Usage

gpmt gp_log_collector [ -failed-segs | -c ID1,ID2,... | -hostfile FILE | -h HOST1,HOST2,... ]
        [ -start YYYY-MM-DD ] [ -end YYYY-MM-DD ]
        [ -dir PATH ] [ -segdir PATH ] [ -a ]

Description

When run with no options, gp_log_collector will collect the coordinator and standby logs for the current day. When the "-hostfile" option is set, gp_log_collector will collect coordinator, standby, and all segment instances running on the specified segment hosts.

Note: Hostnames provided through -hostfile or -h must match the hostname column in gp_segment_configuration table in the database.  

OPTIONS:
    -failed-segs       Query gp_configuration_history for list of faulted content IDs.
    -free-space        default=10  free space threshold which will abort log collection if reached
    -c                 Comma seperated list of content IDs.
    -hostfile          Read hostnames from a hostfile.
    -h                 Comma seperated list of hostnames.
    -start             Start date for logs to collect (defaults to current date)
    -end               End date for logs to collect (defaults to current date)
    -a                 Answer Yes to all prompts
    -dir               Working directory (defaults to current directory)
    -segdir            Segment temporary directory (defaults to /tmp)
    -skip-master       Skip master log collection
    -with-gptext       Collect all gptext logs within gpdb logs
    -with-gptext-only  Only Collect gptext logs

Examples

Collect coordinator logs for today:

gpmt gp_log_collector 

Collect coordinator logs and all segment hosts from hostfile for today:

gpmt gp_log_collector -hostfile ~/gpconfig/hostfile 

Collect logs for any segments marked down in gp_segment_configuration_history from 2016-03-21 to today:

gpmt gp_log_collector -failed-segs -start 2016-03-21

Collect logs for content 13, 12, and 15 for today:

gpmt gp_log_collector -c "13,12,15"

Collect logs from 'sdw1' and 'sdw4' for today:

gpmt gp_log_collector -h sdw1,sdw4

Collect logs from 'sdw2' between 2016-03-21 and 2016-03-24:

gpmt gp_log_collector -h sdw2 -start 2016-03-21 -end 2016-03-24

Collect GPDB logs between 2016-3-21 and 2016-03-24 with all GPText logs for all segments

Note: GPText log size is quite small so it doesn’t need the time params, which means that the below time won't restrict the log collection for GPText log. However, It would restrict the collections for GPDB logs.

gpmt gp_log_collector -with-gptext -start 2016-03-21 -end 2016-03-24    

Collect only GPText logs for all segments (without any GPDB logs):      

gpmt gp_log_collector -with-gptext-only

 

Failed segments 

When a segment fails over to their mirrors or simply fails without explanation, use gp_log_collector. It will scan gp_configuration_history using the time frame specified by the -start and the -end flags. Content ID logs will be collected for any identified Data Base IDs.
 

Skip coordinator 

When running gp_log_collector, the generated tarball can be very large. This is because the coordinator logs will contain a lot of information.

Note: Sometimes support will only require the segment logs. In this case, users can skip the coordinator logs, reducing the size of the resulting tar file.
 

Information to be sent to support

The tool creates an archive based on the given flags in the current working directory using the timestamp in the file name. For example, file "gp_log_collection_2016-05-16_08-50-06.tar" was created on 16th May 2016 at 08:50:06.

Note: Users must send the generated archive to support.

For informational purposes only

The archive will collect information from the following: 

1. Database (psql):

- version
- uptime   >>>> database uptime
- pg_settings
- pg_resqueue
- pg_database
- gp_segment_configuration
- gp_configuration_history

2. Segment servers (Coordinator, Primary, and Mirror):

Command outputs:

- uname-a
- sysctl-a
- psaux
- netstat-rn
- netstat-i
- lsof
- ifconfig
- free
- df-h

System files:

- /etc/redhat-release
- /etc/sysctl.conf
- /etc/sysconfig/network
- /etc/security/limits.conf 

Database related files:

- $SEG_DIR/pg_hba.conf      
- $SEG_DIR/pg_log/           
- $SEG_DIR/postgresql.conf  
- ~/gpAdminLogs 

GPText related files:

- $GPTXTHOME/lib/python/gptextlib/consts.py  // Installation configuration file
- gptext-state-D  //command result of `gptext-state -D` 
- <gptext data dir>/solr*/solr.in.sh
- <gptext data dir>/solr*/log4j.properties
- <gptext data dir>/zoo*/logs/*
- <gptext data dir>/solr*/logs/*
- commands/bash/-c_echo $PATH
- commands/bash/-c_ps -ef | grep solr
- commands/bash/-c_ps -ef | grep zookeeper

Click on the following link for a video demonstration of this KB:

 

Additional Information

Other KBs related to GPMT:

• GPMT: Greenplum Magic Tool (GPMT) and gpsupport
• analyze_session: This tool automates data collection for hung Greenplum sessions.
• packcore: This tool is for packaging core files with the required libraries and binary to make sure that these core files can be opened properly for analysis.