Advanced troubleshooting of Windows shares (CIFS) with VMware Data Recovery (vDR)
search cancel

Advanced troubleshooting of Windows shares (CIFS) with VMware Data Recovery (vDR)

book

Article ID: 305339

calendar_today

Updated On:

Products

VMware

Issue/Introduction

This article provides guidance on advanced troubleshooting techniques for issues with CIFS/SMB shares when used as dedupe stores on a VMware Data Recovery (vDR) appliance.

For basic CIFS troubleshooting steps, see Troubleshooting CIFS Shares issues for VMware Data Recovery (1037855).


Symptoms:



Environment

VMware Data Recovery 2.x

Resolution

Note: Ensure that the vDR appliance has 100% exclusive access to the destination dedupe datastore.

Note: Mechanisms that lock files, such as those employed by backups or virus scanning, will impact the vDR appliance and may cause undesired behavior within the appliance.

Advanced troubleshooting steps


  1. On the vDR appliance, log into the console as root and check the /var/log/messages file for any CIFS-related messages by running the command:

    # grep CIFS /var/log/messages

    Nov 30 09:46:27 localhost kernel: CIFS VFS: No response for cmd 50 mid 34376
    Nov 30 09:46:27 localhost kernel: CIFS VFS: No response for cmd 50 mid 34375
    Nov 30 09:46:48 localhost kernel: CIFS VFS: server not responding
    Nov 30 09:46:48 localhost kernel: CIFS VFS: No response for cmd 114 mid 34377


  2. Check the state of the mounted CIFS shares by running the command:

    # cat /proc/fs/cifs/DebugData

    Display Internal CIFS Data Structures for Debugging
    ---------------------------------------------------
    CIFS Version 1.60RH
    Active VFS Requests: 0
    Servers:
    1) Name: 10.16.224.189 Domain: WORKGROUP Uses: 1 OS: Windows 7 Ultimate 7601 Service Pack 1
    NOS: Windows 7 Ultimate 6.1 Capability: 0x1e3fc
    SMB session status: 1 TCP status: 1
    Local Users To Server: 1 SecMode: 0x3 Req On Wire: 0
    Shares:
    1) \\10.16.224.189\share Mounts: 1 Type: NTFS DevInfo: 0x20 Attributes: 0xc700ff
    PathComponentMax: 255 Status: 0x1 type: DISK
    MIDs:


  3. Verify that the vDR appliance can successfully connect to the Windows server providing the CIFS share by running the command:

    # smbclient -d 7 -L //SMB_HOST_OR_IP -W WorkGroup -U user%password

    Note: The -d parameter specifies the output verbosity. The value is configurable from 0 (the default) to 7 (most verbose).

    Example usage:

    # smbclient -d 7 -L //10.0.0.1 -W MyWorkGroup -U MyUser%MyPassword

    Note: If you see the session setup failed: SUCCESS - 0 error, you must uninstall all Windows Live Essentials applications from the Windows 7 machine hosting the CIFS share. For more information, see Samba bug 7577.

Debug Troubleshooting


Enabling verbose logging to disk

To aid in troubleshooting, we must ensure that all kernel messages are being logged to the system log.

To enable kernel logging:

  1. Log into a console or SSH session as root on the vDR appliance.
  2. Verify whether all kernel messages are being logged to /var/log/kernel.log by running the command:

    # grep kern /etc/syslog.conf

    If all kernel messages are being logged, output is similar to:

    kern.* /var/log/kernel.log

  3. If all kernel messages are not being logged, enable them by running:

    # echo "kern.* /var/log/kernel.log" >> /etc/syslog.conf

    Note: Verify that the content of the /etc/syslog.conf file is similar to this example by running the command:

    # cat /etc/syslog.conf

    Example content:

    kern.* /var/log/kernel.log
    *.info;mail.none;authpriv.none;cron.none /var/log/messages
    authpriv.* /var/log/secure
    mail.* -/var/log/maillog
    cron.* /var/log/cron
    *.emerg *
    uucp,news.crit /var/log/spooler
    local7.* /var/log/boot.log


  4. To apply the changes, restart the syslog service on the vDR appliance by running the command:

    # service syslog restart

VMware recommends increasing the CIFS logging level while troubleshooting CIFS-related issues. The additional logging information is then sent to /var/log/messages on the vDR appliance.

To enable increased CIFS logging:

  1. Log into a console or SSH session as root on the vDR appliance.
  2. Enable increased CIFS logging by running the command:

    # echo 3 > /proc/fs/cifs/cifsFYI

  3. Verify the value in cifsFYI by running:

    # cat /proc/fs/cifs/cifsFYI
    3


    Note: The value is configurable from 0 (the default) to 7 (most verbose).

Note: The increased CIFS logging setting will persist only until the next vDR appliance reboot. It then reverts to the default value of 0 .


Troubleshooting network connectivity between vDR and the CIFS server


When checking for CIFS issues, it is useful to check the network communication between the vDR appliance and the CIFS server. This can be achieved using a packet capture tool such as tcpdump . The vDR appliance does not include a packet capture tool, so you must install tcpdump to use these steps.

Note: This process requires that the vDR appliance has Internet connectivity. For vDR appliances without Internet connectivity, the tcpdump package (in RPM format) must be manually downloaded, uploaded to the vDR appliance, and installed using the rpm tool.

To install tcpdump on vDR appliances connected to the Internet:

  1. Log into the console of the vDR appliance as the root user.
  2. Install tcpdump on the vDR appliance by running the command:

    # yum install tcpdump

  3. The installer prompts you to confirm installing the package. Answer yes to install tcpdump .
  4. When complete, you see a message similar to:

    Installed:
    tcpdump.x86_64 14:3.9.4-15.el5
    Complete!

To capture CIFS packets (on port 445) between the CIFS server and the vDR appliance to a file, run the command:

# tcpdump -i eth0 -s0 -w /tmp/cifs-traffic.pcap host cifs_server.example.com and port 445

The output is similar to:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
<Ctrl+c>
387995 packets captured
407094 packets received by filter
19012 packets dropped by kernel


Note: Press Ctrl+c to stop packet capture.


To analyze the communication data in the capture file between the CIFS server and the vDR appliance, run the command:

# tcpdump -r /tmp/cifs-traffic.pcap | less

The output is similar to:

09:45:28.839695 IP 10.16.224.189.microsoft-ds > 10.16.225.8.53293: P 1:187(186) ack 116 win 260 <nop,nop,timestamp 163093413 61721271>
09:45:28.840064 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: . ack 187 win 827 <nop,nop,timestamp 61721273 163093413>
09:45:28.840866 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: P 116:264(148) ack 187 win 829 <nop,nop,timestamp 61721274 163093413>
09:45:28.841874 IP 10.16.224.189.microsoft-ds > 10.16.225.8.53293: P 187:405(218) ack 264 win 259 <nop,nop,timestamp 163093413 61721274>
09:45:28.842644 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: P 264:412(148) ack 405 win 829 <nop,nop,timestamp 61721276 163093413>
09:45:28.843876 IP 10.16.224.189.microsoft-ds > 10.16.225.8.53293: P 405:623(218) ack 412 win 259 <nop,nop,timestamp 163093413 61721276>
09:45:28.845164 IP 10.16.225.8.53293 > 10.16.224.189.microsoft-ds: P 412:528(116) ack 623 win 829 <nop,nop,timestamp 61721278 163093413>


Note: The capture file can be imported and analyzed in a packet capture tool such as Wireshark.





Additional Information

Troubleshooting CIFS Shares issues for VMware Data Recovery
VMware Data Recovery (vDR) での Windows 共有 (CIFS) の高度なトラブルシューティング