DS 8.x PXE Troubleshooting

Products

Deployment Solution Client Management Suite

Issue/Introduction

If PXE is not working where should I start troubleshooting?

Can you help me understand the PXE booting process better?

Environment

Deployment Solution 8.x

Resolution

The content of this KB is specifically to troubleshoot the PXE boot process was implemented in DS 8.x. We will not troubleshoot specific PXE Error codes in this KB - those are industry standard and well documented elsewhere.

The Process

Client system PXE boots (EFI or BIOS)
Site Server with NBS services receives the request for a PXE response.
Site Server sends a request to the NS/SMP for how to respond
The SMP runs a sql query to determine the appropriate action
Response is sent back to the PXE Server Service.
PXE Server formulates a custom response for that client based on options returned from the SMP

The Options

The options (response XML) returned from the SMP include Boot, Client, Image, Architecture, and if there's a Job running or not. It looks a little like this:

The possible values of each of these are:

Boot attribute

0 – Pxe Boot
1 – Automation Boot
2 – Production Boot

Client attribute

0 – Managed
1 – Predefined
2 – Unknown

Image attribute should have image name

Arch attribute

```
1 – x86
2 – x64
```

Jobrunning attribute (If there is a job scheduled for this client or not)

```
0 – Not running
1 – Running
```

Troubleshooting

Be aware of the process.

If you're getting M0F, that's not an error - that's the PXE server telling the client to go to production. Why it's being told to go to production may be an error, but the response is not, and it indicates that PXE is listening and responding. Once you know the process, you'll know where something may not be working as expected and be able to more quickly find an answer.

Know the logs.

The PXE log, which is off by default, is called SBSLog_PXE will show you most of the activity of the PXE Server service. A log level of 5 which shows you most of what you need to know and may save space. Complete logging is at either 15 or 255 which are essentially the same. This log file is in the same location as the server Service, which is, by default, under "Altiris Agent\Agents\Deployment\Task Handler\SBS," if installed.

You'll see lines like the following:

Pxe image added 'Something' for arch 'x64' [As the service starts, it literally enumerates the images it finds in the "images" folder]
PXE Server socket created, IPAddress: <PXE SERVE IP>, Port 67
PXE Server socket created, IPAddress: <PXE SERVE IP>, Port 4011 [Shows you that the PXE server service bound on the necessary ports
Not a valid IP address - Packet was filtered out. [means this packet wasn't looking for a PXE server]
Req: IP=<CLIENT IP ADDRES>, MAC=<CLIENT MAC ADDRESS>, UUID=4C4C4544-0044-4A10-8051-C4C04F315231, SerNum=DDJQ1R1 [This is a valid PXE request]
NS response parse failure [This tells us that data sent back from the NS was received, but was invalid - that is, not what we expected]

Communication Overview

The 4th bullet above shows the request from a client, which again, looks like this:

Tue Dec 31 11:20:15 - 1704 SbsPxeInterface.cpp Sbs::SbsPxeInterface::ListenDone 168 Debug Req: IP=<CLIENT IP ADDRESS>, MAC=<CLIENT MAC ADDRESS>, UUID=4C4C4544-0044-4A10-8051-C4C04F315231, SerNum=DDJQ1R1

This is the data sent back to the NS, which is used in a Query that can be seen in a Profile trace.

The ASPX will then return the data to the PXE server, and in the PXE logs, you should see an entry like the following:

Tue Jan 07 12:58:40 - 5840 SbsHttpClient.cpp Sbs::SbsHttpClient::OperationDone 136 Debug NSResp: bytes read 314, 'HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 67
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Tue, 07 Jan 2014 18:58:40 GMT

<response boot="0" client="0" image="a" arch="" jobrunning="1" />'

Known Issues in communication:

We have seen a few issues in the PXE Server log that are worth mentioning if you see them.

The 5th bullet in the list of possible entries (above under "know the logs") shows "NS Response parse failure". This indicates that data returning from the NS after a request is bad. We have seen at least two types of bad data returned, including:

Partial data. In the above example of what a good response should be, we saw on one client the header data separated from the response XML data that otherwise includes the boot, client, image, and arch data. This is because a protocol on the network was splitting the packets. If this happens, our PXE server service doesn't know how to handle it, or didn't. A point fix was released to re-assemble the packets and is expected to be part of HF3.
Denied request. In another example, the return data was a denial of service from IIS when we asked for the ASPX page. This looked somewhat like this:
Tue Jan 07 16:44:13 - 2828 SbsHttpClient.cpp Sbs::SbsHttpClient::OperationDone 117 Error NS response parse failure, bytes read 512, 'HTTP/1.1 403 Forbidden
Content-Type: text/html
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Tue, 07 Jan - 21:44:13 GMT
Content-Length: 1233
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>403 - Forbidden: Access is denied.</title>
<style type="text/css">
You'll notice that though the NS Response parse failure was read making it look like other errors, the actual DATA received shows an IIS 403 error indicating our request was denied/forbidden. The data was not actually a response from the ASPX but a response from IIS in web format with all the accompanying HTML.

Pay Attention to Configuration.

Know how to configure the PXE options in the console under Settings | Deployment | NBS General Settings. Also be aware that, for these settings to actually be working, they must be reflected in the SBConfiguration.XML file on the Site Server. That is the file that the PXE Server Service is actually paying attention to, and is located in the same directory.

Some key notes:

Respond to Managed Computers is not necessary to boot to PXE for managed computers. If you send a job to boot to PXE, and this option is disabled, the managed computer will boot to PXE. The "Respond" option is required for those locations where often a technician (or user) may purposely want to boot to PXE without asking for a job to be sent.
Respond to Unknown computers IS required if you want unknown computers to ever boot to PXE. Unless you manage them as a Predefined computer
Respond to Predefined Computers is required for Predefined computer booting.
Remember that with managed computers, you may want the default to be "Next Device" which really means to boot to production, normally. Most managed computers don't want to automatically always end up in Automation!