Troubleshooting Failed Uploads in Pivotal Web Services

Products

• Services Suite

Issue/Introduction

Symptoms:
When pushing an application with the cf cli it is possible for the process to fail during the upload or prior to the buildpack running. In these situations, it is difficult to understand what failed as the application does not completely stage and hence there is not logging available.

Environment

Cause

There are many possible causes of upload failures. This article lists some of the common ones, but other problems may occur as well.

1. Insufficient upload bandwidth, network latency that is too high, or other networking related problems. Typically resulting in an error like:

   Server error, status code: 502.

2. Your upload is too large.
3. Your application has file names that are too long.
4. Your application has a large number of small files.

Resolution

This section lists the resolutions for the problems listed in the causes section. The numbers in this section correspond to the problems listed above.

1. The run.pivotal.io servers are located in the US (AWS US-East). In most cases, if you are pushing from a location in the US and have sufficient upload bandwidth, you should not see any issues. If you are pushing from a country that is outside of the US or from a location with limited bandwidth or with a high latency network you may see some issues. This is due to the fact that you have only a finite amount of time to upload your application, currently limited at 15 minutes.
   
   If you find that you are hitting this limit, you may want to try first copying (rsync works good for this) your application to a computer or server located in the US and then pushing from that location. Because run.pivotal.io is run out on Amazon EC2, one of Amazon's free EC2 instances makes an ideal jump box for pushing your application.
2. Your upload is too large. As of the writing of this post, we test uploads up to 1G in size. If your upload is larger than this, you may want to look at breaking the application into smaller chunks or hosting large static assets elsewhere (S3, Akamai, a second application, etc.).
   
   It is also recommended that you double check the path that you are specifying with the -p argument to cf push (or the one set in your manifest.yml file). Because the cf cli will upload everything in the path that you have specified, if the path is pointing to the wrong location you may be uploading many more files than you would expect. This can both slow down your uploads and cause you to go above the upload limit.  
   
   For most applications, like Ruby and Node.js, the path should point to your project directory. For Java based applications, the path should point to your packaged application (i.e. WAR or JAR file). If there are files under the path directory that you do not want to upload, you can ignore these by placing a .cfignore file in the same directory as your manifest file.
3. Having a large number (more than a few thousand) of small files (less than 65K) can cause problems with the cf push process. Small files (less than 65k) are not subject to the same upload caching as large files. This means that small files are uploaded every time that you push the application, rather than just the first time. Because of this, having a large number of small files in your project will slow down the push process.