ERB templating of the nginx.conf file no longer supported on Staticfile Buildpack v1.6.1 and above

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

OVERVIEW 

This KB article goes over an application error that one may encounter if: 

• You have an application using a version of the Staticfile Buildpack above or at version 1.6.1
• You have an application that uses an nginx.conf file
• The application in question overrides the nginx.conf file using ERB templating. 

For reference, here is an example of how ERB may be used within an nginx.conf file. Take note of the contents between the <%= and %>.  Any content between these symbols is considered ERB templating. In our case this would be the code snippet <%= ENV["APP_ROOT"] %> and <%= ENV["PORT"] %>

worker_processes 1;
daemon off;
error_log <%= ENV["APP_ROOT"] %>/nginx/logs/error.log;
events { worker_connections 1024; }
http {
  log_format cloudfoundry '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent';
  access_log <%= ENV["APP_ROOT"] %>/nginx/logs/access.log cloudfoundry;
  default_type application/octet-stream;
  include mime.types;
  sendfile on;
  gzip on;
  tcp_nopush on;
  keepalive_timeout 30;
 
  server {
    listen <%= ENV["PORT"] %>;  
 
    location / {
      root <%= ENV["APP_ROOT"] %>/public;
      try_files $uri $uri/ /index.html =404;
      index index.html index.htm Default.htm;
    }
  }
}

ISSUE CAUSE

The reason this error occurs under the conditions described above is because starting in Staticfile buildpack version 1.6.1 (and later versions), ERB templating to override the nginx.conf file has been deprecated. Please refer to the release notes page as well as the specific commit to the Staticfile buildpack v1.6.1 release removing ERB templating support for more details. 
 

ISSUE SYMPTOMS

Symptoms of this issue arising are two-fold: 

1. You try to push an application using a Staticfile Buildpack beyond version 1.6.1, and you get a warning stating that overriding the nginx.conf is deprecated in the Staticfile buildpack, and that its recommended to use the NGINX buildpack instead. Furthermore, we notice that we continue to get the "Instances starting" message and it appears to hang on that message for quite some time: 

 -----> Configuring nginx
   **WARNING** overriding nginx.conf is deprecated and highly discouraged, as it breaks the functionality of the Staticfile and Staticfile.auth configuration directives. Please use the NGINX buildpack available at: https://github.com/cloudfoundry/nginx-buildpack
   Exit status 0
   Uploading droplet, build artifacts cache...
   Uploading droplet...
   Uploading build artifacts cache...
   Uploaded build artifacts cache (2.5M)

Waiting for app nginx-test to start...

Instances starting...
Instances starting...
Instances starting...
Instances starting...
Instances starting...
Instances starting...
Instances starting...
Instances starting...
Instances starting..

2. If we check the application logs (via cf logs $APP_NAME --recent) for the aforementioned application, we get this error specifically pointing out the ENV["APP_ROOT"] variable that was incased in ERB templating syntax. But in general, the error will show whatever variable is being templated by ERB: 

[APP/PROC/WEB/0] ERR + /home/vcap/app/start_logging.sh 2023-08-10T16:21:57.45-0400
[APP/PROC/WEB/0] ERR + nginx -p /home/vcap/app/nginx -c /home/vcap/app/nginx/conf/nginx.conf 2023-08-10T16:21:57.46-0400
[APP/PROC/WEB/0] ERR 2023/08/10 20:21:57 [emerg] 95#0: invalid log level "ENV["APP_ROOT"]" in /home/vcap/app/nginx/conf/nginx.conf:4 2023-08-10T16:22:57.49-0400
[HEALTH/0] ERR Timed out after 1m0s (30 attempts) waiting for readiness check to succeed: failed to make TCP connection to 10.255.151.201:8080: dial tcp 10.255.151.201:8080: connect: connection refused 2023-08-10T16:22:57.49-0400
[CELL/0] ERR Failed after 1m0.314s: readiness health check never passed. 2023-08-10T16:22:57.49-0400
[CELL/SSHD/0] OUT Exit status 0 2023-08-10T16:23:13.88-0400
[CELL/0] OUT Cell b87d02d7-54d6-4c9c-826f-f8b08937dd61 stopping instance 44a07a32-48d4-4bd0-6d0d-a53b 2023-08-10T16:23:13.88-0400
[CELL/0] OUT Cell b87d02d7-54d6-4c9c-826f-f8b08937dd61 destroying container for instance 44a07a32-48d4-4bd0-6d0d-a53b 2023-08-10T16:23:13.91-0400
[API/0] OUT Process has crashed with type: "web" 2023-08-10T16:23:13.94-0400
[API/0] OUT App instance exited with guid 3b48dff6-3db6-4210-9a65-a75a886636ef payload: {"instance"=>"44a07a32-48d4-4bd0-6d0d-a53b", "index"=>0, "cell_id"=>"b87d02d7-54d6-4c9c-826f-f8b08937dd61", "reason"=>"CRASHED", "exit_description"=>"Instance never healthy after 1m0.314s: Timed out after 1m0s (30 attempts) waiting for readiness check to succeed: failed to make TCP connection to 10.255.151.201:8080: dial tcp 10.255.151.201:8080: connect: connection refused; process did not exit", "crash_count"=>3, "crash_timestamp"=>1691698993874934340, "version"=>"3e44d709-bb7b-42ff-9719-efbd575c8ca9"} 2023-08-10T16:23:14.00-0400
[PROXY/0] OUT Exit status 137 2023-08-10T16:23:14.51-0400
[CELL/0] OUT Cell b87d02d7-54d6-4c9c-826f-f8b08937dd61 successfully destroyed container for instance 44a07a32-48d4-4bd0-6d0d-a53b

Environment

Product Version: 

4.x

5.x

6.x

10.x

Resolution

WORKAROUNDS

There are 2 potential workarounds for this issue: 

WORKAROUND 1
Use a Staticfile instead of an nginx.conf file when using versions of the Staticfile buildpack v1.6.1 or greater. In other words, we would remove the nginx.conf file in the app root directory, and replace it with a Staticfile.  Below is an example of what we would have in our app root folder which would consist of the Staticfile and the manifest.yml file for the application itself. 


Note that we can either use a blank Staticfile OR include some custom Staticfile parameters into the Staticfile itself which is documented here

This workaround is recommended if there is a significant overlap between the contents of the nginx.conf file that is generated by the Staticfile and our custom nginx.conf file OR if we are comfortable with using the Staticfile configuration parameters to customize the nginx.conf file. Here is documentation referencing how to customize the Staticfile. As a reference, a customized Staticfile may look something like this: 

ssi: enabled
pushstate: enabled
host_dot_files: d
# BasicAuth: enabled
force_https:true
http_strict_transport_security: true

As previously mentioned, the Staticfile buildpack itself dynamically generates an nginx.conf file via the Staticfile. We can confirm this by SSH'ing into the deployed application that has been deployed using a Staticfile, and checking for the location of the nginx.conf file in the app container. 


The full output of the dynamically generated nginx.conf file that we found in the screenshot above is shown below: 

worker_processes 1;
daemon off;
error_log /home/vcap/app/nginx/logs/error.log;
events { worker_connections 1024; }
http {
  charset utf-8;
  log_format cloudfoundry '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent';
  access_log /home/vcap/app/nginx/logs/access.log cloudfoundry;
  default_type application/octet-stream;
  include mime.types;
  sendfile on;
  gzip on;
  gzip_disable "msie6";
  gzip_comp_level 6;
  gzip_min_length 1100;
  gzip_buffers 16 8k;
  gzip_proxied any;
  gunzip on;
  gzip_static always;
  gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/xml+rss;
  gzip_vary on;
  tcp_nopush on;
  keepalive_timeout 30;
  port_in_redirect off; # Ensure that redirects don't include the internal container PORT - 8080
  server_tokens off;
  map $http_x_forwarded_host $best_host {
    "~^([^,]+),?.*$" $1;
    ''               $host;
  }
  map $http_x_forwarded_prefix $best_prefix {
    "~^([^,]+),?.*$" $1;
    ''               '';
  }
  map $http_x_forwarded_proto $best_proto {
    "~^([^,]+),?.*$" $1;
    ''               '';
  }
  server {
                listen 8080;
    server_name localhost;
    root /home/vcap/app/public;
                if ($best_proto != "https") { return 301 https://$best_host$best_prefix$request_uri; }
    location / {
        index index.html index.htm Default.htm;
    }
      location ~ /\. {
        deny all;
        return 404;
      }
  }
}

  It would be best to test to see if the dynamically generated nginx.conf file by the Staticfile works in a way so that the application itself functions properly. If not, it may be advisable to consider workaround 2 which is listed below. 


WORKAROUND 2
Use a multi-buildpack workaround involving the Ruby Buildpack and the NGINX Buildpack. This workaround is recommended if you would want to still use ERB templating within the nginx.conf file. This workaround consists of including 4 files in the app root directory. They include: 

• nginx.conf.erb
• Procfile
• A dummy nginx.conf file
• manifest.yml

As an example, here is a github repo that illustrates this workaround with all 4 files mentioned above. 

1. nginx.conf.erb
To go over the files required for this workaround, we have the nginx.conf.erb file which is interpreted by the Ruby Buildpack to perform ERB templating on that file, as it's an .erb file. Essentially, we can supply the contents of the nginx.conf.erb file with our nginx.conf that we want to be templated. Here is an example of what an nginx.conf.erb would look like: 

worker_processes 1;
daemon off;
error_log <%= ENV["APP_ROOT"] %>/nginx/logs/error.log;
events { worker_connections 1024; }
http {
  log_format cloudfoundry '$http_x_forwarded_for - $http_referer - [$time_local] "$request" $status $body_bytes_sent';
  access_log <%= ENV["APP_ROOT"] %>/nginx/logs/access.log cloudfoundry;
  default_type application/octet-stream;
  include mime.types;
  sendfile on;
  gzip on;
  tcp_nopush on;
  keepalive_timeout 30;
 
  server {
    listen <%= ENV["PORT"] %>;  
 
    location / {
      root <%= ENV["APP_ROOT"] %>/public;
      try_files $uri $uri/ /index.html =404;
      index index.html index.htm Default.htm;
    }
  }
}

NOTE: In the nginx.conf.erb file above, if we get an error in the application logs regarding mime.types, we can comment out the "include mime.types;" entry by replacing it with "#include mime.types;" via using a hashtag

2. Procfile
Next, we have a Procfile which contains the commands we need to run in the app container upon startup. In our case, we want to create the APP_ROOT directory if we have a custom app root folder within the container to store nginx logs, as well as template the nginx.conf.erb file, and generate it as an nginx.conf file. Here is an example Procfile: 

web: mkdir -p /home/vcap/app/nginx/logs && erb nginx.conf.erb | tee nginx.conf && varify -buildpack-yml-path ./buildpack.yml ./nginx.conf $HOME/modules $DEP_DIR/nginx/modules && nginx -p $PWD -c ./nginx.conf

• As seen above, the command mkdir -p /home/vcap/app/nginx/logs creates this directory in advance to prevent the app from erroring out in the application logs due to that particular directory not being found.
• The command erb nginx.conf.erb | tee nginx.conf uses ERB from the Ruby buildpack to template the contents of our nginx.conf.erb file using the erb command, and pipes it into a nginx.conf file using tee. 

• And lastly, we have the command varify -buildpack-yml-path ./buildpack.yml ./nginx.conf $HOME/modules $DEP_DIR/nginx/modules && nginx -p $PWD -c ./nginx.conf which is responsible for validating the NGINX buildpack being used. 

3. nginx.conf
Next, we have a dummy nginx.conf file that we have to provide to get around the NGINX buildpack validation process. The contents of this file would be something like below.

NOTE: If we do not provide this file in the application root, we will face an error when attempting this workaround

# This conf isn't really used.
# This is to trick the nginx_buildpack's staging::ValidateNginxConf()
# to consider this a valid app

events {}
http {
  server {
    listen {{port}};
  }
}

4. manifest.yml
Lastly, we have our application manifest.yml file that we must have in our app root. Note that we have to define the APP_ROOT variable in the manifest.yml file as it is being referenced in the nginx.conf.erb file as an environment variable in ERB syntax i.e <%= ENV["APP_ROOT"] %>. In our case, we have this defined as "/home/vcap/app": 

---
applications:
- name: nginxtest3
  env:
    APP_ROOT: "/home/vcap/app"
    FORCE_HTTPS: 'false'
  buildpacks:
  - ruby_buildpack
  - nginx_buildpack
  stack: cflinuxfs3
  processes:
  - type: web
    instances: 1
    memory: 1024M
    disk_quota: 1024M
    log-rate-limit-per-second: 16K
    health-check-type: port

Note that we MUST have the ruby_buildpack placed above the nginx_buildpack in the manifest.yml as we need to have the ERB templating run first before the nginx buildpack verification. 

From here, we can confirm that we have all of the necessary files in our app root directory: 


We can now try to push our application via CF CLI:

cf push -f ./manifest.yml

We can also check the application logs as well and see that the ERB templating of the nginx.conf.erb file has been taken care of by the ruby buildpack: 

cf logs $APP_NAME --recent

For further confirmation, if we SSH into this application, we can see that there is indeed an nginx.conf file that has been templated from our nginx.conf.erb file via the Ruby buildpack: