Errand Push-Pivotal-Account Fails during Update in Pivotal Cloud Foundry
search cancel

Errand Push-Pivotal-Account Fails during Update in Pivotal Cloud Foundry

book

Article ID: 297600

calendar_today

Updated On:

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

When attempting an upgrade from Pivotal Application Service 1.9.x to 1.10.x, errand push-pivotal-account fails.

Running errand manually also fails with the error message: 

FAILED
Start app timeout

TIP: Application must be listening on the right port. Instead of hard coding the port, use the $PORT environment variable.

Use 'cf logs pivotal-account-green --recent' for more information

Check `cf logs pivotal-account-green` output in order to see the error message below: 

2017-04-12T01:31:15.17-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.177 INFO 15 --- [ost-startStop-1] o.f.core.internal.util.VersionPrinter : Flyway 3.2.1 by Boxfuse
2017-04-12T01:31:15.37-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.378 INFO 15 --- [ost-startStop-1] o.f.c.i.dbsupport.DbSupportFactory : Database: jdbc:mysql://172.20.250.18:3306/account?verifyServerCertificate=false&useSSL=false&requireSSL=false (MySQL 5.5)
2017-04-12T01:31:15.41-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.418 INFO 15 --- [ost-startStop-1] o.f.core.internal.command.DbValidate : Validated 2 migrations (execution time 00:00.023s)
2017-04-12T01:31:15.45-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.456 INFO 15 --- [ost-startStop-1] o.f.core.internal.command.DbMigrate : Current version of schema `account`: 1
2017-04-12T01:31:15.45-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.459 ERROR 15 --- [ost-startStop-1] o.s.b.c.embedded.tomcat.TomcatStarter : Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'org.springframework.session.jdbc.config.annotation.web.http.JdbcHttpSessionConfiguration': Unsatisfied dependency expressed through method 'setCookieSerializer' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'application': Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Schema `account` contains a failed migration to version 1 !
2017-04-12T01:31:15.49-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.494 WARN 15 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.529 ERROR 15 --- [ main] o.s.boot.SpringApplication : Application startup failed
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:137) ~[spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:536) ~[spring-context-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]

Environment


Cause

The account database is corrupted.

Resolution

The current workaround is to recreate the account database.

Here are the list of steps: 

1.) In Operations Manager, open the Pivotal Application Service tile, go to credentials tab and get the credentials for the MySQL Admin, the username should be root and the password will be random. Save these as you'll need them in step 5.

2.) Log into your Operations Manager VM or whichever VM you use to execute bosh commands. Make sure you're logged in as director.

3.) Verify you're targeting your cf-*.yml deployment by running bosh deployment, if you're not targeting it, you can run bosh deployment /var/tempest/workspaces/default/deployments/cf-*.yml

4.) bosh ssh into one of your mysql VM nodes, not the proxy or replication canary

5.) Run mysql -u root -p account to connect to the account DB. If the mysql command doesn't work, you might have to use the -h localhost flag to make sure it picks up the host is the mysql VM. After you execute the command, you'll need the root password you obtained in step 1.).

6.) After you login successfully, you can run show tables; To list all tables in the account DB. You will see something similar to this:

MariaDB [account]> show tables;
+---------------------------+
| Tables_in_account |
+---------------------------+
| SPRING_SESSION |
| SPRING_SESSION_ATTRIBUTES |
| schema_version |
| zone_client |
+---------------------------+

7.) Now that you have the list, you now need to drop them by running drop table if exists <table1>,<table2>,<table3>;, place the table names in step 6 into the command.

Example:

MariaDB [account]> drop table if exists SPRING_SESSION, SPRING_SESSION_ATTRIBUTES, schema_version, zone_client;

8.) Now that you have dropped all the tabled, exit mysql, exit the mysql VM, then you're back on the Operations Manager, run bosh run errand push-pivotal-account again.

This time should be successful.


Powered by Wolken Software