Symptoms:
↵
- PKS cluster upgrade will get stuck at draining the node and it fails.
- You see messages similar to the following:
I, [2019-02-23T13:43:35.524172 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Applying VM state
I, [2019-02-23T13:45:14.166716 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Running pre-start for worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3)
I, [2019-02-23T13:45:27.234148 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Starting instance worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3)
I, [2019-02-23T13:45:27.707511 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Waiting for 10.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:45:37.708100 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 10.0 seconds
I, [2019-02-23T13:45:37.723070 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Waiting for 15.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:45:52.723546 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 15.0 seconds
I, [2019-02-23T13:45:52.740554 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Waiting for 15.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:46:07.740966 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 15.0 seconds
I, [2019-02-23T13:46:07.760749 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Waiting for 15.0 seconds to check worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) status
I, [2019-02-23T13:46:22.761471 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Checking if worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3) has been updated after 15.0 seconds
I, [2019-02-23T13:46:22.778678 #15670] [instance_update(worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3))] INFO -- DirectorJobRunner: Running post-start for worker/839d39f5-c434-43c5-ba74-bd7bc7bde149 (3)
I, [2019-02-23T13:46:49.969120 #15670] [instance_update(worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15))] INFO -- DirectorJobRunner: Updating instance worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15), changes: "stemcell, packages, configuration, job"
I, [2019-02-23T13:46:49.995086 #15670] [instance_update(worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15))] INFO -- DirectorJobRunner: Running drain for worker/d30b3ac2-1ac9-47a5-b45d-c3bcc6aed5e1 (15)