VCF Orchestrator UI is not accessible due to "Runaway kube-system CronJob pods exhaust node memory, causing postgres OOM crash-loops and loss of database connectivity for vco-app."
search cancel

VCF Orchestrator UI is not accessible due to "Runaway kube-system CronJob pods exhaust node memory, causing postgres OOM crash-loops and loss of database connectivity for vco-app."

book

Article ID: 450809

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

  • VCF Orchestrator UI Unavailable with error "No healthy Upstream". 
  • can see the following error on /services-logs/prelude/vco-app/file-logs/vco-server-app.log 

    YYYY-MM-DD-T INFO vco [host='vco-app-####-###' thread='main' user='' org='' trace=''] {} com.vmware.o11n.infra.DatabaseVerifier - Could not establish a connection to the database.
    YYYY-MM-DD-T ERROR vco [host='vco-app-8564475667-kjvkg' thread='main' user='' org='' trace=''] {} com.vmware.o11n.infra.InfrastructureContextConfig - Unable to validate  Database settings. Retrying in 20s!
    java.lang.RuntimeException: Could not establish a connection to the database.
            
  • kubectl get pods -n kube-system shows many Completed/Error

    prelude-network-monitor-cron pods, and/or a state-enforcement-cron pod that has been Running for days.

  • can see the following error on  /services-logs/prelude/postgres-0/file-logs/postgres.log file 

    YYYY-MM-DD UTC [136781] FATAL terminating connection due to unexpected postmaster exit
    YYYY-MM-DD UTC [136683] FATAL terminating connection due to unexpected postmaster exit
    LOG:  server process (PID 11610) was terminated by signal 9: Killed

Environment

VCF Orchestrator   9.1.0.0100

Cause

A missing backoffLimit in a kube-system CronJob causes unbounded pod multiplication, retaining hundreds of containers on the node. This accumulation traps cri-dockerd in a permanent container-polling loop, directly exhausting node memory and degrading Kubelet health. Ultimately, the severe memory pressure forces the kernel's OOM killer to repeatedly terminate the unprotected Postgres database, causing crash-loops and connectivity loss which could to lead to DB connectivity with vco-app server. 

Resolution

Resolution Steps
This patch is distributed as a single base64 blob - no file needs to be copied onto the node. Every command below decodes the same blob and only changes the trailing flag.

  1. Take a VM snapshot (without memory, quiesced) of all appliance nodes.
  2. Save the patch as a shell variable once per session. Every later step in this KB reuses it:

    PATCH64="IyEvYmluL2Jhc2gKIyBCYWNrcG9ydCBvZiBkZjZlNWFlNWFmN2NiNDIyNDhlZmY3YjhlM2M5MjIwMTRhNDUwY2MwIChWQ09QUy03MzY3MSkgdG8gOS4xLngsIHBsdXMKIyBoYXJkZW5pbmcgb2YgdGhlIHR3byBrdWJlLXN5c3RlbSBDcm9uSm9icyBhZ2FpbnN0IGxlZnRvdmVyLXBvZCBhY2N1bXVsYXRpb24uCiMKIyBXaHkgdGhpcyBtYXR0ZXJzIGZvciB0aGUgY3JpLWRvY2tlcmQgLyBkb2NrZXJkIG1lbW9yeSBpbmNpZGVudDoKIyBjcmktZG9ja2VyZCBzdGFydHMgYSBwZXJtYW5lbnQgcGVyLWNvbnRhaW5lciBnb3JvdXRpbmUgdGhhdCBhc2tzIGRvY2tlcmQgZm9yIHRoYXQKIyBjb250YWluZXIncyBSVyBsYXllciBzaXplIGV2ZXJ5IDYwIHNlY29uZHMgLSBmb3IgRVZFUlkgY29udGFpbmVyIGluIEFOWSBzdGF0ZSwKIyBpbmNsdWRpbmcgZXhpdGVkIG9uZXMgKHZlcmlmaWVkIGluIGNyaS1kb2NrZXJkIDAuMy4yMjogTGlzdENvbnRhaW5lclN0YXRzIGJ1aWxkcyBhCiMgQ29udGFpbmVyRmlsdGVyIHdpdGhvdXQgU3RhdGUsIGFuZCBMaXN0Q29udGFpbmVycyB1c2VzIEFsbDp0cnVlKS4gUmV0YWluZWQgY3JvbiBqb2IKIyBwb2RzIHRoZXJlZm9yZSBhZGQgcGVybWFuZW50IHBvbGxpbmcgbG9hZCBhZ2FpbnN0IGRvY2tlcmQuIEEgaGFuZ2luZwojIHN0YXRlLWVuZm9yY2VtZW50IGpvYiBjb21iaW5lZCB3aXRoIGJhY2tvZmZMaW1pdD02IG11bHRpcGxpZXMgbGVmdG92ZXIgcG9kcyBieSB1cCB0bwojIDd4IHBlciBmYWlsZWQgam9iLCBhbmQgMjAvMjAgaGlzdG9yeSBsaW1pdHMga2VlcCB0aGVtIGFsbCBvbiB0aGUgbm9kZS4KIwojIFRoaXMgcGF0Y2ggcmVxdWlyZXMgTk8gc2VydmljZSByZXN0YXJ0IGFuZCBOTyBkZXBsb3kuc2ggcnVuLgojCiMgVXNhZ2UgKHJ1biBhcyByb290LCBvbiBldmVyeSBub2RlKToKIyAgIC4vcGF0Y2guc2ggLS1vYnNlcnZlICAgICAjIHJlcG9ydCBvbmx5LCBjaGFuZ2Ugbm90aGluZwojICAgLi9wYXRjaC5zaCAtLWFwcGx5CiMgICAuL3BhdGNoLnNoIC0tcmV2ZXJ0CiMKIyBUdW5hYmxlcyAob3ZlcnJpZGUgdmlhIGVudmlyb25tZW50KToKIyAgIFNVQ0NFU1NfSElTVE9SWSAgIHN1Y2Nlc3NmdWxKb2JzSGlzdG9yeUxpbWl0IHRvIHNldCAoZGVmYXVsdCAzLCBwcm9kdWN0IHNoaXBzIDIwKQojICAgRkFJTEVEX0hJU1RPUlkgICAgZmFpbGVkSm9ic0hpc3RvcnlMaW1pdCB0byBzZXQgICAgIChkZWZhdWx0IDUsIHByb2R1Y3Qgc2hpcHMgMjApCgpzZXQgLUVldW8gcGlwZWZhaWwKCnByPSJWQ09QUy03MzY3MS1iYWNrcG9ydCIKbG9nRGlyPSIvdmFyL2xvZy92bXdhcmUvcHJlbHVkZSIKdGltZXN0YW1wPSIkKGRhdGUgJyslWS0lbS0lZC0lSC0lTS0lUycpIgpCQUNLVVBfRElSPSIvZGF0YS9wYXRjaC1iYWNrdXAvJHByLSR0aW1lc3RhbXAiCgpoPSIkKGhvc3RuYW1lKSIKbG9nKCkgeyBlY2hvICJbJDFdWyQoZGF0ZSAnKyVZLSVtLSVkLSVILSVNLSVTJyldWyRoXSAkMiIgPiYyOyB9CmRpZSgpIHsgbG9nIEVSUk9SICIkMSI7IGV4aXQgMTsgfQoKU1VDQ0VTU19ISVNUT1JZPSIke1NVQ0NFU1NfSElTVE9SWTotM30iCkZBSUxFRF9ISVNUT1JZPSIke0ZBSUxFRF9ISVNUT1JZOi01fSIKClNFX1NIPSIvb3B0L3NjcmlwdHMvc3RhdGVfZW5mb3JjZW1lbnQuc2giClNUQVRVU19QWT0iL29wdC9weXRob24tbW9kdWxlcy92cmFjbGkvY29tbWFuZHMvc3RhdHVzLnB5IgpDSEFSVF9TRT0iL29wdC9jaGFydHMvcHJlbHVkZS1rOHMvdGVtcGxhdGVzL3N0YXRlLWVuZm9yY2VtZW50LWNyb24ueWFtbCIKQ0hBUlRfTk09Ii9vcHQvY2hhcnRzL3ByZWx1ZGUtazhzL3RlbXBsYXRlcy9uZXR3b3JrLW1vbml0b3IueWFtbCIKTlM9Imt1YmUtc3lzdGVtIgpDUk9OUz0ic3RhdGUtZW5mb3JjZW1lbnQtY3JvbiBwcmVsdWRlLW5ldHdvcmstbW9uaXRvci1jcm9uIgoKTU9ERT0iIgp3aGlsZSBbWyAkIyAtZ3QgMCBdXTsgZG8KICAgIGNhc2UgIiQxIiBpbgogICAgICAgIC0tb2JzZXJ2ZXwtLWFwcGx5fC0tcmV2ZXJ0KSBNT0RFPSIkezEjLS19IiA7OwogICAgICAgICopIGRpZSAiVW5rbm93biBhcmd1bWVudDogJDEiIDs7CiAgICBlc2FjCiAgICBzaGlmdApkb25lCltbIC1uICIkTU9ERSIgXV0gfHwgZGllICJTcGVjaWZ5IG9uZSBvZiAtLW9ic2VydmUgfCAtLWFwcGx5IHwgLS1yZXZlcnQgKHNlZSBoZWFkZXIpLiIKW1sgIiQoaWQgLXUpIiA9PSAiMCIgXV0gfHwgZGllICJNdXN0IHJ1biBhcyByb290LiIKCm1rZGlyIC1wICIkbG9nRGlyIgpleGVjIDI+ID4odGVlIC1hICIkbG9nRGlyL3BhdGNoLSRwci0kdGltZXN0YW1wLmxvZyIpCmV4ZWMgPiYyCgprYygpIHsga3ViZWN0bCAiJEAiIDI+L2Rldi9udWxsOyB9CgpyZXBvcnRfc3RhdGUoKSB7CiAgICBsb2NhbCBuc19jdCBzYl9jdCBleF9jdCBqb2JzIGZhaWxlZCBwbGVnIG5vdHJlYWR5IGt1YmVfcmVzdGFydHMKICAgIG5zX2N0PSIkKGRvY2tlciBwcyAtYXEgLS1maWx0ZXIgJ2xhYmVsPWlvLmt1YmVybmV0ZXMuZG9ja2VyLnR5cGU9Y29udGFpbmVyJyAyPi9kZXYvbnVsbCB8IHdjIC1sIHwgdHIgLWQgJyAnKSIKICAgIHNiX2N0PSIkKGRvY2tlciBwcyAtYXEgLS1maWx0ZXIgJ2xhYmVsPWlvLmt1YmVybmV0ZXMuZG9ja2VyLnR5cGU9cG9kc2FuZGJveCcgMj4vZGV2L251bGwgfCB3YyAtbCB8IHRyIC1kICcgJykiCiAgICBleF9jdD0iJChkb2NrZXIgcHMgLWFxIC0tZmlsdGVyICdsYWJlbD1pby5rdWJlcm5ldGVzLmRvY2tlci50eXBlPWNvbnRhaW5lcicgLS1maWx0ZXIgc3RhdHVzPWV4aXRlZCAyPi9kZXYvbnVsbCB8IHdjIC1sIHwgdHIgLWQgJyAnKSIKICAgIGpvYnM9IiQoa2MgZ2V0IGpvYnMgLW4gIiROUyIgLS1uby1oZWFkZXJzIHwgd2MgLWwgfCB0ciAtZCAnICcpIgogICAgIyAuc3RhdHVzLnN1Y2NlZWRlZCBpcyAxIG9uIGEgY29tcGxldGVkIEpvYiBhbmQgPG5vbmU+IG90aGVyd2lzZS4gUmVhZGluZyB0aGUKICAgICMgZmllbGQgYnkgbmFtZSBhdm9pZHMgZGVwZW5kaW5nIG9uIGt1YmVjdGwncyBjb2x1bW4gbGF5b3V0LgogICAgZmFpbGVkPSIkKGtjIGdldCBqb2JzIC1uICIkTlMiIC1vIGN1c3RvbS1jb2x1bW5zPVM6LnN0YXR1cy5zdWNjZWVkZWQgLS1uby1oZWFkZXJzIFwKICAgICAgICAgICAgICAgIHwgZ3JlcCAtdmMgJ14xJCcgfHwgdHJ1ZSkiCiAgICBsb2cgSU5GTyAiY29udGFpbmVycyBwb2xsZWQgYnkgY3JpLWRvY2tlcmQgKG5vbi1zYW5kYm94LCBhbnkgc3RhdGUpOiAkbnNfY3QiCiAgICBsb2cgSU5GTyAiICBvZiB3aGljaCBhbHJlYWR5IGV4aXRlZCAocG9sbGVkIGJ1dCBkZWFkKSAgICAgICAgICAgICA6ICRleF9jdCIKICAgIGxvZyBJTkZPICIgIHBvZCBzYW5kYm94ZXMgKG5vdCBwb2xsZWQpICAgICAgICAgICAgICAgICAgICAgICAgICAgIDogJHNiX2N0IgogICAgbG9nIElORk8gInJldGFpbmVkIEpvYnMgaW4gJE5TOiAkam9icyAgIChub3QteWV0LWNvbXBsZXRlL2ZhaWxlZDogJGZhaWxlZCkiCiAgICBmb3IgYyBpbiAkQ1JPTlM7IGRvCiAgICAgICAgbG9nIElORk8gIiQocHJpbnRmICclLTMwcycgIiRjIik6IHN1Y2Nlc3M9JChrYyBnZXQgY3JvbmpvYiAiJGMiIC1uICIkTlMiIC1vIGpzb25wYXRoPSd7LnNwZWMuc3VjY2Vzc2Z1bEpvYnNIaXN0b3J5TGltaXR9JykgZmFpbGVkPSQoa2MgZ2V0IGNyb25qb2IgIiRjIiAtbiAiJE5TIiAtbyBqc29ucGF0aD0ney5zcGVjLmZhaWxlZEpvYnNIaXN0b3J5TGltaXR9JykgYmFja29mZkxpbWl0PSQoa2MgZ2V0IGNyb25qb2IgIiRjIiAtbiAiJE5TIiAtbyBqc29ucGF0aD0ney5zcGVjLmpvYlRlbXBsYXRlLnNwZWMuYmFja29mZkxpbWl0fScpIgogICAgICAgIGxvZyBJTkZPICIgIHJldGFpbmVkIHBvZHM6ICQoa2MgZ2V0IHBvZHMgLW4gIiROUyIgLS1uby1oZWFkZXJzIHwgZ3JlcCAtYyAiJGMiIHx8IHRydWUpIgogICAgZG9uZQogICAgIyBncmVwIC1jIHByaW50cyAwIGFuZCBleGl0cyAxIHdoZW4gdGhlcmUgaXMgbm8gbWF0Y2gsIHNvIHVzZSBgfHwgdHJ1ZWAsIG5vdAogICAgIyBgfHwgZWNobyAwYCwgd2hpY2ggd291bGQgZW1pdCB0aGUgY291bnQgdHdpY2UuCiAgICBsb2cgSU5GTyAiVkNPUFMtNzM2NzEgc2NyaXB0IHRpbWVvdXQgcHJlc2VudCAgOiAkKGdyZXAgLWMgU1RBVEVfRU5GT1JDRU1FTlRfVElNRU9VVCAiJFNFX1NIIiAyPi9kZXYvbnVsbCB8fCB0cnVlKSIKICAgIGxvZyBJTkZPICJWQ09QUy03MzY3MSBzdGF0dXMucHkgdGltZW91dCBwcmVzZW50OiAkKGdyZXAgLWMgREVGQVVMVF9DSEVDS19USU1FT1VUICIkU1RBVFVTX1BZIiAyPi9kZXYvbnVsbCB8fCB0cnVlKSIKCiAgICAjIFBMRUcva3ViZWxldCBpbnN0YWJpbGl0eSBjaGVjay4gVGhpcyBpcyB0aGUgZGVhdGgtc3BpcmFsIHNpZ25hbDogY3JpLWRvY2tlcmQKICAgICMgb3ZlcmxvYWRlZCBieSB0b28gbWFueSBwb2xsZWQgY29udGFpbmVycyBzbG93cyBkb3duIGt1YmVsZXQncyBDUkkgY2FsbHMsCiAgICAjIHdoaWNoIHB1c2hlcyBQTEVHIHJlbGlzdCBwYXN0IGl0cyB1bmhlYWx0aHkgdGhyZXNob2xkLCB3aGljaCBtYWtlcyBrdWJlbGV0CiAgICAjIG1pc3MgdGhhdCBzaG9ydCBjcm9uIHBvZHMgYWxyZWFkeSBleGl0ZWQgLSBzbyB0aGV5IGhpdCBhY3RpdmVEZWFkbGluZVNlY29uZHMKICAgICMgYW5kIGdldCByZXRyaWVkIGV2ZW4gdGhvdWdoIHRoZXkgc3VjY2VlZGVkLiBTZWUgUkVBRE1FIGZvciB0aGUgZnVsbCBjaGFpbi4KICAgIHBsZWc9IiQoam91cm5hbGN0bCAtdSBrdWJlbGV0IC0tc2luY2UgJzI0IGhvdXJzIGFnbycgLS1uby1wYWdlciAyPi9kZXYvbnVsbCB8IGdyZXAgLWMgJ1BMRUcgaXMgbm90IGhlYWx0aHknIHx8IHRydWUpIgogICAgbm90cmVhZHk9IiQoam91cm5hbGN0bCAtdSBrdWJlbGV0IC0tc2luY2UgJzI0IGhvdXJzIGFnbycgLS1uby1wYWdlciAyPi9kZXYvbnVsbCB8IGdyZXAgLWMgJ05vZGUgYmVjYW1lIG5vdCByZWFkeScgfHwgdHJ1ZSkiCiAgICBrdWJlX3Jlc3RhcnRzPSIkKHN5c3RlbWN0bCBzaG93IC1wIE5SZXN0YXJ0cyAtLXZhbHVlIGt1YmVsZXQuc2VydmljZSAyPi9kZXYvbnVsbCB8fCB0cnVlKSIKICAgIGxvZyBJTkZPICJrdWJlbGV0ICdQTEVHIGlzIG5vdCBoZWFsdGh5JyAoMjRoKSAgOiAke3BsZWc6LXVua25vd259IgogICAgbG9nIElORk8gImt1YmVsZXQgJ05vZGUgYmVjYW1lIG5vdCByZWFkeScgKDI0aCk6ICR7bm90cmVhZHk6LXVua25vd259IgogICAgbG9nIElORk8gImt1YmVsZXQgcmVzdGFydHMgc2luY2UgYm9vdCAgICAgICAgICA6ICR7a3ViZV9yZXN0YXJ0czotdW5rbm93bn0iCiAgICBpZiBbWyAiJHtwbGVnOi0wfSIgLWd0IDAgfHwgIiR7bm90cmVhZHk6LTB9IiAtZ3QgMCBdXTsgdGhlbgogICAgICAgIGxvZyBXQVJOICJQTEVHL2t1YmVsZXQgaW5zdGFiaWxpdHkgZGV0ZWN0ZWQgLSBrdWJlY3RsIGNhbGxzIGJlbG93IG1heSBiZSBzbG93IG9yIgogICAgICAgIGxvZyBXQVJOICJuZWVkIGEgcmV0cnkuIFRoYXQgc2xvd25lc3MgaXMgaXRzZWxmIGNvbmZpcm1hdGlvbiBvZiB0aGUgZGlhZ25vc2lzLCBub3QiCiAgICAgICAgbG9nIFdBUk4gImEgcmVhc29uIHRvIGhvbGQgb2ZmOiBhcHBseWluZyB0aGlzIHBhdGNoIHJlZHVjZXMgdGhlIGNvbnRhaW5lciBjb3VudCIKICAgICAgICBsb2cgV0FSTiAiZHJpdmluZyB0aGUgaW5zdGFiaWxpdHkuIgogICAgZmkKfQoKIyA9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09IE9CU0VSVkUgPT0KaWYgW1sgIiRNT0RFIiA9PSAib2JzZXJ2ZSIgXV07IHRoZW4KICAgIGxvZyBJTkZPICI9PT0gb2JzZXJ2ZSBtb2RlOiBubyBjaGFuZ2VzIHdpbGwgYmUgbWFkZSA9PT0iCiAgICByZXBvcnRfc3RhdGUKICAgIGxvZyBJTkZPICJXb3VsZCBzZXQgaGlzdG9yeSBsaW1pdHMgdG8gJFNVQ0NFU1NfSElTVE9SWS8kRkFJTEVEX0hJU1RPUlkgYW5kIGJhY2tvZmZMaW1pdD0wIG9uOiAkQ1JPTlMiCiAgICBleGl0IDAKZmkKCiMgPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0gUkVWRVJUID09CmlmIFtbICIkTU9ERSIgPT0gInJldmVydCIgXV07IHRoZW4KICAgIGxvZyBJTkZPICI9PT0gcmV2ZXJ0aW5nICRwciA9PT0iCiAgICBmb3IgYyBpbiAkQ1JPTlM7IGRvCiAgICAgICAga2MgcGF0Y2ggY3JvbmpvYiAiJGMiIC1uICIkTlMiIC0tdHlwZSBtZXJnZSBcCiAgICAgICAgICAgIC1wICd7InNwZWMiOnsic3VjY2Vzc2Z1bEpvYnNIaXN0b3J5TGltaXQiOjIwLCJmYWlsZWRKb2JzSGlzdG9yeUxpbWl0IjoyMCwiam9iVGVtcGxhdGUiOnsic3BlYyI6eyJiYWNrb2ZmTGltaXQiOjZ9fX19JyBcCiAgICAgICAgICAgICYmIGxvZyBJTkZPICIkYyByZXN0b3JlZCB0byAyMC8yMCwgYmFja29mZkxpbWl0PTYiIFwKICAgICAgICAgICAgfHwgbG9nIFdBUk4gImNvdWxkIG5vdCByZXN0b3JlICRjIgogICAgZG9uZQogICAgbG9nIFdBUk4gIkZpbGUgY2hhbmdlcyBhcmUgTk9UIGF1dG8tcmV2ZXJ0ZWQuIFJlc3RvcmUgdGhlbSBmcm9tIHRoZSBuZXdlc3QgYmFja3VwOiIKICAgIGxzIC0xZCAvZGF0YS9wYXRjaC1iYWNrdXAvIiRwciItKiAyPi9kZXYvbnVsbCB8fCB0cnVlCiAgICBsb2cgV0FSTiAiICBjcCA8YmFja3VwPi9zdGF0ZV9lbmZvcmNlbWVudC5zaCAkU0VfU0giCiAgICBsb2cgV0FSTiAiICBjcCA8YmFja3VwPi9zdGF0dXMucHkgJFNUQVRVU19QWSIKICAgIGxvZyBXQVJOICIgIGNwIDxiYWNrdXA+L3N0YXRlLWVuZm9yY2VtZW50LWNyb24ueWFtbCAkQ0hBUlRfU0UiCiAgICBsb2cgV0FSTiAiICBjcCA8YmFja3VwPi9uZXR3b3JrLW1vbml0b3IueWFtbCAkQ0hBUlRfTk0iCiAgICBsb2cgSU5GTyAiTm8gc2VydmljZSByZXN0YXJ0IGlzIG5lZWRlZCBhZnRlciByZXN0b3JpbmcuIgogICAgZXhpdCAwCmZpCgojID09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PSBBUFBMWSA9PQpsb2cgSU5GTyAiPT09IGFwcGx5aW5nICRwciA9PT0iCmxvZyBJTkZPICItLS0gc3RhdGUgQkVGT1JFIC0tLSIKcmVwb3J0X3N0YXRlCm1rZGlyIC1wICIkQkFDS1VQX0RJUiIKZm9yIGYgaW4gIiRTRV9TSCIgIiRTVEFUVVNfUFkiICIkQ0hBUlRfU0UiICIkQ0hBUlRfTk0iOyBkbwogICAgW1sgLWYgIiRmIiBdXSAmJiBjcCAtYSAiJGYiICIkQkFDS1VQX0RJUi8kKGJhc2VuYW1lICIkZiIpIgpkb25lCmxvZyBJTkZPICJiYWNrZWQgdXAgb3JpZ2luYWxzIHRvICRCQUNLVVBfRElSIgoKIyAtLS0gMS4gaG9zdCBzY3JpcHQ6IHRpbWVvdXRzIHNvIHN0YXRlIGVuZm9yY2VtZW50IGNhbm5vdCBoYW5nIGZvcmV2ZXIgLS0tLS0tLS0KaWYgZ3JlcCAtcSAnU1RBVEVfRU5GT1JDRU1FTlRfVElNRU9VVCcgIiRTRV9TSCIgMj4vZGV2L251bGw7IHRoZW4KICAgIGxvZyBJTkZPICIkU0VfU0ggYWxyZWFkeSBwYXRjaGVkOyBza2lwcGluZy4iCmVsc2UKICAgIHJjPTA7IHNldCArZQogICAgcGF0Y2ggLS1uby1iYWNrdXAtaWYtbWlzbWF0Y2ggLXMgLWYgLS1yZWplY3QtZmlsZT0tIC1wMSAtZCAvIDw8ICdTRV9QQVRDSF9FT0YnCi0tLSAvb3B0L3NjcmlwdHMvc3RhdGVfZW5mb3JjZW1lbnQuc2gKKysrIC9vcHQvc2NyaXB0cy9zdGF0ZV9lbmZvcmNlbWVudC5zaApAQCAtMTYsOCArMTYsMTAgQEAgZnVuY3Rpb24gbG9nX2Vycm9yKCkgewogICBsb2cgIiQxIiAiRVJST1IiCiB9CiAKK1NUQVRFX0VORk9SQ0VNRU5UX1RJTUVPVVQ9MzAwCisKIGxvZ19pbmZvICJDaGVja2luZyBmaXJzdCBib290IHN0YXR1cyIKLXZyYWNsaSBzdGF0dXMgZmlyc3QtYm9vdCB8fCBleGl0IDEKK3ZyYWNsaSBzdGF0dXMgZmlyc3QtYm9vdCAtLXdhaXQgJHtTVEFURV9FTkZPUkNFTUVOVF9USU1FT1VUfSB8fCBleGl0IDEKIAogbG9nX2luZm8gIkFwcGx5aW5nIFBSRUxVREVfUFJPWFkgaXB0YWJsZXMgcnVsZXMiCiB2cmFjbGkgY2x1c3RlciBleGVjIC0tIC9vcHQvc2NyaXB0cy9pcHRhYmxlcy9yZWxvYWQtcHJveHkKQEAgLTI1LDcgKzI3LDcgQEAgdnJhY2xpIGNsdXN0ZXIgZXhlYyAtLSAvb3B0L3NjcmlwdHMvaXB0YWJsZXMvcmVsb2FkLXByb3h5CiBsb2dfaW5mbyAiU2NhbGluZyBwb2RzIgogdnJhY2xpIGNsdXN0ZXIgc2NhbGUtcG9kcwogCi1pZiB2cmFjbGkgc3RhdHVzIGRlcGxveTsgdGhlbgoraWYgdnJhY2xpIHN0YXR1cyBkZXBsb3kgLS13YWl0ICR7U1RBVEVfRU5GT1JDRU1FTlRfVElNRU9VVH07IHRoZW4KIAlsb2dfaW5mbyAiSGFuZGxlIGxpY2Vuc2luZy4iCiAgICAgW1sgLXggL29wdC9zY3JpcHRzL3NldF9saWNlbnNlX2ZsYWdzLnNoIF1dICYmIC9vcHQvc2NyaXB0cy9zZXRfbGljZW5zZV9mbGFncy5zaCB8fCBleGl0IDEKIApTRV9QQVRDSF9FT0YKICAgIHJjPSQ/OyBzZXQgLWUKICAgIGlmIFtbICRyYyAtZXEgMCBdXSAmJiBiYXNoIC1uICIkU0VfU0giOyB0aGVuCiAgICAgICAgbG9nIElORk8gInBhdGNoZWQgJFNFX1NIICh0aW1lb3V0cyBvbiB2cmFjbGkgc3RhdHVzIGZpcnN0LWJvb3QgLyBkZXBsb3kpIgogICAgZWxzZQogICAgICAgIGxvZyBFUlJPUiAiZmFpbGVkIHRvIHBhdGNoICRTRV9TSCAocmM9JHJjKSAtIHJlc3RvcmluZyBvcmlnaW5hbCIKICAgICAgICBjcCAtYSAiJEJBQ0tVUF9ESVIvJChiYXNlbmFtZSAiJFNFX1NIIikiICIkU0VfU0giCiAgICAgICAgZGllICJhYm9ydGluZzsgbm8gb3RoZXIgZmlsZSBjaGFuZ2VzIHdlcmUgbWFkZSB5ZXQiCiAgICBmaQpmaQoKIyAtLS0gMi4gdnJhY2xpIHN0YXR1czogYm91bmQgdGhlIGNoZWNrcyBldmVuIHdoZW4gLS13YWl0IGlzIG5vdCBwYXNzZWQgLS0tLS0tLS0KIyBEZWZlbmNlIGluIGRlcHRoLiBTdGVwIDEgYWxyZWFkeSBwYXNzZXMgLS13YWl0IGV4cGxpY2l0bHksIHNvIGEgZmFpbHVyZSBoZXJlIGlzCiMgbm90IGZhdGFsIHRvIHRoZSBmaXguCmlmIGdyZXAgLXEgJ0RFRkFVTFRfQ0hFQ0tfVElNRU9VVCcgIiRTVEFUVVNfUFkiIDI+L2Rldi9udWxsOyB0aGVuCiAgICBsb2cgSU5GTyAiJFNUQVRVU19QWSBhbHJlYWR5IHBhdGNoZWQ7IHNraXBwaW5nLiIKZWxzZQogICAgcmM9MDsgc2V0ICtlCiAgICBwYXRjaCAtLW5vLWJhY2t1cC1pZi1taXNtYXRjaCAtcyAtZiAtLXJlamVjdC1maWxlPS0gLXAxIC1kIC8gPDwgJ1NUX1BBVENIX0VPRicKLS0tIC9vcHQvcHl0aG9uLW1vZHVsZXMvdnJhY2xpL2NvbW1hbmRzL3N0YXR1cy5weQorKysgL29wdC9weXRob24tbW9kdWxlcy92cmFjbGkvY29tbWFuZHMvc3RhdHVzLnB5CkBAIC0yNiw2ICsyNiw5IEBAIGZyb20gY3R5cGVzIGltcG9ydCBjX2NoYXJfcAogQHZyYWNsaV9jb21tYW5kCiBjbGFzcyBTdGF0dXNDb21tYW5kcyhCYXNlQ29tbWFuZCk6CiAgICAgbmFtZSA9ICJzdGF0dXMiCisgICAgREVGQVVMVF9DSEVDS19USU1FT1VUID0gMTIwCisgICAgU1VCUFJPQ0VTU19USU1FT1VUID0gNjAKKwogICAgIGRlZiBfX2luaXRfXyhzZWxmLCBsb2dnZXI6IGxvZ2dpbmcuTG9nZ2VyLCBwYXJzZXJfZ3JvdXA6IGFyZ3BhcnNlLkFjdGlvbik6CiAgICAgICAgIHN1cGVyKCkuX19pbml0X18obG9nZ2VyKQogICAgICAgICBzZWxmLnZhY29uZmlnID0gVkFDb25maWcoKQpAQCAtMTMxLDI0ICsxMzQsMjcgQEAgY2xhc3MgU3RhdHVzQ29tbWFuZHMoQmFzZUNvbW1hbmQpOgogICAgICAgICAgICAgIyBTdGFydCBpbiBiYWNrZ3JvdW5kCiAgICAgICAgICAgICBjaGVja19yZXN1bHQgPSBtdWx0aXByb2Nlc3NpbmcuVmFsdWUoImkiLCAxMjQsIGxvY2s9RmFsc2UpCiAgICAgICAgICAgICBtYW5hZ2VyID0gbXVsdGlwcm9jZXNzaW5nLk1hbmFnZXIoKQotICAgICAgICAgICAgY2hlY2tfb3V0cHV0ID0gbWFuYWdlci5WYWx1ZShjX2NoYXJfcCwgIiIpCi0gICAgICAgICAgICBjaGVjayA9IG11bHRpcHJvY2Vzc2luZy5Qcm9jZXNzKAotICAgICAgICAgICAgICAgIHRhcmdldD1jaGVja19oYW5kbGVyLCBhcmdzPVtjaGVja19yZXN1bHQsIGNoZWNrX291dHB1dF0pCi0gICAgICAgICAgICBjaGVjay5zdGFydCgpCisgICAgICAgICAgICB0cnk6CisgICAgICAgICAgICAgICAgY2hlY2tfb3V0cHV0ID0gbWFuYWdlci5WYWx1ZShjX2NoYXJfcCwgIiIpCisgICAgICAgICAgICAgICAgY2hlY2sgPSBtdWx0aXByb2Nlc3NpbmcuUHJvY2VzcygKKyAgICAgICAgICAgICAgICAgICAgdGFyZ2V0PWNoZWNrX2hhbmRsZXIsIGFyZ3M9W2NoZWNrX3Jlc3VsdCwgY2hlY2tfb3V0cHV0XSkKKyAgICAgICAgICAgICAgICBjaGVjay5zdGFydCgpCiAKLSAgICAgICAgICAgICMgV2FpdCB1bnRpbCBmaW5pc2ggb3IgdGltZW91dCBleHBpcmUuIFdhaXQgaW5maW5pdGVseSBpZiB3YWl0IHRpbWUgbm90IHNldAotICAgICAgICAgICAgY2hlY2suam9pbihOb25lIGlmIHRpbWVfd2FpdCA9PSAwIGVsc2UgdGltZV9sZWZ0KQorICAgICAgICAgICAgICAgIGpvaW5fdGltZW91dCA9IHRpbWVfbGVmdCBpZiB0aW1lX3dhaXQgPiAwIGVsc2Ugc2VsZi5ERUZBVUxUX0NIRUNLX1RJTUVPVVQKKyAgICAgICAgICAgICAgICBjaGVjay5qb2luKGpvaW5fdGltZW91dCkKIAotICAgICAgICAgICAgIyBSZXBvcnQgZmFpbHVyZSBpZiBjaGVja3MgY29tcGxldGlvbiBoYXMgdGltZWQgb3V0LgotICAgICAgICAgICAgaWYgY2hlY2suaXNfYWxpdmUoKToKLSAgICAgICAgICAgICAgICBjaGVjay50ZXJtaW5hdGUoKQotICAgICAgICAgICAgICAgIGNoZWNrLmpvaW4oKQorICAgICAgICAgICAgICAgICMgUmVwb3J0IGZhaWx1cmUgaWYgY2hlY2tzIGNvbXBsZXRpb24gaGFzIHRpbWVkIG91dC4KKyAgICAgICAgICAgICAgICBpZiBjaGVjay5pc19hbGl2ZSgpOgorICAgICAgICAgICAgICAgICAgICBjaGVjay50ZXJtaW5hdGUoKQorICAgICAgICAgICAgICAgICAgICBjaGVjay5qb2luKCkKIAotICAgICAgICAgICAgICAgIHAuc3RhdHVzX2NvZGUgPSAxMjQKLSAgICAgICAgICAgICAgICByZXR1cm4KKyAgICAgICAgICAgICAgICAgICAgcC5zdGF0dXNfY29kZSA9IDEyNAorICAgICAgICAgICAgICAgICAgICByZXR1cm4KIAotICAgICAgICAgICAgcmVzdWx0ID0gY2hlY2tfcmVzdWx0LnZhbHVlCi0gICAgICAgICAgICBvdXRwdXQgPSBjaGVja19vdXRwdXQudmFsdWUKKyAgICAgICAgICAgICAgICByZXN1bHQgPSBjaGVja19yZXN1bHQudmFsdWUKKyAgICAgICAgICAgICAgICBvdXRwdXQgPSBjaGVja19vdXRwdXQudmFsdWUKKyAgICAgICAgICAgIGZpbmFsbHk6CisgICAgICAgICAgICAgICAgbWFuYWdlci5zaHV0ZG93bigpCiAKICAgICAgICAgICAgICMgQnJlYWsgb24gc3VjY2Vzc2Z1bCBjaGVjawogICAgICAgICAgICAgaWYgcmVzdWx0ID09IDA6CkBAIC0yMDksNyArMjE1LDEzIEBAIGNsYXNzIFN0YXR1c0NvbW1hbmRzKEJhc2VDb21tYW5kKToKICAgICAgICAgXQogCiAgICAgICAgIGZvciBpLCBjbWQgaW4gZW51bWVyYXRlKGNvbW1hbmRzKToKLSAgICAgICAgICAgIHJlcyA9IHN1YnByb2Nlc3MucnVuKGNtZCwgc3Rkb3V0PXN1YnByb2Nlc3MuUElQRSwgc3RkZXJyPXN1YnByb2Nlc3MuUElQRSwgdW5pdmVyc2FsX25ld2xpbmVzPSJ1dGYtOCIsIHNoZWxsPVRydWUpCisgICAgICAgICAgICB0cnk6CisgICAgICAgICAgICAgICAgcmVzID0gc3VicHJvY2Vzcy5ydW4oY21kLCBzdGRvdXQ9c3VicHJvY2Vzcy5QSVBFLCBzdGRlcnI9c3VicHJvY2Vzcy5QSVBFLAorICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHVuaXZlcnNhbF9uZXdsaW5lcz0idXRmLTgiLCBzaGVsbD1UcnVlLCB0aW1lb3V0PXNlbGYuU1VCUFJPQ0VTU19USU1FT1VUKQorICAgICAgICAgICAgZXhjZXB0IHN1YnByb2Nlc3MuVGltZW91dEV4cGlyZWQ6CisgICAgICAgICAgICAgICAgb3V0cHV0LnZhbHVlID0gIk5vdCByZWFkeSIKKyAgICAgICAgICAgICAgICByZXN1bHQudmFsdWUgPSAxCisgICAgICAgICAgICAgICAgcmV0dXJuIHJlc3VsdC52YWx1ZQogICAgICAgICAgICAgaWYgcmVzLnJldHVybmNvZGUgPiAwOgogICAgICAgICAgICAgICAgIG91dHB1dC52YWx1ZSA9ICJOb3QgcmVhZHkiCiAgICAgICAgICAgICAgICAgcmVzdWx0LnZhbHVlID0gMQpAQCAtMjI5LDcgKzI0MSwxMyBAQCBjbGFzcyBTdGF0dXNDb21tYW5kcyhCYXNlQ29tbWFuZCk6CiAgICAgICAgIF0KIAogICAgICAgICBmb3IgaSwgY21kIGluIGVudW1lcmF0ZShjb21tYW5kcyk6Ci0gICAgICAgICAgICByZXMgPSBzdWJwcm9jZXNzLnJ1bihjbWQsIHN0ZG91dD1zdWJwcm9jZXNzLlBJUEUsIHN0ZGVycj1zdWJwcm9jZXNzLlBJUEUsIHVuaXZlcnNhbF9uZXdsaW5lcz0idXRmLTgiKQorICAgICAgICAgICAgdHJ5OgorICAgICAgICAgICAgICAgIHJlcyA9IHN1YnByb2Nlc3MucnVuKGNtZCwgc3Rkb3V0PXN1YnByb2Nlc3MuUElQRSwgc3RkZXJyPXN1YnByb2Nlc3MuUElQRSwKKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB1bml2ZXJzYWxfbmV3bGluZXM9InV0Zi04IiwgdGltZW91dD1zZWxmLlNVQlBST0NFU1NfVElNRU9VVCkKKyAgICAgICAgICAgIGV4Y2VwdCBzdWJwcm9jZXNzLlRpbWVvdXRFeHBpcmVkOgorICAgICAgICAgICAgICAgIG91dHB1dC52YWx1ZT0iZmFpbGVkIGNoZWNrIHt9OiB0aW1lZCBvdXQgYWZ0ZXIge31zIi5mb3JtYXQoIiAiLmpvaW4oY21kKSwgc2VsZi5TVUJQUk9DRVNTX1RJTUVPVVQpCisgICAgICAgICAgICAgICAgcmVzdWx0LnZhbHVlPWkrMQorICAgICAgICAgICAgICAgIHJldHVybiByZXN1bHQudmFsdWUKICAgICAgICAgICAgIGlmIHJlcy5yZXR1cm5jb2RlID4gMDoKICAgICAgICAgICAgICAgICBvdXRwdXQudmFsdWU9ImZhaWxlZCBjaGVjayB7fTogZXhpdCBjb2RlOiBbe31dLCBzdGRvdXQ6IFt7fV0sIHN0ZGVycjogW3t9XSIuZm9ybWF0KCIgIi5qb2luKGNtZCksIHJlcy5yZXR1cm5jb2RlLCByZXMuc3Rkb3V0LCByZXMuc3RkZXJyKQogICAgICAgICAgICAgICAgIHJlc3VsdC52YWx1ZT1pKzEKQEAgLTI1MSw3ICsyNjksMTQgQEAgY2xhc3MgU3RhdHVzQ29tbWFuZHMoQmFzZUNvbW1hbmQpOgogICAgICAgICBjbWQgPSBbInZyYWNsaSIsICJoZWFsdGgiLCAiY2hlY2siLCAiLS1jaGVja3MtbG9jYXRpb24iLCAiL29wdC9oZWFsdGgvZGVmYXVsdC1zcGVjcy8iLCAiLS1qc29uIiwgIi0tcHJvZmlsZSIsICJkZWZhdWx0Il0KIAogICAgICAgICBwcm9jZXNzID0gc3VicHJvY2Vzcy5Qb3BlbihjbWQsIHN0ZG91dD1zdWJwcm9jZXNzLlBJUEUsIHN0ZGVycj1zdWJwcm9jZXNzLlBJUEUsIHVuaXZlcnNhbF9uZXdsaW5lcz0idXRmLTgiKQotICAgICAgICBjbWRfb3V0cHV0ID0gcHJvY2Vzcy5jb21tdW5pY2F0ZSgpWzBdCisgICAgICAgIHRyeToKKyAgICAgICAgICAgIGNtZF9vdXRwdXQgPSBwcm9jZXNzLmNvbW11bmljYXRlKHRpbWVvdXQ9c2VsZi5TVUJQUk9DRVNTX1RJTUVPVVQpWzBdCisgICAgICAgIGV4Y2VwdCBzdWJwcm9jZXNzLlRpbWVvdXRFeHBpcmVkOgorICAgICAgICAgICAgcHJvY2Vzcy5raWxsKCkKKyAgICAgICAgICAgIHByb2Nlc3MuY29tbXVuaWNhdGUoKQorICAgICAgICAgICAgb3V0cHV0LnZhbHVlID0gIk5vdCBIZWFsdGh5IgorICAgICAgICAgICAgcmVzdWx0LnZhbHVlID0gMTI0CisgICAgICAgICAgICByZXR1cm4gcmVzdWx0LnZhbHVlCiAgICAgICAgIGlmIHByb2Nlc3MucmV0dXJuY29kZSA+IDAgb3IgImZhaWxlZCIgaW4gY21kX291dHB1dDoKICAgICAgICAgICAgIG91dHB1dC52YWx1ZSA9ICJOb3QgSGVhbHRoeSIKICAgICAgICAgICAgIHJlc3VsdC52YWx1ZSA9IHByb2Nlc3MucmV0dXJuY29kZQpTVF9QQVRDSF9FT0YKICAgIHJjPSQ/OyBzZXQgLWUKICAgIGlmIFtbICRyYyAtZXEgMCBdXSAmJiBweXRob24zIC1jICJpbXBvcnQgYXN0LHN5czsgYXN0LnBhcnNlKG9wZW4oJyRTVEFUVVNfUFknKS5yZWFkKCkpIjsgdGhlbgogICAgICAgIGxvZyBJTkZPICJwYXRjaGVkICRTVEFUVVNfUFkgKGNoZWNrICsgc3VicHJvY2VzcyB0aW1lb3V0cykiCiAgICBlbHNlCiAgICAgICAgbG9nIFdBUk4gImNvdWxkIG5vdCBjbGVhbmx5IHBhdGNoICRTVEFUVVNfUFkgKHJjPSRyYykgLSByZXN0b3JpbmcgaXQgYW5kIGNvbnRpbnVpbmcuIgogICAgICAgIGxvZyBXQVJOICJUaGUgc2NyaXB0LWxldmVsIHRpbWVvdXQgZnJvbSBzdGVwIDEgaXMgdGhlIGxvYWQtYmVhcmluZyBwYXJ0IG9mIHRoaXMgZml4LiIKICAgICAgICBjcCAtYSAiJEJBQ0tVUF9ESVIvJChiYXNlbmFtZSAiJFNUQVRVU19QWSIpIiAiJFNUQVRVU19QWSIKICAgIGZpCmZpCgojIC0tLSAzLiBsaXZlIENyb25Kb2JzOiBzdG9wIHBvZCBtdWx0aXBsaWNhdGlvbiBhbmQgc2hvcnRlbiByZXRlbnRpb24gLS0tLS0tLS0tCiMgQWZmZWN0cyBmdXR1cmUgSm9icyBvbmx5OyBub3RoaW5nIHJ1bm5pbmcgaXMgZGlzdHVyYmVkLiBMb3dlcmluZyB0aGUgaGlzdG9yeQojIGxpbWl0cyBhbHNvIG1ha2VzIHRoZSBqb2IgY29udHJvbGxlciBnYXJiYWdlLWNvbGxlY3QgdGhlIGV4aXN0aW5nIHN1cnBsdXMuCmZvciBjIGluICRDUk9OUzsgZG8KICAgIGlmIGtjIHBhdGNoIGNyb25qb2IgIiRjIiAtbiAiJE5TIiAtLXR5cGUgbWVyZ2UgXAogICAgICAgIC1wICJ7XCJzcGVjXCI6e1wic3VjY2Vzc2Z1bEpvYnNIaXN0b3J5TGltaXRcIjokU1VDQ0VTU19ISVNUT1JZLFwiZmFpbGVkSm9ic0hpc3RvcnlMaW1pdFwiOiRGQUlMRURfSElTVE9SWSxcImpvYlRlbXBsYXRlXCI6e1wic3BlY1wiOntcImJhY2tvZmZMaW1pdFwiOjB9fX19IjsgdGhlbgogICAgICAgIGxvZyBJTkZPICJwYXRjaGVkIENyb25Kb2IgJGMgLT4gaGlzdG9yeSAkU1VDQ0VTU19ISVNUT1JZLyRGQUlMRURfSElTVE9SWSwgYmFja29mZkxpbWl0PTAiCiAgICBlbHNlCiAgICAgICAgbG9nIFdBUk4gImNvdWxkIG5vdCBwYXRjaCBDcm9uSm9iICRjIChpcyB0aGlzIHRoZSBub2RlIHJ1bm5pbmcgdGhlIGNvbnRyb2wgcGxhbmU/KSIKICAgIGZpCmRvbmUKCiMgLS0tIDQuIG9uLWRpc2sgY2hhcnRzIHNvIHRoZSBDcm9uSm9iIHNldHRpbmdzIHN1cnZpdmUgYSBkZXBsb3kuc2ggLS0tLS0tLS0tLS0KZm9yIGYgaW4gIiRDSEFSVF9TRSIgIiRDSEFSVF9OTSI7IGRvCiAgICBbWyAtZiAiJGYiIF1dIHx8IHsgbG9nIFdBUk4gIiRmIG5vdCBmb3VuZDsgc2tpcHBpbmciOyBjb250aW51ZTsgfQogICAgcmM9MDsgc2V0ICtlCiAgICBweXRob24zIC0gIiRmIiAiJFNVQ0NFU1NfSElTVE9SWSIgIiRGQUlMRURfSElTVE9SWSIgPDwgJ1BZRU9GJwppbXBvcnQgcmUsIHN5cwpwYXRoLCBzdWNjLCBmYWlsID0gc3lzLmFyZ3ZbMV0sIHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXQpzcmMgPSBvcGVuKHBhdGgpLnJlYWQoKQpvcmlnID0gc3JjCnNyYyA9IHJlLnN1YihyJ14oXHMqKXN1Y2Nlc3NmdWxKb2JzSGlzdG9yeUxpbWl0OlxzKlxkK1xzKiQnLAogICAgICAgICAgICAgbGFtYmRhIG06ICIlc3N1Y2Nlc3NmdWxKb2JzSGlzdG9yeUxpbWl0OiAlcyIgJSAobS5ncm91cCgxKSwgc3VjYyksIHNyYywgZmxhZ3M9cmUuTSkKc3JjID0gcmUuc3ViKHInXihccyopZmFpbGVkSm9ic0hpc3RvcnlMaW1pdDpccypcZCtccyokJywKICAgICAgICAgICAgIGxhbWJkYSBtOiAiJXNmYWlsZWRKb2JzSGlzdG9yeUxpbWl0OiAlcyIgJSAobS5ncm91cCgxKSwgZmFpbCksIHNyYywgZmxhZ3M9cmUuTSkKIyBFbnN1cmUgYmFja29mZkxpbWl0OiAwIHNpdHMgdW5kZXIgam9iVGVtcGxhdGUuc3BlYwppZiBub3QgcmUuc2VhcmNoKHInXlxzKmJhY2tvZmZMaW1pdDonLCBzcmMsIGZsYWdzPXJlLk0pOgogICAgc3JjLCBuID0gcmUuc3VibihyJyheICBqb2JUZW1wbGF0ZTpcbiAgICBzcGVjOlxuKSg/PSAgICAgICknLAogICAgICAgICAgICAgICAgICAgICByJ1wxICAgICAgYmFja29mZkxpbWl0OiAwXG4nLCBzcmMsIGNvdW50PTEsIGZsYWdzPXJlLk0pCiAgICBpZiBuID09IDA6CiAgICAgICAgc3lzLnN0ZGVyci53cml0ZSgiQkFDS09GRi1BTkNIT1ItTk9ULUZPVU5EXG4iKTsgc3lzLmV4aXQoMykKaWYgc3JjID09IG9yaWc6CiAgICBwcmludCgibm8gY2hhbmdlIG5lZWRlZCIpOyBzeXMuZXhpdCgwKQpvcGVuKHBhdGgsICJ3Iikud3JpdGUoc3JjKQpwcmludCgidXBkYXRlZCIpClBZRU9GCiAgICByYz0kPzsgc2V0IC1lCiAgICBpZiBbWyAkcmMgLWVxIDAgXV07IHRoZW4KICAgICAgICBsb2cgSU5GTyAidXBkYXRlZCBjaGFydCAkZiIKICAgIGVsc2UKICAgICAgICBsb2cgV0FSTiAiY291bGQgbm90IHVwZGF0ZSBjaGFydCAkZiAocmM9JHJjKTsgbGl2ZSBDcm9uSm9iIGlzIHN0aWxsIHBhdGNoZWQsIgogICAgICAgIGxvZyBXQVJOICJidXQgYSBmdXR1cmUgZGVwbG95LnNoIHdvdWxkIHJlc2V0IGl0LiBFZGl0IGl0IGJ5IGhhbmQuIgogICAgZmkKZG9uZQoKbG9nIElORk8gIi0tLSBzdGF0ZSBBRlRFUiAtLS0iCnJlcG9ydF9zdGF0ZQpsb2cgSU5GTyAiPT09ICRwciBhcHBsaWVkIC0gbm8gc2VydmljZSByZXN0YXJ0IHJlcXVpcmVkID09PSIKbG9nIElORk8gIlN1cnBsdXMgam9iIHBvZHMgYXJlIGdhcmJhZ2UtY29sbGVjdGVkIGJ5IHRoZSBqb2IgY29udHJvbGxlciB3aXRoaW4gYSBmZXcgbWludXRlcy4iCmxvZyBJTkZPICJCYWNrdXBzOiAkQkFDS1VQX0RJUiIKbG9nIElORk8gIlZlcmlmeSB3aXRoOiAkMCAtLW9ic2VydmUiCmV4aXQgMAo="
  3. Inspect first - this changes nothing:
    base64 -d <<< "$PATCH64" | bash -s – --observe


    Note:  the "containers polled by cri-dockerd" figure, the retained pod counts, and the kubelet PLEG/restart counters. If PLEG counters are non-zero, expect the next step to be slower than usual.

  4. . Apply:

    vracli cluster exec – bash -c "base64 -d <<< '$PATCH64' | bash -s – --apply"
  5. Verify (allow a few minutes for job garbage collection):

    base64 -d <<< "$PATCH64" | bash -s – --observe
kubectl get cronjob -n kube-system -o custom-columns=\
NAME:.metadata.name,BACKOFF:.spec.jobTemplate.spec.backoffLimit,\
OK:.spec.successfulJobsHistoryLimit,FAIL:.spec.failedJobsHistoryLimit



  6. Steps to revert: 
    base64 -d <<< "$PATCH64" | bash -s – --revert

    (if PATCH64 is not set in the current shell, run step 2 again first)

    Restores the CronJobs to 20/20 with backoffLimit=6. File changes are not auto-reverted (they are strictly safer than the originals); restore them from the backup directory this patch creates on first run, if required. No restart needed.

 

Downtime for above steps: 
None. This patch does not restart any service and does not require running the product's deploy/reconfigure command.

Powered by Wolken Software