Orchestrator (vco-app) logs fill the disk space of the VCFA appliance
search cancel

Orchestrator (vco-app) logs fill the disk space of the VCFA appliance

book

Article ID: 458715

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

This issue is observed only in environments that have been upgraded from Aria Automation to VCFA

Symptoms:

  • Disk usage on the VCFA nodes is high or at 100%, caused by Orchestrator logs.
  • When run command "kubectl get pods -A" we can see orphaned pods (`Evicted`, `Error`, `ContainerStatusUnknown`, ...).
  • The vco-app does not start; it waits on a Liquibase lock (`databasechangeloglock`).

Environment

VCF Automation 9.0.x

VCF Automation 9.1.x

Cause

When the environment was upgraded from Aria Automation to VCFA, it copied the database which contain vco configuration that do not have rotation configuration (log rotation was diferent compare to VCFA). The vco service reads configuration from the vco database first instead from the configuration file that cause logs to pile and eventually filling the disk space.

 

Resolution

The engineering team are working to provie the fix in the next VCFA version.

Workaround:

  • SSH into the VCFA appliance and become root
    ssh vmware-system-user@<VCFA Platform FQDN>
    sudo -i
  • Run the below script from CLI to update the vco-servise log rotation configuration and cleanup orphaned pods:
    base64 -d <<< '/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4D3/E5ddADkZSf35ZUq9VGwG5P4dAl9ueWfmpmXMCr1wq1V1HdWApe6uXlwtoBNZ7CVjVP549zkkeMuV1bqAoOdtRJor7tmCvyeddH5KYN2U9gEDJrDczAnJ/dYO/8njEIoArsR7/lN9cG+gzaulPPkTdTEXAcCNno6xpBzo/Cajc0ujApn5Z0+Yjr8+Tg/Bz9MX7CglGP6iBeYXwLpf6IVIZBGCbvdvvZoSH1CLPOkkxtdJpzAxUqw1TGGjACG3tfwVU27gW+flZIaghrqNKL+nK+yQ2fvKvLNXG0RZ1sAeNP7pHWIasmaaEEatXTE5En2onmsmGXUPG4STP2AXhdilREpn+E29u835+pX71e9NkU/T8GR8Tev2xTtrdCPxCzkJ4wcguP4ZkEQCGYjBncQWn7y5qnOOIA0J4lKglUL9E/NDNfjyLOuLni3EBy2Y3zEHlezuMgRvnX0BOHTFL9xGUOwUiJidLRQGA+52UmcoQqQS7wlJNkVWUb1KdvDBmqgeLL5g46oErm+73Jl2MglTkCKPszEx/PKoZY9Y2ESS+80vdhGb/d99bcvlsgQwIInrBmSVCClEOoBTEV/t3HgyF78gDgMe+QDCcC8l3PnkDeL0eUNSxiCkpWmQMuwdIMYxWxt62fiiwOoFB6Jud/DeHgHI1Nw932ttmuA/yM9wrDMC2fXsFKGO4yYpkMWhUoV81e6SG64OkOqG8jPy5ijeofr2fwfXhVWGcLTlCacloZTK50AXDGv0gtzOK93lWHfPY/h2+7EgQqtWfMc0JxyMOG1me1IfMF3PRunIWlm2vy5pLIAOVcjQ0QOSL4SOl/Q83GAss4aKJUrZLfiiyfnfVX3ELSsIRLFnua63DTx3VTvmffdQeAFSR6ffhER7nxzUXdPMWdIQ5MarnFsEAzs9Ikacfbn/HTvZIFUVcYCHoWstxD3lBYB95RiWOlWlTO4VhYCt1R9V0bP+eNoh7tL8/yfoIm7Rf/tnrvLV64ZLmMnPz4h0uh9OneJ6FqnAImmJWiICijT80XXtCnXQSXOfPAu01RGHgHNuxIEQkfxCrCwBuxsGg4s8Hi0cCI2WQu+DHXNyiWC/qT/IBn2Wf2NX7YBi8IRUbR32oaWIQYGCLiNShrC5ttVsKSfBPNLmFMYxHFtyf6h0DTinyBx9Pb/O5IGTxX20CCbPJKELbUILMHrPesgH9saN+UwVhyPGFq026z67uZfcoYiW9QJ8Y3aV7dcEV8DB+HsI1DG1NLnup0Kzo8g4EA4wTEym2nRlxUS5/7p/zZLblmYz+PLk5rH5iNtvwbSHi3ZzXw99rWs8pU9Te8sla/TRFx4t7+VUIWTTcMvl4fcoajQb0/+p38NhjYUoKXbd+Kl2J5JkpCGjsEbS0bkPxVhlGr0V0x0PsnsT1eNJ63ySmehFkkujBDRO0p9MeG8L55AFHs2V+R+qi2UOjVG9oykTClrIpYioT4lpPjJ7QnZcUJIVmf1jNcnKkAbBKMG6GSiFm2lcZrN8SMpMUu732yg/ErlpBlfBQlBQ70NcZbBNkN5NYX5Ex6Vn1Qg7NLbmQfIcJBPuMWhvUcgCvWhl+24UYQDhA9cHlo8ipaoUMHdZDGVqOSYlBmMifXsCgKnLWu3OK6bGpa92j42KXGBWqrPgPCEj9Hm4r4Valtp/x388YupqVIaixJXhuYq/c34bSvkfGYXsrnc5FUUZEq5YgWtzgBsgze9trvdH/JYNAxB3NSthiRUFaq9XqBhdX4+J0FsiOp0sSwSZ7/vyx5P3IoBOhlfeeBuEd4QA0hIhpQ+Ubd0z57pyUWIuFxTciuipenoXsdnka4Ks0rfD0hVQXIZ1Vv9jWym6bYLMKI9WRpsm+qe1G1fJVEo9Xw0Rl+QcDcAnrTPj39AINZwXB4b7VPpgBsJmsz8jiExiknk7Mr1Zp6evSndbRaybx9jn6haFzVG0IeOOv1FUpbSUaa7syQO1/iwANhYv++OUBhVzALJb3ubWChATO1HiCIjenXc2V3LuA0+MteM+k/qpIeClf1rHYMz/0YPik0zZNRXJOnyBqwV1nG2wjCUBm25VXqG3MbZH5eZ8htcE5U9Fs1fKZtQo39V/j974U6EsNgMPz2d+EOAv9Rt9tLnpTk/xpoOJApIKsDY1lKgRKvcBDttcjfVyNZ1UPgERiNcsXuW71qyzhRIUFrZ2A9ZqCVVz3t72im0rK4Zr38Gz/DuPq1Sx7bZI73bZzJ6xjMbCi3fvOTKNL/qzJELXZHgEJ405nsZCnxt9aGO+IoOed7/IyHIb9gBDBPkkk1eq/ck84LAsa6gL7U0AVdZPhvpmI6AMYyChVnnFL5SOrqhuvab5r2tTHAoKdmZkGLQE3O2kMC678IG18P8tOk46bYB3oQXH1R6Znaomt/Yj8PU+mM43YdpCtfr/cxuDoFst1wd09ZGcRx/Gj7TKV5c0Q+IwzW7ocTVWGTZcEb0n2mzXXPqOTfld1ruXDhLLy/tlaRxWLQGyMYn6FUSHG2tNjUt4xFpfzUxnbF5HEA1rLCxeJ/jOxj4IHfbtzzAh0UUjyKaj/ejQ/lamGxYd4GsKJaHaXwT1CIE930fA6Ogj53JR8lTkyhdqXCZ3fATRT8FG5RY+0fiQzhKs2Q5MibeDnUAK+KlPPx35Qi5AeJ51REXGng/9ls8MUmr9UFyoM21fU92bCdnLmrxl7j18fOuda2SWSW381xG9YiIvO/TNhnpdBigCAVVj19UKqofGZBPzKAosQDf1v2RpUmsXD3LBVr0GTw5sYsj2AJPqXjOu2UYuNcIVXHQddP/4yekbW8iepxwMc+T4nNbRRfsNYmtafiHulf9B7H+zoVIsy9q5sPsXla7EQ0xNZKUly6n3D0Un744xlUY481NU9AIgz+9auBaLN5IiFItvddwduYRRIsNtLQ7d1nqU2J8SQrDwQ+Dg8aC4ufzIBo5aeB7Vt94FYd5/rSi/yX8cHv6J56kxPFJKIP3rvDpoZ8Z2pRqheSvSA9PppXIOm8kUvYJRujj/JgWWcn6WASGbZa9Ks2Tv8un73KxRID/adEGAgPtI5EUlQbTJMgOeypXh32BYVzDxBA0PmqCq7Qr/seD8PlKB99FFkZySVAzUeRKK3ndoPcilDDoqt4bfKJyXMuiy95cgvq2bDhwShBmPQWNvLR1mCchngpDNKzAakJciKfEZmVlKFE99fXfo2iUXP7wyYgb8GYZUlI1rggtXuBAJsJTVQMOQaC21hXEeULuXUJyABhP3xzDTDxlagoKLOMGd5u0nj5FjzT7u2xyouBsuNjt9CDsEHYUFnvltf2chf3ejoDcRcXl24ZMaTd09LYZDXTWhcGjCmOZveLkrMsEclXqP+JB2XBskKLuFclBrE1nB6yYCzSad4lYSR5wVvqYEnyVXgbh2jqw1FhGsqLG0ycTa8YryDJGWwGifNWh2wf2MhLBkkuY7G5kV8AGXUsoizlkhENI/k4w5jZjvW9QwxH8nleoyyNwH7LPPZvxOnQwvvDDaDf5JLjI9pWYw5ybBG3Zw0r8XE34sctJM7ZoUCKue0ILnKTZcNJyZ6DwsqAnpCHDXAgAH5MwBM7EiypXCKhg2mrzfKxRiV+DW5PIrX0K2h8xHkfbY1NVI50sbsUUReD6hy55Rmia9QVNA06eh5cAA8yHLSERzcvb5mma3I6ON6giG3Y0oyHodYN+wIoWZ6xgSGaqCUiOX/vQDPquex0ntp0MSu0SM6jooRdiWfRUqYkSptEYWcKuOWgWO8kvVroLxJHUCrxh6tf3E1oBrBzvkXGtBbxxC/FcAWRPrXtbbe6Q3oiOHAczzN+pWLO9FecPIiihyRQLgNILaaEaTwMRQf1fiFRClnSWmFmLBj9vf9/3o/BPs3MqqV27g6TtLpjbkWauWydes9rLxgWycbVXH4JOoBmxljAtPAzeGbn3ai2loyg7nHxAVNtJTCan4IY5JqswYhEb+gqkPvexZN7FWRmf/kv82SkeqJnzqpDqKMDelQ2vDw4rBFUR3wxU69UzxVKsZYyvW43+de/c415tT/sUFLeeSVlTbVhiiZK7QMdul1pX5lGif4LddWK5LDxcw5ethx18RhKbrgXsoI73+20uHFYZEgaHNt95MvBiZ2nqlebmSxZvEnIXC3oaStwsV3PbrIEw0koDeSCrJH8fBIY9D/Eg90CPf+B3j59FJ1FNtc1dZlRybXhNWMZPvIIsXfpFq/I4ykE1iS7T5ThKP4vZC0aDA/C4o4jWgecAcJRaNc2BTLmCQ18XYCyCP0BlgZqv3NfVstw+I/OQAhY/VllINx9WpLuALg27lIzJfuvTV75lW79+ksNhVwHthLDZEq3UEOMBVgGlm8OzOEbrKTT4JDtZum2FLoHWMl+OwWjOLl8QT1ZhJ80Y/gM2H0iz7R/kfiAkcifVZAS+O6AonPbx1vVIrtGZaWz09T3UrJ++DrtmiDYvWTGhgTmgVQsUlOUX/u4ZjXYT1cfmqBA38NcgtsQb2oUnncpxGA9MO9ktgCLmVkDAHyGDr0T9O06ayjRqM4EUUjuFd9xwBGrHx/7Qiui38SCcwrjOSCgTx1E5tUTn5Ti5JMs+xPYS0Och/fvVuP475pBdUkZ2eopPWBD3/z2yXQU2Gfce1Ta6mYuuJIoVaMre9pbfJrxTW8rHM1PWZw4jy3zDHIL+bIlb6KqSqlQleaL4vx5Ro69rkHv0uKwaZlrS0EzGlQV2CRRNdV9/T3bBF9YsW8DqN5I6/iRxx6S26EbXZ7G2YhEGzi8M21PYXnRJ4PvjZDB04H523LVlNLsxqO5Er4QgKI0gF0eYTlrKGDhjbuHYCUoiAAeBpI8gzjWLg/6n1yk3ZRWmeJdrccKEdoykSZGoUwQUbw4dHs9495DbanTQimgTHXPmtR0Jx7miDofmnqBSMy5kxG7TRqe8xAy8J3w4s8oPOFGGZcQRV7Yyr68r09aFQkQhR7vtw+iLiILZE2EL3kieed8QFst4r1ZJIQeoE4zcYDwfaFMOAvrOmm1l+lHhrKWicE2ygnt3hL/emOeN/uf/ZJRWQpX3GEuHb7/thlIrIHWytX/w9Blu7jP1/KS1ETwoiYnVoF2eRdov5BZreWUGRM8OrHX7oZ7tNgusolfzkrH34XrjKiWUjzc8GyseKD593O/L0juWcyXcfvXkSwWoAQowKDyg8rPRkEpEih3ubyXBks3OEKivyVwhR1uklvNoS7RLc9hiVo/xq9eIoJ1r591y+onBEDHyHGpOH5iYXRW5nQsms7DldniLj+kWteq6u+1KuFRADhZgujMGeieRO25w7MoveNT5mO3HB6y3SjUV03f6cgwIZNZ5Tc2wxmZDV6thIoWTDN70myO3f9gUjAtKcvCxswppUyI8Z10YDGlCgCI3WsZMeLzow8wXdd1fPkF1CnuehlBC676y0Te759G6gZ2kTzqfhpQWg4tl9XjMDCQXQxxdHQdBdG4/kk1/ek72sw65b+hoeAX8pve4lCQmTsrNhap++uvibbZnqHxATisQN37u0u1glgxCPscyfeqaZEHFo9IO0ttFub6HRou7jM7Er6VfXMj4X7xyHhxfYGnMyckMu08lhS8sxe2vqCtAdEk2AOwf8GP++7WvS0q60vwnEyQmbyz7PStgD4MKbCCtio9oBzirbVdUmMHD+go/W1R+scHwCl6yW1LgvsPcWSJ9AfkKrRm/TxrZIrjNnc4NNOpVFyU4E2Is29RWqEk6hcRp7vhePZ+tCtMr6wrSva0aoLmy55xPV98tQhZYLHTMAZVKxU5+Q+P0Y9P3o7RjHwdmAZaivTHANz5dfr9K7Wtvy0iLr/hGVx6J+Nr5JF2Zh8mD0FRTJlSM5qM1JulKdkcf0yDBh+lvPTRO1eXJoEVmh8XNF7So1Q2WTbtAVLpWiyYXLsaxg2l/tWDRgc3ati1aMF0eslF83kDtIMR0HpeCffpkbRS1PjW7VrXVUHZwUytDOQCUE0FwRQNHVYvu9QJ1Ls16EpR/m5ewywQVU35gO5I99c3CcsQPwDFwGDGL6CiE0bEEzAL69ApfIRlYg4XnOBiEGsJLCDKyzX8sQh3aIkZ7RIV/57jJKlZctyzse4Pnz78Q5KeYY6tZQFsU37Z8lel/8A59/YkgCQDE+mk6tQaaWCk4mo3tMKn7uA35F7p9Uyqy7Xc/gi1mggf6W9YbKLIe9stYeWLmD1sUTi3fT/qMw8vKhin1eIv48YEtHftrEF5LFgoox/dgi07kSfLy85gZzzgHyZ/szMiJSbTYYcGx27IGgMfMwdJDAMYJD9Tzk9F9dp/fgAe0Osemny2ow7xVmYBNX7OiNkFGLxxxsSBgLqZ3PtVVtTLNxT/1G1rCkpTspRysSpKOARAPq2muQ0rrj2o7jtSNOdj+gxUL/u4ixdBp8uWrbZPVtp90muckb9p4YLMx+PPBdWdkUI8tTlfTCUuc4D3vfkhbJuUwlfceouPtt8wxS2xUf5YjtfjO1M1IHx9caWMkYTlPQHndbrpTPttUIvb48XqwFTwsShrY9Ahqep1Ge2Jazn8yYambCw/9SfJwrw9q2TmaMzVVHJEqRLQs2R+g3walXLuhm/r8Qz6V8udD+ZfmPMiitwoUMAN8FDnXk12g9BZNisTL5obgQcYYBQP9qoZkfr2u/ogcHVfuOHYNZHFUSUCaQ3VsCMz+uLAAAAABwJSM6ZpeNhQABsyeAfAAAHWFYnLHEZ/sCAAAAAARZWg==' | tar -xJO | bash