This issue is observed only in environments that have been upgraded from Aria Automation to VCFA
Symptoms:
kubectl get pods -A" we can see orphaned pods (`Evicted`, `Error`, `ContainerStatusUnknown`, ...).VCF Automation 9.0.x
VCF Automation 9.1.x
When the environment was upgraded from Aria Automation to VCFA, it copied the database which contain vco configuration that do not have rotation configuration (log rotation was diferent compare to VCFA). The vco service reads configuration from the vco database first instead from the configuration file that cause logs to pile and eventually filling the disk space.
The engineering team are working to provie the fix in the next VCFA version.
Workaround:
ssh vmware-system-user@<VCFA Platform FQDN>
sudo -ibase64 -d <<< '/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4D3/E5ddADkZSf35ZUq9VGwG5P4dAl9ueWfmpmXMCr1wq1V1HdWApe6uXlwtoBNZ7CVjVP549zkkeMuV1bqAoOdtRJor7tmCvyeddH5KYN2U9gEDJrDczAnJ/dYO/8njEIoArsR7/lN9cG+gzaulPPkTdTEXAcCNno6xpBzo/Cajc0ujApn5Z0+Yjr8+Tg/Bz9MX7CglGP6iBeYXwLpf6IVIZBGCbvdvvZoSH1CLPOkkxtdJpzAxUqw1TGGjACG3tfwVU27gW+flZIaghrqNKL+nK+yQ2fvKvLNXG0RZ1sAeNP7pHWIasmaaEEatXTE5En2onmsmGXUPG4STP2AXhdilREpn+E29u835+pX71e9NkU/T8GR8Tev2xTtrdCPxCzkJ4wcguP4ZkEQCGYjBncQWn7y5qnOOIA0J4lKglUL9E/NDNfjyLOuLni3EBy2Y3zEHlezuMgRvnX0BOHTFL9xGUOwUiJidLRQGA+52UmcoQqQS7wlJNkVWUb1KdvDBmqgeLL5g46oErm+73Jl2MglTkCKPszEx/PKoZY9Y2ESS+80vdhGb/d99bcvlsgQwIInrBmSVCClEOoBTEV/t3HgyF78gDgMe+QDCcC8l3PnkDeL0eUNSxiCkpWmQMuwdIMYxWxt62fiiwOoFB6Jud/DeHgHI1Nw932ttmuA/yM9wrDMC2fXsFKGO4yYpkMWhUoV81e6SG64OkOqG8jPy5ijeofr2fwfXhVWGcLTlCacloZTK50AXDGv0gtzOK93lWHfPY/h2+7EgQqtWfMc0JxyMOG1me1IfMF3PRunIWlm2vy5pLIAOVcjQ0QOSL4SOl/Q83GAss4aKJUrZLfiiyfnfVX3ELSsIRLFnua63DTx3VTvmffdQeAFSR6ffhER7nxzUXdPMWdIQ5MarnFsEAzs9Ikacfbn/HTvZIFUVcYCHoWstxD3lBYB95RiWOlWlTO4VhYCt1R9V0bP+eNoh7tL8/yfoIm7Rf/tnrvLV64ZLmMnPz4h0uh9OneJ6FqnAImmJWiICijT80XXtCnXQSXOfPAu01RGHgHNuxIEQkfxCrCwBuxsGg4s8Hi0cCI2WQu+DHXNyiWC/qT/IBn2Wf2NX7YBi8IRUbR32oaWIQYGCLiNShrC5ttVsKSfBPNLmFMYxHFtyf6h0DTinyBx9Pb/O5IGTxX20CCbPJKELbUILMHrPesgH9saN+UwVhyPGFq026z67uZfcoYiW9QJ8Y3aV7dcEV8DB+HsI1DG1NLnup0Kzo8g4EA4wTEym2nRlxUS5/7p/zZLblmYz+PLk5rH5iNtvwbSHi3ZzXw99rWs8pU9Te8sla/TRFx4t7+VUIWTTcMvl4fcoajQb0/+p38NhjYUoKXbd+Kl2J5JkpCGjsEbS0bkPxVhlGr0V0x0PsnsT1eNJ63ySmehFkkujBDRO0p9MeG8L55AFHs2V+R+qi2UOjVG9oykTClrIpYioT4lpPjJ7QnZcUJIVmf1jNcnKkAbBKMG6GSiFm2lcZrN8SMpMUu732yg/ErlpBlfBQlBQ70NcZbBNkN5NYX5Ex6Vn1Qg7NLbmQfIcJBPuMWhvUcgCvWhl+24UYQDhA9cHlo8ipaoUMHdZDGVqOSYlBmMifXsCgKnLWu3OK6bGpa92j42KXGBWqrPgPCEj9Hm4r4Valtp/x388YupqVIaixJXhuYq/c34bSvkfGYXsrnc5FUUZEq5YgWtzgBsgze9trvdH/JYNAxB3NSthiRUFaq9XqBhdX4+J0FsiOp0sSwSZ7/vyx5P3IoBOhlfeeBuEd4QA0hIhpQ+Ubd0z57pyUWIuFxTciuipenoXsdnka4Ks0rfD0hVQXIZ1Vv9jWym6bYLMKI9WRpsm+qe1G1fJVEo9Xw0Rl+QcDcAnrTPj39AINZwXB4b7VPpgBsJmsz8jiExiknk7Mr1Zp6evSndbRaybx9jn6haFzVG0IeOOv1FUpbSUaa7syQO1/iwANhYv++OUBhVzALJb3ubWChATO1HiCIjenXc2V3LuA0+MteM+k/qpIeClf1rHYMz/0YPik0zZNRXJOnyBqwV1nG2wjCUBm25VXqG3MbZH5eZ8htcE5U9Fs1fKZtQo39V/j974U6EsNgMPz2d+EOAv9Rt9tLnpTk/xpoOJApIKsDY1lKgRKvcBDttcjfVyNZ1UPgERiNcsXuW71qyzhRIUFrZ2A9ZqCVVz3t72im0rK4Zr38Gz/DuPq1Sx7bZI73bZzJ6xjMbCi3fvOTKNL/qzJELXZHgEJ405nsZCnxt9aGO+IoOed7/IyHIb9gBDBPkkk1eq/ck84LAsa6gL7U0AVdZPhvpmI6AMYyChVnnFL5SOrqhuvab5r2tTHAoKdmZkGLQE3O2kMC678IG18P8tOk46bYB3oQXH1R6Znaomt/Yj8PU+mM43YdpCtfr/cxuDoFst1wd09ZGcRx/Gj7TKV5c0Q+IwzW7ocTVWGTZcEb0n2mzXXPqOTfld1ruXDhLLy/tlaRxWLQGyMYn6FUSHG2tNjUt4xFpfzUxnbF5HEA1rLCxeJ/jOxj4IHfbtzzAh0UUjyKaj/ejQ/lamGxYd4GsKJaHaXwT1CIE930fA6Ogj53JR8lTkyhdqXCZ3fATRT8FG5RY+0fiQzhKs2Q5MibeDnUAK+KlPPx35Qi5AeJ51REXGng/9ls8MUmr9UFyoM21fU92bCdnLmrxl7j18fOuda2SWSW381xG9YiIvO/TNhnpdBigCAVVj19UKqofGZBPzKAosQDf1v2RpUmsXD3LBVr0GTw5sYsj2AJPqXjOu2UYuNcIVXHQddP/4yekbW8iepxwMc+T4nNbRRfsNYmtafiHulf9B7H+zoVIsy9q5sPsXla7EQ0xNZKUly6n3D0Un744xlUY481NU9AIgz+9auBaLN5IiFItvddwduYRRIsNtLQ7d1nqU2J8SQrDwQ+Dg8aC4ufzIBo5aeB7Vt94FYd5/rSi/yX8cHv6J56kxPFJKIP3rvDpoZ8Z2pRqheSvSA9PppXIOm8kUvYJRujj/JgWWcn6WASGbZa9Ks2Tv8un73KxRID/adEGAgPtI5EUlQbTJMgOeypXh32BYVzDxBA0PmqCq7Qr/seD8PlKB99FFkZySVAzUeRKK3ndoPcilDDoqt4bfKJyXMuiy95cgvq2bDhwShBmPQWNvLR1mCchngpDNKzAakJciKfEZmVlKFE99fXfo2iUXP7wyYgb8GYZUlI1rggtXuBAJsJTVQMOQaC21hXEeULuXUJyABhP3xzDTDxlagoKLOMGd5u0nj5FjzT7u2xyouBsuNjt9CDsEHYUFnvltf2chf3ejoDcRcXl24ZMaTd09LYZDXTWhcGjCmOZveLkrMsEclXqP+JB2XBskKLuFclBrE1nB6yYCzSad4lYSR5wVvqYEnyVXgbh2jqw1FhGsqLG0ycTa8YryDJGWwGifNWh2wf2MhLBkkuY7G5kV8AGXUsoizlkhENI/k4w5jZjvW9QwxH8nleoyyNwH7LPPZvxOnQwvvDDaDf5JLjI9pWYw5ybBG3Zw0r8XE34sctJM7ZoUCKue0ILnKTZcNJyZ6DwsqAnpCHDXAgAH5MwBM7EiypXCKhg2mrzfKxRiV+DW5PIrX0K2h8xHkfbY1NVI50sbsUUReD6hy55Rmia9QVNA06eh5cAA8yHLSERzcvb5mma3I6ON6giG3Y0oyHodYN+wIoWZ6xgSGaqCUiOX/vQDPquex0ntp0MSu0SM6jooRdiWfRUqYkSptEYWcKuOWgWO8kvVroLxJHUCrxh6tf3E1oBrBzvkXGtBbxxC/FcAWRPrXtbbe6Q3oiOHAczzN+pWLO9FecPIiihyRQLgNILaaEaTwMRQf1fiFRClnSWmFmLBj9vf9/3o/BPs3MqqV27g6TtLpjbkWauWydes9rLxgWycbVXH4JOoBmxljAtPAzeGbn3ai2loyg7nHxAVNtJTCan4IY5JqswYhEb+gqkPvexZN7FWRmf/kv82SkeqJnzqpDqKMDelQ2vDw4rBFUR3wxU69UzxVKsZYyvW43+de/c415tT/sUFLeeSVlTbVhiiZK7QMdul1pX5lGif4LddWK5LDxcw5ethx18RhKbrgXsoI73+20uHFYZEgaHNt95MvBiZ2nqlebmSxZvEnIXC3oaStwsV3PbrIEw0koDeSCrJH8fBIY9D/Eg90CPf+B3j59FJ1FNtc1dZlRybXhNWMZPvIIsXfpFq/I4ykE1iS7T5ThKP4vZC0aDA/C4o4jWgecAcJRaNc2BTLmCQ18XYCyCP0BlgZqv3NfVstw+I/OQAhY/VllINx9WpLuALg27lIzJfuvTV75lW79+ksNhVwHthLDZEq3UEOMBVgGlm8OzOEbrKTT4JDtZum2FLoHWMl+OwWjOLl8QT1ZhJ80Y/gM2H0iz7R/kfiAkcifVZAS+O6AonPbx1vVIrtGZaWz09T3UrJ++DrtmiDYvWTGhgTmgVQsUlOUX/u4ZjXYT1cfmqBA38NcgtsQb2oUnncpxGA9MO9ktgCLmVkDAHyGDr0T9O06ayjRqM4EUUjuFd9xwBGrHx/7Qiui38SCcwrjOSCgTx1E5tUTn5Ti5JMs+xPYS0Och/fvVuP475pBdUkZ2eopPWBD3/z2yXQU2Gfce1Ta6mYuuJIoVaMre9pbfJrxTW8rHM1PWZw4jy3zDHIL+bIlb6KqSqlQleaL4vx5Ro69rkHv0uKwaZlrS0EzGlQV2CRRNdV9/T3bBF9YsW8DqN5I6/iRxx6S26EbXZ7G2YhEGzi8M21PYXnRJ4PvjZDB04H523LVlNLsxqO5Er4QgKI0gF0eYTlrKGDhjbuHYCUoiAAeBpI8gzjWLg/6n1yk3ZRWmeJdrccKEdoykSZGoUwQUbw4dHs9495DbanTQimgTHXPmtR0Jx7miDofmnqBSMy5kxG7TRqe8xAy8J3w4s8oPOFGGZcQRV7Yyr68r09aFQkQhR7vtw+iLiILZE2EL3kieed8QFst4r1ZJIQeoE4zcYDwfaFMOAvrOmm1l+lHhrKWicE2ygnt3hL/emOeN/uf/ZJRWQpX3GEuHb7/thlIrIHWytX/w9Blu7jP1/KS1ETwoiYnVoF2eRdov5BZreWUGRM8OrHX7oZ7tNgusolfzkrH34XrjKiWUjzc8GyseKD593O/L0juWcyXcfvXkSwWoAQowKDyg8rPRkEpEih3ubyXBks3OEKivyVwhR1uklvNoS7RLc9hiVo/xq9eIoJ1r591y+onBEDHyHGpOH5iYXRW5nQsms7DldniLj+kWteq6u+1KuFRADhZgujMGeieRO25w7MoveNT5mO3HB6y3SjUV03f6cgwIZNZ5Tc2wxmZDV6thIoWTDN70myO3f9gUjAtKcvCxswppUyI8Z10YDGlCgCI3WsZMeLzow8wXdd1fPkF1CnuehlBC676y0Te759G6gZ2kTzqfhpQWg4tl9XjMDCQXQxxdHQdBdG4/kk1/ek72sw65b+hoeAX8pve4lCQmTsrNhap++uvibbZnqHxATisQN37u0u1glgxCPscyfeqaZEHFo9IO0ttFub6HRou7jM7Er6VfXMj4X7xyHhxfYGnMyckMu08lhS8sxe2vqCtAdEk2AOwf8GP++7WvS0q60vwnEyQmbyz7PStgD4MKbCCtio9oBzirbVdUmMHD+go/W1R+scHwCl6yW1LgvsPcWSJ9AfkKrRm/TxrZIrjNnc4NNOpVFyU4E2Is29RWqEk6hcRp7vhePZ+tCtMr6wrSva0aoLmy55xPV98tQhZYLHTMAZVKxU5+Q+P0Y9P3o7RjHwdmAZaivTHANz5dfr9K7Wtvy0iLr/hGVx6J+Nr5JF2Zh8mD0FRTJlSM5qM1JulKdkcf0yDBh+lvPTRO1eXJoEVmh8XNF7So1Q2WTbtAVLpWiyYXLsaxg2l/tWDRgc3ati1aMF0eslF83kDtIMR0HpeCffpkbRS1PjW7VrXVUHZwUytDOQCUE0FwRQNHVYvu9QJ1Ls16EpR/m5ewywQVU35gO5I99c3CcsQPwDFwGDGL6CiE0bEEzAL69ApfIRlYg4XnOBiEGsJLCDKyzX8sQh3aIkZ7RIV/57jJKlZctyzse4Pnz78Q5KeYY6tZQFsU37Z8lel/8A59/YkgCQDE+mk6tQaaWCk4mo3tMKn7uA35F7p9Uyqy7Xc/gi1mggf6W9YbKLIe9stYeWLmD1sUTi3fT/qMw8vKhin1eIv48YEtHftrEF5LFgoox/dgi07kSfLy85gZzzgHyZ/szMiJSbTYYcGx27IGgMfMwdJDAMYJD9Tzk9F9dp/fgAe0Osemny2ow7xVmYBNX7OiNkFGLxxxsSBgLqZ3PtVVtTLNxT/1G1rCkpTspRysSpKOARAPq2muQ0rrj2o7jtSNOdj+gxUL/u4ixdBp8uWrbZPVtp90muckb9p4YLMx+PPBdWdkUI8tTlfTCUuc4D3vfkhbJuUwlfceouPtt8wxS2xUf5YjtfjO1M1IHx9caWMkYTlPQHndbrpTPttUIvb48XqwFTwsShrY9Ahqep1Ge2Jazn8yYambCw/9SfJwrw9q2TmaMzVVHJEqRLQs2R+g3walXLuhm/r8Qz6V8udD+ZfmPMiitwoUMAN8FDnXk12g9BZNisTL5obgQcYYBQP9qoZkfr2u/ogcHVfuOHYNZHFUSUCaQ3VsCMz+uLAAAAABwJSM6ZpeNhQABsyeAfAAAHWFYnLHEZ/sCAAAAAARZWg==' | tar -xJO | bash