Is it Ok to delete the files and folders from \CA\DSM\PatchManagement\downloads ?
Client Automation - All Supported versions
There is no need to keep the downloaded files and they can be deleted every week or so.
You can also configure this delete task in Patch Manager Administration tab --> Data Pruning option ( As shown in below screenshot. )
Prune download records
Records older than (days): Specify number of days you would like to keep the files in download folder. (As shown in below picture)
For Example : Below screenshot shows files deletion is configured every 20 days.