You will need to use the Custom File Type Detection tool to identify the custom file type of the encrypted .zip or .rar file. Please see Symantec_DLP_11.0_Detection_Customization_Guide.pdf which gives you details on how to use the File Type Analyzer utility.
You may find the section "Tutorial 2: Detecting an encrypted ZIP file format" on page 35 particularly useful. The Custom File Type Detection tool mentioned in the PDF file applies to versions 11.0 and higher.
If you need further assistance with this please contact your Symantec Consultant or Professional Services team.
References:Can Vontu DLP detect password protected .zip or .pdf files on the Endpoint? Can a password-protected document be indexed?