The base install pax file was not uploaded to the USS directory as BINARY.
Delete the original pax file in the USS directory and uploaded it again in binary mode.
Also consider using the BPXBATCH utility program documented in section Create a Product Directory from the Pax File
Below are some of the files that get unpaxed for r11:
AVHB0B0
AVHB0B0/AREADME
AVHB0B0/S0001.MF20.MSPS.TFMSYM.R110BP0.AVHB0B0.SAMPJCL.pax.Z
AVHB0B0/SMPPTFIN
AVHB0B0/SMPPTFIN/S0002.MF20.MSPS.TFMSYM.R110BP0.AVHB0B0.SMPMCS.pax.Z
AVHB0B0/SMPRELF
AVHB0B0/SMPRELF/MF20.MSPS.TFMSYM.R110BP0.CAI.CAVHB00.F1.pax.Z
AVHB0B0/SMPRELF/MF20.MSPS.TFMSYM.R110BP0.CAI.CAVHB00.F2.pax.Z
AVHB0B0/SMPRELF/MF20.MSPS.TFMSYM.R110BP0.CAI.CAVHB00.F3.pax.Z
AVHB0B0/SMPRELF/MF20.MSPS.TFMSYM.R110BP0.CAI.CAVHB00.F4.pax.Z
AVHB0B0/SMPRELF/MF20.MSPS.TFMSYM.R110BP0.CAI.CAVHB00.F5.pax.Z