The policy and policy type may differ in behavior is due to a setting in the Service Properties called "Perform WS-Security processing for this service". When that checkbox is checked, it results in the message being buffered before it reaches policy, and some processing of the message is performed before reaching the service policy as well. Therefore the message is checked against the cluster-wide property (io.xmlPartMaxBytes which can be set to -1 for unlimited) first, before it reaches policy. If the message passes the cluster-wide property, then it makes it to the policy, and is checked against the Limit Message Size assertion (if it exists). If the message fails the check against the cluster-wide property to begin with, it never makes it to policy, regardless of the value of the "Limit Message Size" assertion.
If the checkbox is unchecked, then the message goes to the service policy directly, and the message is compared against the Limit Message Size assertion if it exists, and if the assertion doesn't exist, will be compared to the cluster-wide property.
The difference between REST vs SOAP in this instance is that SOAP services by default have this checkbox checked, whereas REST services typically have this checkbox unchecked. If you uncheck this value from a SOAP service, it will behave like a default REST service. If you check this value in a REST service, it will behave like a default SOAP service.
So to simplify it, a request to a REST service or SOAP service with WS-Security processing enabled, is buffered and therefore checked against the Cluster-Wide Property first. If it exceeds the limit, you receive "Error: Unable to read stream: the specified maximum data size limit would be exceeded". If it is under the limit, the message continues to the policy, and processed accordingly.
The REST service and SOAP service with WS-Security processing disabled, goes to the policy first, and ONLY adheres to the Limit Message Size assertion (if it exists) and ignores the cluster-wide property. If the Limit Message Size assertion doesn't exist in the policy, then it adheres to the cluster-wide property.