GEL/REST API <?> and error 400 with special characters

Products

• Clarity PPM On Premise
• Clarity PPM SaaS

Issue/Introduction

Special characters (such as the £ British Pound [GBP] currency character) as they are read from the database in a GEL script that uses the REST API to write the content into a String attribute leads to unexpected characters (<?> symbol).
If the string contains a newline (non-printable character), the REST API fails completely returning a 400 error.

Steps to Reproduce

1. Create two new String attribute in the Project object (e.g.: my_string1 and my_string2) and enable them in the MUX (z_my_string1 and z_mystring_2).
2. Add the attributes into a Classic UX view (e.g.: Project Edit View).
3. Configure the fields to be a Text Area, instead of a Text Entry.
4. Create a new project instance and set the value to '£ GBP'
5. Create a GEL script that reads from the database the value of my_string1 for that instance, and uses the REST API to patch the value of my_string2 with the read value.
6. Check the value of my_string2.

Expected Results: my_string2 displays '£ GBP'.

Actual Results: my_string2 displays '<?> GBP' ('<?>' is a single, special character)

7. Update the value from Step 4 to '£ GBP[newline]GBP' ('[newline]' is a new line with enter key, not actual text)
8. Execute the same script as in Step 5.

Expected Results: my_string2 displays '£ GBP[newline]GBP'.

Actual Results: REST API request fails with an error 400.

Resolution

For the first scenario (£ GBP), specify the correct charset. Example fragment of code:

      <core:set var="createProject" encode="0" escapeText="true">{"z_my_string2": "${my_string1}"}</core:set>
      <core:new var="entity" className="org.apache.http.entity.StringEntity">
        <core:arg value="${createProject}" type="java.lang.String" />
        <core:arg value="UTF-8" type="java.lang.String" />
      </core:new>

For the second scenario (£ GBP[newline]GBP), before making the API call the payload needs to be escaped as the control characters are being used in the payload. Example fragment of code:

      <core:invokeStatic className="org.apache.commons.text.StringEscapeUtils" method="escapeJson" var="escaped_my_string1">
        <core:arg value="${my_string1}" type="java.lang.String"/>
      </core:invokeStatic>

[...]

      <core:set var="createProject" encode="0" escapeText="true">{"z_my_string2": "${escaped_my_string1}"}</core:set>

Additional Information

Ampersand value escaped always in REST via GEL