Greenplum 7.8.0: SUBSTR fails with "invalid byte sequence for encoding UTF8: 0xc3" on multibyte characters

Products

• VMware Tanzu Data Suite
• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

After upgrading to Greenplum 7.8.0, queries using the SUBSTR() or SUBSTRING() function may fail with the following error: ERROR: invalid byte sequence for encoding "UTF8": 0xc3 (segXX sliceX XXX.XX.X.XX:XXXX pid=XXXXXX) SQL state: 22021

This typically occurs when:

• The string contains 2-byte or multibyte UTF-8 characters, for example: 'Ä', 'ö', 'ü'.
• The truncation length specified in the function lands exactly on the first byte of a multibyte character.
• The issue was not present in previous versions (Greenplum 7.7.0).

Cause

This error indicates that the SUBSTR function is incorrectly calculating the truncation point based on bytes instead of characters. When a multibyte character like 'Ä' (UTF-8 hex: C3 84) is split after the first byte, it leaves behind an "orphaned" 0xc3 byte. Because 0xc3 is not a valid standalone UTF-8 character, the database throws an encoding error.

Resolution

This behavior is currently being investigated as a regression in Greenplum 7.8.0.

A fix will be available in Greenplum 7.8.2 and above.