GPSS Does Not Support Kafka Topics with Zstd Compression

Products

• VMware Tanzu Greenplum

Issue/Introduction

Issue:
When attempting to load data from a Kafka topic that uses Zstd compression, GPSS fails to read and load the data successfully.

Description:
If a Kafka topic is created with zst compression enabled, any GPSS job that tries to load data from this topic will fail with an error.

Example:

Kafka Topic Creation and Data Production

/opt/kafka/bin/kafka-topics.sh --create --topic zstg_kafka_topic --bootstrap-server localhost:9092 --config compression.type=zstd
/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic zstg_kafka_topic < /data/etl/sample_data.csv

Load Data Using GPSS

# gpkafka load --quit-at-eof gpss_job.yaml
20251007 14:06:49 - 2741264 [INFO] gpss version: 2.1.0
...
20251007 14:06:49 - 2741264 [INFO] gpkafka job has started
StartTime           EndTime         MsgNum      MsgSize     InsertedRecords RejectedRecords Speed
20251007 14:06:50 9760f8c7 2741264 [ERROR] Worker 9760f8c792363d5e7dc0705b849280be.0 failed due to kafka read worker 0 got error: Local: Not implemented
20251007 14:06:49.98400 20251007 14:06:50.18700     0           0               0           0           0B/sec
20251007 14:06:50 9760f8c7 2741264 [INFO] Target table: "public"."data_from_kafka"
20251007 14:06:50 9760f8c7 2741264 [INFO] Inserted 0 rows
20251007 14:06:50 9760f8c7 2741264 [INFO] Rejected 0 rows

 

Resolution

Root Cause:
R&D has confirmed that the current version of GPSS (latest as of now: v2.1) does not support loading data from Kafka topics that use Zstd compression.

Workaround:
Disable Zstd compression for the Kafka topic before running the GPSS load job.

Future Plan:
Support for Kafka topics with Zstd compression is already included in the GPSS product roadmap.
Please monitor the GPSS release notes to confirm when this feature becomes available in future versions.