Valkey shared VM smoke tests failed due to "Internal Server Error"
Article ID: 406423
Updated On:
Products
Issue/Introduction
After Running Tanzu for Valkey on Cloud Foundry smoke tests, it fails with "Internal Server Error" as shown. Manually running the smoke tests using "bosh -d p-redis-###### run-errand smoke-tests" results in the same error.
[32m[2025-07-01 13:19:01.06 (UTC)]> curl -k -H Expect: -s https://dc12##########ebdf.###.###.###.###.com/status/health [0m
app is running
[32m[2025-07-01 13:19:01.13 (UTC)]> curl -k -H Expect: -s -d data=myvalue -X PUT https://dc12##########ebdf.###.###.###.###.com/mykey [0m
<h1>Internal Server Error</h1>
[32m[2025-07-01 13:19:02.17 (UTC)]> curl -k -H Expect: -s -d data=myvalue -X PUT https://dc12##########ebdf.###.###.###.###.com/mykey [0m
<h1>Internal Server Error</h1>
When creating a test shared VM service instance, and accessing the Redis CLI (see section About the Redis CLI), the SET and GET commands can be executed with no error.
After binding the shared VM service instance to an test app, and executing the PUT KEY curl command manually, the same curl error persists.
git clone [email protected]:pivotal-cf/cf-redis-example-app.git
cd cf-redis-example-app
cf push redis-example-app -m 256M -b ruby_buildpack --no-start
cf bind-service redis-example-app YOUR-REDIS-SHARED-VM-SERVICE
cf start redis-example-app
cf env redis-example-app
curl -k -H Expect: -s https://YOUR-APP-ROUTE/status/health # shows "app is running"
curl -k -H Expect: -s -d data=myvalue -X PUT https://YOUR-APP-ROUTE/mykey # shows "Internal Server Error"
App logs retrieved via "cf logs redis-example-app --recent" shows the below error:
2025-07-08T17:58:59.75+0800 [APP/PROC/WEB/0] ERR 2025-07-08 09:58:59 - Redis::CannotConnectError - Operation already in progress - connect(2) for ##.##.##.##:41565 (redis://##.##.##.##:41565):
2025-07-08T17:58:59.75+0800 [APP/PROC/WEB/0] ERR /home/vcap/deps/0/ruby/lib/ruby/3.2.0/socket.rb:1217:in `__connect_nonblock'
2025-07-08T17:58:59.75+0800 [APP/PROC/WEB/0] ERR /home/vcap/deps/0/ruby/lib/ruby/3.2.0/socket.rb:1217:in `connect_nonblock'
2025-07-08T17:58:59.75+0800 [APP/PROC/WEB/0] ERR /home/vcap/deps/0/ruby/lib/ruby/3.2.0/socket.rb:60:in `connect_internal'
...
2025-07-08T17:58:59.75+0800 [APP/PROC/WEB/0] ERR /home/vcap/deps/0/vendor_bundle/ruby/3.2.0/gems/webrick-1.8.1/lib/webrick/server.rb:310:in `block in start_thread'
2025-07-08T17:58:59.75+0800 [APP/PROC/WEB/0] ERR 10.52.245.168 - - [08/Jul/2025:09:58:59 +0000] "PUT /mykey HTTP/1.1" 500 30 0.0022
2025-07-08T17:58:59.75+0800 [RTR/1] OUT redis-example-app.##.##.##.##.com - [2025-07-08T09:58:59.744607893Z] "PUT /mykey HTTP/1.1" 500 12 30 "-" "curl/7.81.0" "##.##.##.##:34844" "##.##.##.##:61002" x_forwarded_for:"10.##.##.168" x_forwarded_proto:"https" vcap_request_id:"edc9##########cbbf" response_time:0.007843 gorouter_time:0.000261 app_id:"e653##########9fa3" app_index:"0" instance_id:"a244##########fcdb" x_cf_routererror:"-" x_b3_traceid:"edc9##########cbbf" x_b3_spanid:"72b3######cbbf" x_b3_parentspanid:"-" b3:"edc9f##########cbbf" traceparent:"00-ed##########bf-01" tracestate:"####@gorouter=72b3#####cbbf"
After performing a connectivity check from within the container using the steps outlined, the nc output indicates a network connectivity issue, specifically between the logged-in machine and the Redis server.
# Create service-key. Note down the host, port and password from the service-key
> cf create-service-key <service-instance> <service-instance-key>
> cf service-key <service-instance> <service-instance-key>
# Log into the container via cf ssh, or login as root
> cf ssh <app-name>
or
> login to an app container as a root following How to login to an app container as a root in Tanzu Application Service for VMs
# Run below commands to connect to Redis instance from the app container and check connectivity
> nc <host> <port> # Host and port as per the service-key. This command won't print anything but waits for the user-input
> AUTH <password> # Password as per the service-key. This command should not print any error, expect OK as a response
> PING # should expect PONG as a response
Expected:
/# nc xx.xx.xx.xx 41565
auth #####
+OK
ping
+PONG
Actual:
/# nc xx.xx.xx.xx 41565
/# auth #####
bash: auth: command not found
Environment
Tanzu for Valkey on Cloud Foundry
Resolution
There is a network connectivity issue affecting the communication from the application container to the Redis server.
This issue does not appear to be related to Tanzu tile or the Cloud Foundry platform itself. It is recommended to investigate potential network configurations or firewall rules that may be restricting access and contributing to the observed behavior.
Feedback
