Credhub Service Broker deploy-all errand fails with KeyError: 'guid'
search cancel

Credhub Service Broker deploy-all errand fails with KeyError: 'guid'

book

Article ID: 377117

calendar_today

Updated On:

Products

Operations Manager

Issue/Introduction

This article covers a specific failure case of the deploy-all errand in the Credhub Service Broker tile. 

Under certain circumstance, this errand can fail with output similar to the following:

===== 1970-01-01 00:00:00 UTC Running "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=127.0.0.1 --deployment=credhub-service-broker-1234567890abcdefghij run-errand deploy-all"
Using environment '127.0.0.1' as client 'ops_manager'
Using deployment 'credhub-service-broker-1234567890abcdefghij'
Task 1234567
...TRUNCATED...
Task 1234567 Duration 00:01:59
Task 1234567 done
Errand 'deploy-all' completed with error (exit code 1)
Exit code 1
Instance   deploy-all/abcdefgh-1234-abcd-1234-abcdefghijkl  
Exit Code  1  
Stdout     cf version 6.53.0+8e2b70a4a.2020-10-01  
...TRUNCATED...
          /tmp/setup /var/vcap/bosh  
          cf push --no-route -b binary_buildpack -p /tmp/setup -u process setup -c sleep infinity  
            
Stderr     Using cflinuxfs4 stack  
          Traceback (most recent call last):  
            File "<string>", line 1, in <module>  
          KeyError: 'guid'  
            
1 errand(s)

===== 1970-01-01 01:03:07 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=127.0.0.1 --deployment=credhub-service-broker-1234567890abcdefghij run-errand deploy-all"; Duration: 187s; Exit Status: 1
Exited with 1.
Exited with 1.

 

 

Cause

Please note the KeyError: 'guid' in the above output. This error occurs near the end of the deploy errand when the script creates an app named setup in the credhub-service-broker-org org,  credhub-service-broker-space space. This app exists solely to run a task that finalizes the deploy operation and the script does so by calling the V3 tasks endpoint of the Cloud Controller API (CAPI) directly instead of using cf run-task. It does this in part to retrieve the GUID for the task directly from the JSON response. The KeyError seen in the errand output is a failure in the parse operation caused when the JSON response does not include the GUID. This can occur if there are communication issues with CAPI due to network disruption, but for the purpose of this article the issue is caused due to a quota being set that limited the number of tasks that could be run per app.

To verify that the errand is failing due to a quota, perform the following steps:

  1. Log into the cf CLI as a user with administrative permissions
  2. Target the credhub-service-broker-org org,  credhub-service-broker-space space:
    cf target -o credhub-service-broker-org -s credhub-service-broker-space
  3. Check to see if the setup app is running using the cf apps command. This app will not be deleted in this particular failure scenario.
  4. If the app is running, attempt to run a simple task in the app:
    cf run-task setup -c "echo TEST TASK"
  5. If a quota is set, an error similar to the following will be seen:
    Creating task for app setup in org credhub-service-broker-org / space credhub-service-broker-space as user...
app_task_limit quota exceeded
FAILED

This message shows that a quota has been set to limit the number of tasks that can be run per app, and in this case it must be set to zero as there are no other tasks running for that app. This kind of quota cannot be set directly using the cf CLI, but can be set using the API. Additionally, the space where the Credhub Service Broker apps reside is managed by the platform and does not typically have a quota assigned. This means that either the default org quota has been modified to restrict the tasks per app, or another org quota has been applied to the org.

Resolution

In Cloud Foundry, a space quota cannot remove restrictions imposed by an organization quota. For the purposes of this article, it will be assumed that the desired per app task quota for the default organization quota is supposed to be zero; Therefore, a new quota will need to be created based on the existing applied quota and modified to allow at least one app per task.

IMPORTANT NOTE: The quota definitions provided in this article are intended to function as examples only. For compliance reasons, please make sure to retrieve and modify the existing quota in order to make sure that all other desired restrictions are in place.

  1. Log into the cf CLI as a user with administrative permissions
  2. Target the credhub-service-broker-org org,  credhub-service-broker-space space:
    cf target -o credhub-service-broker-org -s credhub-service-broker-space
  3. Find the name of the quota being applied to the org (e.g. no_tasks_org_quota):
    % cf org credhub-service-broker-org
Getting info for org ojoshua as admin...

name:                 credhub-service-broker-org
domains:              apps.internal, apps.example.com
quota:                no_tasks_org_quota
spaces:               credhub-service-broker-space
isolation segments:
  4. Use cf curl to retrieve the configuration of the current applied org quota (Replace no_tasks_org_quota in the names query parameter with the name found in the previous step):
    % cf curl '/v3/organization_quotas?names=no_tasks_org_quota&page=1&per_page=1'
{
  "pagination": {
...TRUNCATED...
  },
  "resources": [
    {
      "guid": "12345678-1234-1234-1234-123456789012",
      "created_at": "1970-01-01T15:03:00Z",
      "updated_at": "1970-01-01T15:12:18Z",
      "name": "no_tasks_org_quota",
      "apps": {
        "total_memory_in_mb": 10240,
        "per_process_memory_in_mb": null,
        "total_instances": null,
        "per_app_tasks": 0,
        "log_rate_limit_in_bytes_per_second": null
      },
      "services": {
        "paid_services_allowed": true,
        "total_service_instances": 100,
        "total_service_keys": null
      },
      "routes": {
        "total_routes": 1000,
        "total_reserved_ports": 0
      },
      "domains": {
        "total_domains": null
      },
      "relationships": {
...TRUNCATED...
      },
      "links": {
...TRUNCATED...
      }
    }
  ]
}
  5. Copy the contents of the resources array to a separate JSON file (e.g. one_task_org_quota.json)
  6. Remove the guid, created_at, updated_at, relationships, and links fields completely, including all children within any of the fields that are maps.
  7. Update the name field with a unique org quota name (e.g one_task_org_quota)
  8. Update the apps.per_app_tasks value to any nonzero integer or null to set it to unlimited. The resulting file would look something similar to the following if setting the value to one task per app:
    {
  "name": "one_task_org_quota",
  "apps": {
    "total_memory_in_mb": 10240,
    "per_process_memory_in_mb": null,
    "total_instances": null,
    "per_app_tasks": 1,
    "log_rate_limit_in_bytes_per_second": null
  },
  "services": {
    "paid_services_allowed": true,
    "total_service_instances": 100,
    "total_service_keys": null
  },
  "routes": {
    "total_routes": 1000,
    "total_reserved_ports": 0
  },
  "domains": {
    "total_domains": null
  }
}

     

  9. Use cf curl to create the task using the JSON file (Replace one_task_org_quota.json with the name of your JSON file, making sure to include the preceding @)

    % cf curl -X POST -i '/v3/organization_quotas' -d @one_task_org_quota.json
HTTP/1.1 201 Created
...TRUNCATED...
{
  "guid": "cff07d55-4bee-4dad-8918-a103b6b1fa90",
  "created_at": "1970-01-01T16:17:56Z",
  "updated_at": "1970-01-01T16:17:56Z",
  "name": "one_task_org_quota",
  "apps": {
    "total_memory_in_mb": 10240,
    "per_process_memory_in_mb": null,
    "total_instances": null,
    "per_app_tasks": 1,
    "log_rate_limit_in_bytes_per_second": null
  },
  "services": {
    "paid_services_allowed": true,
    "total_service_instances": 100,
    "total_service_keys": null
  },
  "routes": {
    "total_routes": 1000,
    "total_reserved_ports": 0
  },
  "domains": {
    "total_domains": null
  },
  "relationships": {
    "organizations": {
      "data": []
    }
  },
  "links": {
    "self": {
      "href": "https://api.system.example.com/v3/organization_quotas/cff07d55-4bee-4dad-8918-a103b6b1fa90"
    }
  }
}
  10. Make sure that the org quota is now visible in the output of cf org-quotas
  11. Set the org quota for the credhub-service-broker-org org to this new quota:
    cf set-org-quota credhub-service-broker-org one_task_org_quota
  12. Re-run the apply changes on the Credhub Service Broker tile with the deploy all errand enabled
Powered by Wolken Software