DB Connector Unable to Retrieve Chunks of Data
Article ID: 106300
Updated On:
Products
Rally On-Premise
Rally SaaS
Issue/Introduction
When running the connector the data load may be incomplete and the following is observed on the command line:
Additionally, the following error may be seen in the log file:
Retrying the page_urls for the page group startIndexes: 1001 -> 8001 Retrying the page_urls for the page group startIndexes: 1001 -> 8001 Retrying the page_urls for the page group startIndexes: 1001 -> 8001
Additionally, the following error may be seen in the log file:
[2018-07-12 20:03:04 Z] ERROR: DBConnector.run - RallyResponseError in DBConnector.run processing entity: Story -- Unable to retrieve 10 chunks of data
File "X:\dbeif\connector.py", line 184, in run
stats[entity] = self._processEntity(entity, csv_dir_name, populate_db, trash_csv)
File "X:\dbeif\connector.py", line 259, in _processEntity
entity_schema, pk_field = self.retrieveAndSaveAgileCentralData(entity)
File "X:\dbeif\utils\metrixeur.py", line 13, in wrapper
result = f(*args, **kwargs)
File "X:\dbeif\connector.py", line 226, in retrieveAndSaveAgileCentralData
= self.agicen_conn.getAgileCentralData(workspace_name, project, entity, project_scope_up, project_scope_down, self.remapping)
File "X:\dbeif\agile_central_connection.py", line 615, in getAgileCentralData
for item in response:
File "X:\Python\lib\site-packages\pyral\rallyresp.py", line 313, in __next__
self._page[:] = self.__retrieveNextPage()
File "X:\Python\lib\site-packages\pyral\rallyresp.py", line 410, in __retrieveNextPage
chapter = self.__retrievePages()
File "X:\Python\lib\site-packages\pyral\rallyresp.py", line 478, in __retrievePages
raise RallyResponseError("Unable to retrieve %d chunks of data" % num_threads)
<class 'pyral.rallyresp.RallyResponseError'> Unable to retrieve 10 chunks of data
[2018-07-12 20:03:04 Z] ERROR: DBConnectorRunner.run - Unable to retrieve 10 chunks of data
Environment
Release:
Component: ACSAAS
Component: ACSAAS
Cause
This is caused by a configuration file that is requesting too much data and/or complex data.
We have also found that using the Milestones object in the Fetch list can cause performance issues.
We have also found that using the Milestones object in the Fetch list can cause performance issues.
Resolution
- Break the entity types out into multiple config files. This is best practice per the documentation.
- Restrict the number of records pulled by adding a selector to restrict data based on an attribute such as CreationDate. Using an attribute like this, you can also split the selector data into multiple configs. For example in one config you might use CreationDate <= 2017-07-01 and in a second config you would use CreationDate > 2017-07-01
- Remove the Milestones field from the Fetch list
Feedback
Yes
No
Powered by
