Error Message "RPC Timed out" Occurs When Attempting to Change XOS Configuration Items

book

Article ID: 167890

calendar_today

Updated On:

Products

XOS

Issue/Introduction

When attempting to make an XOS configuration item change, you receive the error "SYS-ERR: RPC failed. Detail: RPC: Timed out".

Cause

This could be related to a "calendar" command issued earlier, using the XOS CLI, after which the system has not been reloaded completely ( using the reload all command).

This is the sequence of events that occurs on the system automatically (done by the CPM cbsd daemon ) when one is using  the calendar command :

1. Stop ntpd on the primary CPM and all VAPs
2. Set the clock
3. Start nptd on primary CPM and all VAPs
4. Reload chassis

If, for any reason, the process of stopping the ntpd on the VAPs (rsh <VAP> /etc/rc.d/init.d/ntpd stop) takes too long ( because the VAP is busy or not responsive ) , the cbsd process is held busy, causing the RPC timeout .

The timeout occurs because cbsd is an RPC server, and all subsequent RPC calls will time out because cbsd is still busy stopping the NTP process on the VAPs.

Messages similar to the following may be noted in the logs:

ct 17 13:53:41 cb01 cbshmonitord: [N] Violation (s=1, alarm) cleared:
module:13, item:1601 (H_ID_NTPD_RUNNING), time:"Mon Oct 17 13:53:41 2011"
Oct 17 13:53:41 cb01 cbsalarmlogrd: AlarmID 19439 | Mon Oct 17 13:53:41 2011 |
clear | cp1 | ntpServiceFailure | NTP service failure | CorrelationID 19438

Oct 17 13:54:25 cb01 ntpd[11067]: system event 'event_peer/strat_chg' (0x04)
status 'sync_alarm, sync_local_proto, 2 events, event_restart' (0xc521)
Oct 17 13:54:25 cb01 ntpd[11067]: synchronized to LOCAL(0), stratum 10
Oct 17 13:54:25 cb01 ntpd[11067]: kernel time sync disabled 0001
Oct 17 13:54:25 cb01 ntpd[11067]: system event 'event_sync_chg' (0x03) status
'leap_none, sync_local_proto, 3 events, event_peer/strat_chg' (0x534)
Oct 17 13:54:25 cb01 ntpd[11067]: system event 'event_peer/strat_chg' (0x04)
status 'leap_none, sync_local_proto, 4 events, event_sync_chg' (0x543)

Oct 17 13:58:53 cb01 cli: [E] RPC error RPC: Timed out
Oct 17 13:58:53 cb01 cli: [E] Client call to management daemon failed with
error RPC: Timed out
Oct 17 13:58:53 cb01 cli: [E] RPC error RPC: Timed out

Resolution

Make sure the system is reloaded after the calendar command is issued.  Failure to reload the system can cause the system to become unstable and it is no longer be possible to make configuration changes.