Symptoms
Unable to add new authentication provider using UI.Gets the following error message when attempting to add it:Error 30026 (http: 503): Service is busy. The Data services are busy. Try again later.
Cause
Vnest key in transition caused DT unready result in adding authentication provider failed.
Resolution
1. Check /opt/emc/caspian/fabric/agent/services/object/main/log/provisionsvc.log* around the issue timestamp with keyword REQUEST_AUTHPROVIDER on all nodes. In this case, found error below, which cannot reach remote VDC:
016-09-19 10:46:00,672 [pool-69-thread-1] ERROR ResourceClient.java (line 322) request id 0a7e0c8e:1569e3f4366:22:2404, command type REQUEST_AUTHPROVIDER_CREATE failed with error code ERROR_TIMEOUT and message 'failed to get enough responses voters [urn:storageos:VirtualDataCenterData:xxxxxxx-4385-4484-b13a-1cd0cb159c94, urn:storageos:VirtualDataCenterData:xxxxxxx-0e80-490d-82bc-093895c796b9], responded [urn:storageos:VirtualDataCenterData:xxxxxxx-4385-4484-b13a-1cd0cb159c94], failed to reach [urn:storageos:VirtualDataCenterData:xxxxxx-0e80-490d-82bc-093895c796b9], type AUTH_PROVIDER_KEY, request 0a7e0c8e:1569e3f4366:22:2404'
2. Check DT status, found unready
#curl http://`hostname -i`:9101/stats/dt/DTInitStat/ | grep ""
curl http://`hostname -i`:9101/stats/dt/DTInitStat/ | grep ""
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 188 100 188 0 0 12 0 0:00:15 0:00:15 --:--:-- 48
204800
3. Check vnest-error.log, if there is "vnest key in transition"
Example:
# viprexec -i 'grep -i "key in transition" /opt/emc/caspian/fabric/agent/services/object/main/log/vnest-error.log' Output from host : 192.168.219.1 com.emc.storageos.data.object.exception.ObjectControllerException: key in transition, request 0a7e9511:156c26c97a7:13a:23533 ip 10.126.149.xx:9204
4. If seeing this issue open an SR with ECS support, reference this KB and post the output of the logs.
