Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
c8kv reboots and template attach fails with "access denied"
c8kv deployed on Nfvis / ENCS 4.6.4 and c8kv is upgraded and template attached
Currently, WA is to software reset the device to restore functionality on 17.6.4/17.6.5 but this is not acceptable from the customer as they have hundreds of C8000v routers they need to upgrade. --------------------------------------x---------------------------------------------------------------------- 1. Revert the device to 17.6.1a 2. Removed the 17.6.5 image and reboot the device. 3. Upgrade device again to 17.6.5 4. disable the polling or increase the timeout for the tool that does the reboot if the upgrade is not done within some particular time
When they perform upgrade of the C8000vs to SDWAN IOS-XE 17.6.4 or 17.6.5 sometimes template push is failing with error "access denied (3): access denied" The behavior seems to be inconsistent with some routers failing template push after upgrade and some not. For example, customer upgrades 4 C8000vs at a time - Two of the four routers will fail the template push after the upgrade with same "access denied (3): access denied" The changes customer makes to the templates after the upgrade are small like simple port change to LAN interface from "no shut" to "shut" C8000v device hostname dlchqsdw002 failed attempts: 2023-04-17 06:15:07 BRT show config-pull history detail config-pull events 1 timestamp "2023-04-12 11:52:53" transaction-id push_feature_template_configuration-626a110a-723d-46c9-b870-fcbd99a5dcc1%C8K-C5CC0C9D-D338-309A-F964-3F5938B4521F%ccbd4933-67c7-47de-a726-193130e91381 process-name pycfg-26073 total-time 0:00:20.557102 result failure fail-reason commit-failure fail-message "access denied (3): access denied" config-pull events 2 timestamp "2023-04-17 06:12:01" transaction-id push_feature_template_configuration-18cdb547-f184-4a03-b200-8914dfbad55d%C8K-C5CC0C9D-D338-309A-F964-3F5938B4521F%ccbd4933-67c7-47de-a726-193130e91381 process-name pycfg-15638 total-time 0:00:28.836707 result failure fail-reason commit-failure fail-message "access denied (3): access denied" 1. Device is using template name "dl_c8Kv_encs_2-inet_ame" a. We tried removing all non-essential configuration from the device but still template failed with same error b. Same device template was used with other working routers on same code without issue c. No "CLI-Addon" template is being used 2. Tried using CLI based template but this fails also 3. Reboot of vManage and controllers does not resolve issue 4. Reboot of the device does not resolve issue 5. Lastly, a software reset of the device DOES resolve the issue. Currently, WA is to software reset the device to restore functionality on 17.6.4/17.6.5 but this is not acceptable from the customer as they have hundreds of C8000v routers they need to upgrade. We have setup a lab with customers "config-db" using a copy of the customer template "dl_c8Kv_encs_2-inet_ame" with similar configurations. However, we were not able to replicate the issue in lab following same order of steps below: 1. Onboard C8000v with 17.6.1 code 2. Attach customer template 3. Upgrade C8000v to 17.6.5 4. Successful template push was made Customer vManage is on 20.6.5 10-Aug-2023::10:11:12.379 dlchqsdw001 confd[29815]: devel-aaa User: vmanage-admin[vmanage-admin,85] rejected data access path /ios:native/service/internal op read due to rule "vmanage-admin/deny-Cisco-IOS-XE-native-1" 10-Aug-2023::10:11:12.379 dlchqsdw001 confd[29815]: devel-aaa User: vmanage-admin[vmanage-admin,85] rejected data access path /ios:native/service/pad-config-more op read due to rule "vmanage-admin/deny-Cisco-IOS-XE-native-1" 10-Aug-2023::10:11:12.379 dlchqsdw001 confd[29815]: devel-aaa User: vmanage-admin[vmanage-admin,85] rejected data access path /ios:native/service/pad op read due to rule "vmanage-admin/deny-Cisco-IOS-XE-native-1" 10-Aug-2023::10:11:12.379 dlchqsdw001 confd[29815]: devel-aaa User: vmanage-admin[vmanage-admin,85] rejected data access path /ios:native/service/password-encryption op read due to rule "vmanage-admin/deny-Cisco-IOS-XE-native-1" 10-Aug-2023::10:11:12.380 dlchqsdw001 confd[29815]: devel-aaa User: vmanage-admin[vmanage-admin,85] rejected data access path /ios:native/service/private-config-encryption op read due to rule "vmanage-admin/deny-Cisco-IOS-XE-native-1" 10-Aug-2023::10:11:12.380 dlchqsdw001 confd[29815]: devel-aaa User: vmanage-admin[vmanage-admin,85] rejected data access path /ios:native/service/timestamps op read due to rule "vmanage-admin/deny-Cisco-IOS-XE-native-1" Per Manish analysis: Here is the reason why access denied is seen. This implies that vManage is trying to access some config which is hidden from it. Customer confirmed there is no CLI Add Template being used for this Device Template. They used bare bones configuration with just essential feature templates but still received the same error. In that case, someone from vManage team needs to check why is /native/service is being sent when it is supposed to be hidden from vManage