Hello All,
I have rebuilt the LAB in CML but the same issue happens in EVE-NG when add the vEgdes.
All vEdges can connect to vManage and get a certificate but C8K-BR3-1 after acquiring the certificate it loses connectivity to vManage and won’t connect again anymore.
After a thorough review of the configs I have realized this sis the only node that connects to the vManage using GigabitEthernet 1 (MPLS), all other nodes use Gigabitethernet 2 (Internet link).
I can ping vManage, vSmart, and vBond ip addresses from C8K-BR3-1 before and after adding it to vManage. below are the logs after adding the Chassis-ID and Serial Numbers.
I also checked and all nodes are in the same timezone and the clock is in synch even though the NTP service is not enabled.
*Jun 25 04:46:01.969: ESG: vManage connected
*Jun 25 04:46:01.610: %Cisco-SDWAN-C8K-BR3-1-vdaemon-6-INFO-1400002: Notification: 6/25/2024 4:46:1 control-connection-state-change severity-level:major host-name:“C8K-BR3-1” system-ip:10.2.2.231 personality:vedge peer-type:vbond peer-system-ip::: peer-vmanage-system-ip:0.0.0.0 public-ip:199.1.1.3 public-port:12346 src-color:default remote-color:default uptime:“0:00:00:00” new-state:up
*Jun 25 04:46:01.615: %Cisco-SDWAN-C8K-BR3-1-vdaemon-6-INFO-1400002: Notification: 6/25/2024 4:46:1 security-vsmart-entry-added severity-level:major host-name:“C8K-BR3-1” system-ip:10.2.2.231 serial:“6D8CC10F042D581760467C9F84EF1AE84CFDA316”
*Jun 25 04:46:01.619: %Cisco-SDWAN-C8K-BR3-1-vdaemon-6-INFO-1400002: Notification: 6/25/2024 4:46:1 security-vsmart-entry-added severity-level:major host-name:“C8K-BR3-1” system-ip:10.2.2.231 serial:“6D8CC10F042D581760467C9F84EF1AE84CFDA317”
*Jun 25 04:46:01.989: %Cisco-SDWAN-C8K-BR3-1-vdaemon-6-INFO-1400002: Notification: 6/25/2024 4:46:1 control-connection-state-change severity-level:major host-name:“C8K-BR3-1” system-ip:10.2.2.231 personality:vedge peer-type:vmanage peer-system-ip:10.1.1.101 peer-vmanage-system-ip:0.0.0.0 public-ip:199.1.1.1 public-port:12746 src-color:default remote-color:default uptime:“0:00:00:00” new-state:up
*Jun 25 04:46:02.220: %IOSXE-5-PLATFORM: R0/0: vip-bootstrap: CDB snapshotted in /var/confd0/cdb-backup.cfg, took 0 seconds
*Jun 25 04:46:02.429: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:34008 for netconf over ssh. External groups:
*Jun 25 04:46:11.126: %Cisco-SDWAN-C8K-BR3-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:46:11 security-install-csr severity-level:minor host-name:C8K-BR3-1 system-ip:10.2.2.231
*Jun 25 04:46:12.151: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:34058 for netconf over ssh. External groups:
*Jun 25 04:46:21.545: %Cisco-SDWAN-C8K-BR3-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:46:21 security-install-rcc severity-level:minor host-name:C8K-BR3-1 system-ip:10.2.2.231
*Jun 25 04:46:33.628: %Cisco-SDWAN-C8K-BR3-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:46:33 security-install-rcc severity-level:minor host-name:C8K-BR3-1 system-ip:10.2.2.231
*Jun 25 04:46:33.899: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:34114 for netconf over ssh. External groups:
*Jun 25 04:46:53.577: %Cisco-SDWAN-C8K-BR3-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:46:53 security-install-certificate severity-level:minor host-name:C8K-BR3-1 system-ip:10.2.2.231
*Jun 25 04:47:00.127: ESG: vManage disconnected
*Jun 25 04:47:00.131: %Cisco-SDWAN-C8K-BR3-1-vdaemon-6-INFO-1400002: Notification: 6/25/2024 4:47:0 control-connection-state-change severity-level:major host-name:“C8K-BR3-1” system-ip:10.2.2.231 personality:vedge peer-type:vmanage peer-system-ip:10.1.1.101 peer-vmanage-system-ip:0.0.0.0 public-ip:199.1.1.1 public-port:12746 src-color:default remote-color:default uptime:“0:00:00:58” new-state:down
*Jun 25 04:47:00.901: %Cisco-SDWAN-C8K-BR3-1-vdaemon-6-INFO-1400002: Notification: 6/25/2024 4:47:0 control-connection-state-change severity-level:major host-name:“C8K-BR3-1” system-ip:10.2.2.231 personality:vedge peer-type:vbond peer-system-ip::: peer-vmanage-system-ip:0.0.0.0 public-ip:199.1.1.3 public-port:12346 src-color:default remote-color:default uptime:“0:00:00:59” new-state:down
C8K-BR3-1#show sdwan control local-properties
personality vedge
sp-organization-name VALE
organization-name VALE
root-ca-chain-status Installed
root-ca-crl-status Not-Installed
certificate-status Not-Installed
certificate-validity Not Applicable
certificate-not-valid-before Not Applicable
certificate-not-valid-after Not Applicable
enterprise-cert-status Not Applicable
enterprise-cert-validity Not Applicable
enterprise-cert-not-valid-before Not Applicable
enterprise-cert-not-valid-after Not Applicable
dns-name 199.1.1.3
site-id 30
domain-id 1
protocol dtls
tls-port 0
system-ip 10.2.2.231
chassis-num/unique-id C8K-D0BF39AA-631A-DD6F-B20C-2E420028BEB4
serial-num No certificate installed
subject-serial-num N/A
enterprise-serial-num No certificate installed
token 5fc877ef97e249618967db8be061bba8
keygen-interval 1:00:00:00
retry-interval 0:00:00:19
no-activity-exp-interval 0:00:00:20
dns-cache-ttl 0:00:00:00
port-hopped TRUE
time-since-last-port-hop 0:00:01:59
embargo-check success
device-role edge-router
region-id-set N/A
mrf-migration-mode disabled
mrf-management-region no
number-vbond-peers 1
These are the logs of a healthy node:
*Jun 25 04:40:26.105: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:50706 for netconf over ssh. External groups:
*Jun 25 04:40:35.768: %Cisco-SDWAN-C8K-BR4-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:40:35 security-install-csr severity-level:minor host-name:C8K-BR4-1 system-ip:10.2.2.241
*Jun 25 04:40:36.736: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:50764 for netconf over ssh. External groups:
*Jun 25 04:40:45.481: %Cisco-SDWAN-C8K-BR4-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:40:45 security-install-rcc severity-level:minor host-name:C8K-BR4-1 system-ip:10.2.2.241
*Jun 25 04:40:58.529: %Cisco-SDWAN-C8K-BR4-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:40:58 security-install-rcc severity-level:minor host-name:C8K-BR4-1 system-ip:10.2.2.241
*Jun 25 04:40:58.736: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:50894 for netconf over ssh. External groups:
*Jun 25 04:41:14.951: %Cisco-SDWAN-C8K-BR4-1-action_notifier-6-INFO-1400002: Notification: 6/25/2024 4:41:14 security-install-certificate severity-level:minor host-name:C8K-BR4-1 system-ip:10.2.2.241
*Jun 25 04:42:08.243: %Cisco-SDWAN-Router-OMPD-3-ERRO-400002: vSmart peer 10.1.1.102 state changed to Init
*Jun 25 04:42:08.301: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:51174 for netconf over ssh. External groups:
*Jun 25 04:42:10.580: %Cisco-SDWAN-Router-OMPD-6-INFO-400002: vSmart peer 10.1.1.102 state changed to Handshake
*Jun 25 04:42:10.602: %Cisco-SDWAN-Router-OMPD-5-NTCE-400002: vSmart peer 10.1.1.102 state changed to Up
*Jun 25 04:42:10.603: %Cisco-SDWAN-Router-OMPD-6-INFO-400005: Number of vSmarts connected : 1
*Jun 25 04:42:11.860: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*Jun 25 04:42:13.481: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:51208 for netconf over ssh. External groups:
*Jun 25 04:42:27.708: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User ‘vmanage-admin’ authenticated successfully from 10.1.1.101:51262 for netconf over ssh. External groups:
C8K-BR4-1#show sdwan control local-properties
personality vedge
sp-organization-name VALE
organization-name VALE
root-ca-chain-status Installed
root-ca-crl-status Not-Installed
certificate-status Installed
certificate-validity Valid
certificate-not-valid-before Jun 25 04:40:35 2024 GMT
certificate-not-valid-after Jun 23 04:40:35 2034 GMT
enterprise-cert-status Not Applicable
enterprise-cert-validity Not Applicable
enterprise-cert-not-valid-before Not Applicable
enterprise-cert-not-valid-after Not Applicable
dns-name 199.1.1.3
site-id 40
domain-id 1
protocol dtls
tls-port 0
system-ip 10.2.2.241
chassis-num/unique-id C8K-C84E1DF8-C5E9-650E-8361-47DC9C4A1D04
serial-num 30F716A6
subject-serial-num N/A
enterprise-serial-num No certificate installed
token Invalid
keygen-interval 1:00:00:00
retry-interval 0:00:00:15
no-activity-exp-interval 0:00:00:20
dns-cache-ttl 0:00:02:00
port-hopped TRUE
time-since-last-port-hop 0:00:43:16
embargo-check success
device-role edge-router
region-id-set N/A
number-vbond-peers 1
