[VOL-2404] Issues with OLT hard Reboot (Physical reboot) - OpenCORD JIRA

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: VOLTHA v2.3
Component/s: Testing
Labels:
- stabilization

Story Points:
3
Epic Link:
Porting OpenOLT to Golang

Description

In a working system, OLT is power-cycled, without doing anything to VOLTHA/ONOS. NOTE: this is NOT olt-delete or olt-soft-reboot, where commands are actually sent via volthctl to voltha.

After all the subscribers have been successfully performed with auth+dhcp+pings, noticed following issues

OLT device never disconnected from ONOS during reboots
There was no change in the flows after the OLT device came back up
After the OLT device came back up, pings failed
Tried to perform authentication, and authentication failed too

Logs attached shows the status of the flows, outputs before and after reboot

==================================================

BEFORE REBOOT

===============

deviceId=of:0000b86a97071b4a, flowRuleCount=36

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40976, ETH_TYPE:eapol, VLAN_VID:33], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:4, metadata=METADATA:21004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:1048576, ETH_TYPE:lldp], treatment=[immediate=[OUTPUT:CONTROLLER]]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:32, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:68, UDP_DST:67], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:2, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:16, ETH_TYPE:ipv6, IP_PROTO:17, UDP_SRC:547, UDP_DST:546], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40976, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:68, UDP_DST:67], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:4, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40978, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40980, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:20, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:34, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:16, ETH_TYPE:eapol, VLAN_VID:11], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:b004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:35, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40977, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:17, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:1048576, ETH_TYPE:ipv6, IP_PROTO:17, UDP_SRC:546, UDP_DST:547], treatment=[immediate=[OUTPUT:CONTROLLER]]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:33, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:32, ETH_TYPE:ipv6, IP_PROTO:17, UDP_SRC:547, UDP_DST:546], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:2, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:32, ETH_TYPE:eapol, VLAN_VID:22], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:2, metadata=METADATA:16004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:18, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40976, ETH_TYPE:ipv6, IP_PROTO:17, UDP_SRC:547, UDP_DST:546], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:4, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:40979, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:19, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:36, ETH_TYPE:eapol, VLAN_VID:4091], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:ffb004000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:1048576, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:67, UDP_DST:68], treatment=[immediate=[OUTPUT:CONTROLLER]]

ADDED, bytes=0, packets=0, table=0, priority=10000, selector=[IN_PORT:16, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:68, UDP_DST:67], treatment=[immediate=[OUTPUT:CONTROLLER], meter=METER:1, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=0, priority=1000, selector=[IN_PORT:40976, VLAN_VID:0], treatment=[immediate=[VLAN_ID:33], transition=TABLE:1, meter=METER:4, metadata=METADATA:63004000100000/0]

ADDED, bytes=0, packets=0, table=0, priority=1000, selector=[IN_PORT:1048576, METADATA:b, VLAN_VID:99], treatment=[immediate=[VLAN_POP], transition=TABLE:1, meter=METER:1, metadata=METADATA:b004000000010/0]

ADDED, bytes=0, packets=0, table=0, priority=1000, selector=[IN_PORT:1048576, METADATA:16, VLAN_VID:99], treatment=[immediate=[VLAN_POP], transition=TABLE:1, meter=METER:3, metadata=METADATA:16004000000020/0]

ADDED, bytes=0, packets=0, table=0, priority=1000, selector=[IN_PORT:16, VLAN_VID:0], treatment=[immediate=[VLAN_ID:11], transition=TABLE:1, meter=METER:1, metadata=METADATA:63004000100000/0]

ADDED, bytes=0, packets=0, table=0, priority=1000, selector=[IN_PORT:32, VLAN_VID:0], treatment=[immediate=[VLAN_ID:22], transition=TABLE:1, meter=METER:2, metadata=METADATA:63004000100000/0]

ADDED, bytes=0, packets=0, table=0, priority=1000, selector=[IN_PORT:1048576, METADATA:21, VLAN_VID:99], treatment=[immediate=[VLAN_POP], transition=TABLE:1, meter=METER:4, metadata=METADATA:2100400000a010/0]

ADDED, bytes=0, packets=0, table=1, priority=1000, selector=[IN_PORT:32, VLAN_VID:22], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:99, OUTPUT:1048576], meter=METER:2, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=1, priority=1000, selector=[IN_PORT:1048576, VLAN_VID:11], treatment=[immediate=[VLAN_POP, VLAN_ID:0, OUTPUT:16], meter=METER:1, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=1, priority=1000, selector=[IN_PORT:1048576, VLAN_VID:33], treatment=[immediate=[VLAN_POP, VLAN_ID:0, OUTPUT:40976], meter=METER:4, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=1, priority=1000, selector=[IN_PORT:40976, VLAN_VID:33], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:99, OUTPUT:1048576], meter=METER:4, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=1, priority=1000, selector=[IN_PORT:16, VLAN_VID:11], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:99, OUTPUT:1048576], meter=METER:1, metadata=METADATA:4000000000/0]

ADDED, bytes=0, packets=0, table=1, priority=1000, selector=[IN_PORT:1048576, VLAN_VID:22], treatment=[immediate=[VLAN_POP, VLAN_ID:0, OUTPUT:32], meter=METER:3, metadata=METADATA:4000000000/0]

onos> aaa-usrs

Command not found: aaa-usrs

onos> aaa-users

UserName=user,CurrentState=AUTHORIZED,DeviceId=of:0000b86a97071b4a,MAC=90:E2:BA:8E:70:64,PortNumber=32,SubscriberId=ALPHe3d1cf9d-1

UserName=user,CurrentState=AUTHORIZED,DeviceId=of:0000b86a97071b4a,MAC=90:E2:BA:8E:70:65,PortNumber=16,SubscriberId=BRCM22222222-1

UserName=user,CurrentState=AUTHORIZED,DeviceId=of:0000b86a97071b4a,MAC=90:E2:BA:8E:70:66,PortNumber=40976,SubscriberId=ALPHe3d1ced5-1

onos> dhcpl2relay-allocations

SubscriberId=ALPHe3d1cf9d-1,ConnectPoint=of:0000b86a97071b4a/32,State=DHCPACK,MAC=90:E2:BA:8E:70:64,CircuitId=None,IP Allocated=10.99.22.2,Allocation Timestamp=2019-12-23T06:30:58.770Z

SubscriberId=ALPHe3d1ced5-1,ConnectPoint=of:0000b86a97071b4a/40976,State=DHCPACK,MAC=90:E2:BA:8E:70:66,CircuitId=None,IP Allocated=10.99.33.3,Allocation Timestamp=2019-12-23T06:27:04.352Z

SubscriberId=BRCM22222222-1,ConnectPoint=of:0000b86a97071b4a/16,State=DHCPACK,MAC=90:E2:BA:8E:70:65,CircuitId=None,IP Allocated=10.99.11.6,Allocation Timestamp=2019-12-23T06:30:13.726Z

onos> volt-programmed-subscribers

location=of:0000b86a97071b4a/16 subscriber=[id:BRCM22222222-1,cTag:11,sTag:99,nasPortId:BRCM22222222-1,uplinkPort:-1,slot:-1,hardwareIdentifier:null,ipaddress:null,nasId:null,circuitId:,remoteId:,technologyProfileId:64,upstreamBandwidthProfile:Default,downstreamBandwidthProfile:Default]

location=of:0000b86a97071b4a/40976 subscriber=[id:ALPHe3d1ced5-1,cTag:33,sTag:99,nasPortId:ALPHe3d1ced5-1,uplinkPort:-1,slot:-1,hardwareIdentifier:null,ipaddress:null,nasId:null,circuitId:,remoteId:,technologyProfileId:64,upstreamBandwidthProfile:User_Bandwidth3,downstreamBandwidthProfile:User_Bandwidth3]

location=of:0000b86a97071b4a/32 subscriber=[id:ALPHe3d1cf9d-1,cTag:22,sTag:99,nasPortId:ALPHe3d1cf9d-1,uplinkPort:-1,slot:-1,hardwareIdentifier:null,ipaddress:null,nasId:null,circuitId:,remoteId:,technologyProfileId:64,upstreamBandwidthProfile:User_Bandwidth1,downstreamBandwidthProfile:User_Bandwidth2]

onos> ports -e

id=of:0000000000000001, available=true, local-status=connected 19m45s ago, role=MASTER, type=SWITCH, mfr=Accton Corp., hw=x86-64-accton-as6712-32x-r0, sw=ofdpa 3.0.5.5+accton1.7-1, serial=671232X1538033, chassis=1, driver=ofdpa3, channelId=10.233.102.128:38970, locType=none, managementAddress=10.233.102.128, name=of:0000000000000001, protocol=OF_13

port=1, state=enabled, type=fiber, speed=40000 , adminState=enabled, portMac=cc:37:ab:61:80:49, portName=port1

port=31, state=enabled, type=fiber, speed=40000 , adminState=enabled, portMac=cc:37:ab:61:80:49, portName=port31

id=of:0000b86a97071b4a, available=true, local-status=connected 8m3s ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=open_pon, sw=open_pon, serial=EC1904000654, chassis=b86a97071b4a, driver=voltha, channelId=10.233.102.168:41372, managementAddress=10.233.102.168, protocol=OF_13

port=16, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:10, portName=BRCM22222222-1

port=17, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:11, portName=BRCM22222222-2

port=18, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:12, portName=BRCM22222222-3

port=19, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:13, portName=BRCM22222222-4

port=20, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:14, portName=BRCM22222222-5

port=32, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:20, portName=ALPHe3d1cf9d-1

port=33, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:21, portName=ALPHe3d1cf9d-2

port=34, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:22, portName=ALPHe3d1cf9d-3

port=35, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:23, portName=ALPHe3d1cf9d-4

port=36, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:00:00:00:24, portName=ALPHe3d1cf9d-5

port=40976, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:0a:00:a0:10, portName=ALPHe3d1ced5-1

port=40977, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:0a:00:a0:11, portName=ALPHe3d1ced5-2

port=40978, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:0a:00:a0:12, portName=ALPHe3d1ced5-3

port=40979, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:0a:00:a0:13, portName=ALPHe3d1ced5-4

port=40980, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=08:00:0a:00:a0:14, portName=ALPHe3d1ced5-5

port=1048576, state=enabled, type=fiber, speed=0 , adminState=enabled, portMac=b8:6a:97:07:1b:4a, portName=nni-1048576

onos> devices

id=of:0000000000000001, available=true, local-status=connected 19m53s ago, role=MASTER, type=SWITCH, mfr=Accton Corp., hw=x86-64-accton-as6712-32x-r0, sw=ofdpa 3.0.5.5+accton1.7-1, serial=671232X1538033, chassis=1, driver=ofdpa3, channelId=10.233.102.128:38970, locType=none, managementAddress=10.233.102.128, name=of:0000000000000001, protocol=OF_13

id=of:0000b86a97071b4a, available=true, local-status=connected 8m11s ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=open_pon, sw=open_pon, serial=EC1904000654, chassis=b86a97071b4a, driver=voltha, channelId=10.233.102.168:41372, managementAddress=10.233.102.168, protocol=OF_13

onos>

==================

REBOOTED the OLT

onos> devices

id=of:0000000000000001, available=true, local-status=connected 24m15s ago, role=MASTER, type=SWITCH, mfr=Accton Corp., hw=x86-64-accton-as6712-32x-r0, sw=ofdpa 3.0.5.5+accton1.7-1, serial=671232X1538033, chassis=1, driver=ofdpa3, channelId=10.233.102.128:38970, locType=none, managementAddress=10.233.102.128, name=of:0000000000000001, protocol=OF_13

id=of:0000b86a97071b4a, available=true, local-status=connected 12m33s ago, role=MASTER, type=SWITCH, mfr=VOLTHA Project, hw=open_pon, sw=open_pon, serial=EC1904000654, chassis=b86a97071b4a, driver=voltha, channelId=10.233.102.168:41372, managementAddress=10.233.102.168, protocol=OF_13

onos>

============

AFTER REBOOT

=====