-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Medium
-
Resolution: Done
-
Affects Version/s: VOLTHA v2.12
-
Fix Version/s: VOLTHA v2.12
-
Component/s: openolt-adapter, openolt-agent
-
Labels:
-
Story Points:0
This pipeline shows the failing test for "Verify OLT after Rebooting Physically for DT" (second test inside the FailureScenarios test suite): https://jenkins.opencord.org/view/Community-PODs/job/build_berlin-community-pod-1-gpon_1T8GEM_voltha_DT_master_test/53/
I've also manually tested this in the berlin-community-pod-1 nodes and the result is the same, leading me to think it's a bug.
The steps to reproduce it are the following:
1. Build the test environment as usual, following the steps done for the automated pipelines (install voltha, push the tech-profile and other configurations as necessary and reinstall the OLT software)
2. After that is done, I create the device with voltctl:
voltctl -c /home/jenkins/.volt/config-minimal device create -t openolt -H 10.34.90.50:9191
3. Then I enable the device:
voltctl -c /home/jenkins/.volt/config-minimal device enable <DEVICE-ID>
4. Check the device is ENABLED, ACTIVE and REACHABLE:
voltctl -c /home/jenkins/.volt/config-minimal device list
5. Check the device appears in ONOS:
curl -H 'Authorization: Basic a2FyYWY6a2FyYWY=' http://10.34.90.43:30120/onos/v1/devices {"devices":[{"id":"of:000080a2354d2fc0","type":"SWITCH","available":false,"role":"MASTER","mfr":"VOLTHA Project","hw":"open_pon","sw":"open_pon","serial":"EC1912001823","driver":"voltha","chassisId":"80a2354d2fc0","lastUpdate":"1694021506893","humanReadableLastUpdate":"disconnected 59s ago","annotations":{"channelId":"10.42.2.148:47664","managementAddress":"10.42.2.148","protocol":"OF_13"}}]}
6. Check that the ports list is not empty and that there's a port for each ONU discovered:
curl -H 'Authorization: Basic a2FyYWY6a2FyYWY=' http://10.34.90.43:30120/onos/v1/devices/of:000080a2354d2fc0/ports {"id":"of:000080a2354d2fc0","type":"SWITCH","available":true,"role":"STANDBY","mfr":"VOLTHA Project","hw":"open_pon","sw":"open_pon","serial":"EC1912001823","driver":"voltha","chassisId":"80a2354d2fc0","lastUpdate":"1694021024239","humanReadableLastUpdate":"connected 5m37s ago","annotations":{"channelId":"10.42.2.148:47664","managementAddress":"10.42.2.148","protocol":"OF_13"},"ports":[{"element":"of:000080a2354d2fc0","port":"16777216","isEnabled":true,"type":"fiber","portSpeed":100000,"annotations":{"adminState":"enabled","portMac":"80:a2:35:4d:2f:c0","portName":"nni-16777216"}},{"element":"of:000080a2354d2fc0","port":"256","isEnabled":true,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:00:00:01:00","portName":"SCOM00001c96-1"}},{"element":"of:000080a2354d2fc0","port":"512","isEnabled":true,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:00:00:02:00","portName":"SCOM00001c0b-1"}},{"element":"of:000080a2354d2fc0","port":"1024","isEnabled":true,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:00:00:04:00","portName":"SCOM00001c89-1"}},{"element":"of:000080a2354d2fc0","port":"768","isEnabled":true,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:00:00:03:00","portName":"SCOMa0305f84-1"}},{"element":"of:000080a2354d2fc0","port":"65793","isEnabled":false,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:01:01:01:01","portName":"ADTN21133f28-2"}},{"element":"of:000080a2354d2fc0","port":"65792","isEnabled":true,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:01:01:01:00","portName":"ADTN21133f28-1"}},{"element":"of:000080a2354d2fc0","port":"4129024","isEnabled":true,"type":"copper","portSpeed":1,"annotations":{"adminState":"enabled","portMac":"08:00:3f:3f:01:00","portName":"SCOM00001c82-1"}}]}
7. Reboot the OLT (SSH into it and use command reboot), and wait a little until the /var/logs/openolt.log file shows a line saying the server is listening on 0.0.0.0:9191.
8. Perform checks 4, 5 and 6 again. This time, check 6 will NOT BE OK. The ports list will be empty. In order for it to not be empty and to recover the previous state, the OLT device must be first disabled and re-enabled.
curl -H 'Authorization: Basic a2FyYWY6a2FyYWY=' http://10.34.90.43:30120/onos/v1/devices/of:000080a2354d2fc0/ports {"id":"of:000080a2354d2fc0","type":"SWITCH","available":false,"role":"MASTER","mfr":"VOLTHA Project","hw":"open_pon","sw":"open_pon","serial":"EC1912001823","driver":"voltha","chassisId":"80a2354d2fc0","lastUpdate":"1694021506893","humanReadableLastUpdate":"disconnected 1m6s ago","annotations":{"channelId":"10.42.2.148:47664","managementAddress":"10.42.2.148","protocol":"OF_13"},"ports":[]}
voltctl -c /home/jenkins/.volt/config-minimal device disable <DEVICE-ID> voltctl -c /home/jenkins/.volt/config-minimal device enable <DEVICE-ID>
Guessing by the Robot test, it shouldn't be necessary to disable and re-enable the device in order for the state to be recovered properly, so that leads me to believe it could be a bug. I'm not sure internally what the problem could be tho.
