Delete device leaves stale entries in ONOS

Delete device leaves stale entries in ONOS

• Seen in this job https://jenkins.opencord.org/view/VOLTHA-2.X-Tests/job/build_flex-ocp-cord_TP_voltha_TT_master_test/439

Looking at the logs, the adapter respond with a deletedevice success with the oltadapter responding with a disable device failure after having sent a deletesuccess. There is a race condition between delete and disable. We may need to change the logic such that the device state change to disable is done after an adapter response but not before. We may need intermediate state (e.g in-progress) to handle case where we do not want any action on the device if there is something in progress (or the core can use it’s internal transient state). In any case, i do not see this as an issue, unless the olt adapter gets in an odd situation of a disable device failure but with a delete device success.

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:36.447Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"74ee3c32dd212dd3","op-name":"/voltha.VolthaService/DisableDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","state-change":"","status":"OPERATION_IN_PROGRESS","description":"","device-update":"disableDevice","device-update-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","requested-by":"NB"}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:36.726Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","status":"OPERATION_IN_PROGRESS","description":"","device-update":"deleteDevice","device-update-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","requested-by":"NB","state-change":""}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.173Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","requested-by":"NB","state-change":"","status":"OPERATION_IN_PROGRESS","description":"","device-update":"deleteDevice","device-update-id":"e236ca9b-2340-4d3c-8092-56b6d782cc77"}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.188Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","device-update":"deleteDevice","device-update-id":"ff4e2927-817a-439d-be86-189757c060fb","requested-by":"NB","state-change":"","status":"OPERATION_IN_PROGRESS","description":""}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.198Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","status":"OPERATION_IN_PROGRESS","description":"","device-update":"deleteDevice","device-update-id":"bc88540e-74d1-48f7-b3c8-419b0493a051","requested-by":"NB","state-change":""}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.218Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","state-change":"","status":"OPERATION_SUCCESS","description":"","device-update":"deleteDevice","device-update-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","requested-by":"NB"}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.234Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","requested-by":"NB","state-change":"","status":"OPERATION_SUCCESS","description":"","device-update":"deleteDevice","device-update-id":"e236ca9b-2340-4d3c-8092-56b6d782cc77"}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.248Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","device-update":"deleteDevice","device-update-id":"ff4e2927-817a-439d-be86-189757c060fb","requested-by":"NB","state-change":"","status":"OPERATION_SUCCESS","description":""}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]:

{"level":"info","ts":"2021-09-07T20:22:37.249Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"7398133efa70f8c5","op-name":"/voltha.VolthaService/DeleteDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","state-change":"","status":"OPERATION_SUCCESS","description":"","device-update":"deleteDevice","device-update-id":"bc88540e-74d1-48f7-b3c8-419b0493a051","requested-by":"NB"}

voltha/voltha-voltha-rw-core-849bdfd96c-lntbf[voltha]: {"level":"info","ts":"2021-09-07T20:22:37.636Z","caller":"device/agent_device_update.go:36","msg":"logDeviceUpdate","instanceId":"voltha-voltha-rw-core-849bdfd96c-lntbf","op-id":"74ee3c32dd212dd3","op-name":"/voltha.VolthaService/DisableDevice","device-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","status":"OPERATION_FAILURE","description":"rpc error: code = Internal desc = rpc error: code = NotFound desc = ports-state-update-failed: [

{\"device-id\":\"a741fce8-237f-4dfd-b3e5-fc22f067986a\"}

]: rpc error: code = Canceled desc = rpc error: code = NotFound desc = a741fce8-237f-4dfd-b3e5-fc22f067986a","device-update":"disableDevice","device-update-id":"a741fce8-237f-4dfd-b3e5-fc22f067986a","requested-by":"NB","state-change":"ENABLED->DISABLED"}

The core sends the disable device request to the adapters and sends an Ack back to NB. NB then issues a delete device that the core proceeds as well. It’s just that the delete device is received in the Core before the device is fully disabled by the adapter. The core cannot really order these two events other than creating a temporary state for the device when it is disabled. Since a deletedevice can be sent to the core without a disable then this becomes a little bit tricky to manage.