System reconfiguration issue after ONU reboot

Local VBox tests for ONU upgrade with BBSIM (1 ONU). Upgrade tests are connected with ONU reboot (ONU operState down/up). After the ONU (BBSIM) gets up again, multiple errors appear at flow/techProfile configuration. In the end, from OnuAdapter perspective, it looks like the system is an some endless loop to try to configure and delete tech profile related configuration.

I could not correlate this behavior to any specific behavior of the OnuAdapter.

According info level log files attached. Fist related error log is visible in rwCore:

{"level":"error","ts":"2021-03-16T14:35:52.588Z","caller":"device/manager.go:128","msg":"loading-device-failed","instanceId":"voltha-voltha-rw-core-55754d6854-jn9dn","task-id":"797b655bf4a31ccf","task-name":"/voltha.VolthaService/UpdateLogicalDeviceFlowTable","op-id":"288a4fd2b3fe1d33","op-name":"openflow-flow-add","device-id":"","error":"rpc error: code = InvalidArgument desc = deviceId empty","stacktrace":"github.com/opencord/voltha-lib-go/v4/pkg/log.clogger.Errorw\n\t/go/src/vendor/github.com/opencord/voltha-lib-go/v4/pkg/log/log.go:611\ngithub.com/opencord/voltha-go/rw_core/core/device.(*Manager).getDeviceAgent\n\t/go/src/rw_core/core/device/manager.go:128\ngithub.com/opencord/voltha-go/rw_core/core/device.(*Manager).getDeviceReadOnly\n\t/go/src/rw_core/core/device/manager.go:340\ngithub.com/opencord/voltha-go/rw_core/flowdecomposition.(*FlowDecomposer).updateOutputPortForControllerBoundFlowForParentDevide\n\t/go/src/rw_core/flowdecomposition/flow_decomposer.go:82\ngithub.com/opencord/voltha-go/rw_core/flowdecomposition.(*FlowDecomposer).decomposeFlow\n\t/go/src/rw_core/flowdecomposition/flow_decomposer.go:537\ngithub.com/opencord/voltha-go/rw_core/flowdecomposition.(*FlowDecomposer).DecomposeRules\n\t/go/src/rw_core/flowdecomposition/flow_decomposer.go:59\ngithub.com/opencord/voltha-go/rw_core/core/device.(*LogicalAgent).decomposeAndAdd\n\t/go/src/rw_core/core/device/logical_agent_flow.go:158\ngithub.com/opencord/voltha-go/rw_core/core/device.(*LogicalAgent).flowAdd\n\t/go/src/rw_core/core/device/logical_agent_flow.go:87\ngithub.com/opencord/voltha-go/rw_core/core/device.(*LogicalAgent).updateFlowTable\n\t/go/src/rw_core/core/device/logical_agent_flow.go:59\ngithub.com/opencord/voltha-go/rw_core/core/device.(*LogicalManager).UpdateLogicalDeviceFlowTable\n\t/go/src/rw_core/core/device/logical_manager.go:464\ngithub.com/opencord/voltha-protos/v4/go/voltha._VolthaService_UpdateLogicalDeviceFlowTable_Handler.func1\n\t/go/src/vendor/github.com/opencord/voltha-protos/v4/go/voltha/voltha.pb.go:3547\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1\n\t/go/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31\ngithub.com/opencord/voltha-lib-go/v4/pkg/grpc.mkServerInterceptor.func1\n\t/go/src/vendor/github.com/opencord/voltha-lib-go/v4/pkg/grpc/server.go:155\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1\n\t/go/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34\ngithub.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1\n\t/go/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing/server_interceptors.go:30\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:39\ngithub.com/opencord/voltha-protos/v4/go/voltha._VolthaService_UpdateLogicalDeviceFlowTable_Handler\n\t/go/src/vendor/github.com/opencord/voltha-protos/v4/go/voltha/voltha.pb.go:3549\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/src/vendor/google.golang.org/grpc/server.go:1007\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/src/vendor/google.golang.org/grpc/server.go:1287\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/src/vendor/google.golang.org/grpc/server.go:722"}

Subsequently more errors and also in OltAdapter, e.g.:

{"level":"error","ts":"2021-03-16T14:35:52.595Z","caller":"core/openolt_flowmgr.go:376","msg":"no-onu-id-for-flow","instanceId":"open-olt-voltha-adapter-openolt-7958c5cf88-cbctx","op-id":"288a4fd2b3fe1d33","op-name":"openflow-flow-add","task-id":"2bb0ee90cb70a1b0","task-name":"kafka-rpc-update_flows_incrementally","classifer":\{"eth_type":34958,"in_port":1,"tunnel_id":1,"vlan_vid":4091}

,"action":{"output":2147483645},"device-id":"8b905195-ff8d-4c3b-9856-a102eb38bd2a","onu":0,"port-no":1,"stacktrace":"github.com/opencord/voltha-lib-go/v4/pkg/log.clogger.Errorw\n\t/go/src/github.com/opencord/voltha-openolt-adapter/vendor/github.com/opencord/voltha-lib-go/v4/pkg/log/log.go:611\ngithub.com/opencord/voltha-openolt-adapter/internal/pkg/core.(*OpenOltFlowMgr).processAddFlow\n\t/go/src/github.com/opencord/voltha-openolt-adapter/internal/pkg/core/openolt_flowmgr.go:376\ngithub.com/opencord/voltha-openolt-adapter/internal/pkg/core.(*OpenOltFlowMgr).AddFlow\n\t/go/src/github.com/opencord/voltha-openolt-adapter/internal/pkg/core/openolt_flowmgr.go:2348\ngithub.com/opencord/voltha-openolt-adapter/internal/pkg/core.(*OpenOltFlowMgr).perOnuFlowHandlerRoutine\n\t/go/src/github.com/opencord/voltha-openolt-adapter/internal/pkg/core/openolt_flowmgr.go:2243"}