Missing ACTIVE indication from OLT adapter during rolling upgrade

https://onf-community.slack.com/archives/C027XNKJR6Y/p1649348065952169

Notes from Khen

I looked at the artifacts of the above failures. Here's what I suspect could be happening:

• The OLT adapter restarts and the grpc streams between the adapter and Core are established. The core sends a reconcile request to the openolt adapter to which the openolt adapter responds successfully, i.e. it could initiates the reconcile.
• I did not see in the core logs a device state update from the openolt adapter indicating that the device operational status is active. This means that the device is still in reconciling state.
• A minute later, the ONU adapter restarts and the grpc streams between the adapter and Core are established. The core sends a reconcile request to the openonu adapter.
• The openonu adapter responds with a failure to reconcile message indicating that the device's parent adapter (in this care the openolt adapter) has not established a grpc streaming connection to it. We have this check in place to make sure the OMCI responses can be sent from the openolt adapter to the openonu adapter following an OMCI request from the openonu adapter.
• Looking at the logs, the openolt adapter did not attempt (after its restart) to establish a grpc stream to the openonu adapter.
• The openolt adapter creates its stream to the openonu adapter when the adapter has to send a request (or an indication) to the openonu adapter.
• There could be multiple things that went wrong. The first thing to investigate is to see why the openolt adapter did not send a device operational state of ACTIVE to the core following a reconcile request. It itself may need some indications from BBSIM in order to do that.