ONU Device reconciliation fails randomly

When running the ONU device reconciliation tests, following an ONU adapter restart, the reconciliation fails randomly. One of the reason of this failure is related to the scenario below:

• ONU adapter restarts
• ONU adapter connects to the Core
• The Core connects to the ONU adapter and invoke the reconcile() API of that adapter for each ONU device.
• At the same time the OLT adapter tries to re-establish its connection to the ONU adapter.
• Upon reception of the reconcile() requests, the ONU adapter starts to reconcile the ONU device passed as a parameter. This results in a number of OMCI messages being sent by the ONU adapter to the OLT adapter to proxy to the actual ONU device.
• When the OLT device receives the OMCI response it sends back that response to the ONU adapter. If the OLT adapter was able to establish a grpc connection to the ONU adapter then the response will go through. Otherwise, it fails, prompting the ONU adapter to retry. If the second try fails then the reconciliation fails.

It should be noted that all proxy messages from the ONU adapter to the OLT adapter are asynchronous, i.e. the openonu adapter sends a request to the openonu adapter and gets an Ack back after the openolt sends the request to the actual device. When a response is received from the device then the OLT adapter sends the response back to the ONU adapter using the grpc client it has established to the openonu adapter.

This Jira is opened to address this issue.