No subscriber flows added after volt-add-sub call, Exceptions thrown in ONOS

Setup: ONOS:3 instances, Atomic:3 

After a successful authentication on the ONU, when a volt-add-subscriber-access call is made, there are no subscriber flows added on the ONU. Trying to retrieve flows for the device in ONOS threw following exceptions

Problem was seen twice on the QA POD

Logs can be found here

https://jenkins.opencord.org/view/VOLTHA-2.X-Tests/job/build_onf-demo-pod_Default_voltha_master_test/341/robot/report/FunctionalTests/log.html Delete OLT, ReAdd OLT and Perform Sanity Test

https://jenkins.opencord.org/view/VOLTHA-2.X-Tests/job/build_onf-demo-pod_Default_voltha_master_test/341/ Logfile: DeleteOLT-combined.log.gz

 
karaf@root > flows -s ADDED of:0000b86a97071b4a 02:00:19
02:00:41.079 ERROR [ShellUtil] Exception caught while executing command
org.onosproject.net.flow.FlowRuleStoreException: There is no master for available device of:0000b86a97071b4a
at org.onosproject.store.flow.impl.DeviceFlowTable.getFlowEntries(DeviceFlowTable.java:257)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3605)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.onosproject.store.flow.impl.DeviceFlowTable.getFlowEntries(DeviceFlowTable.java:219)
at org.onosproject.store.flow.impl.ECFlowRuleStore$InternalFlowTable.getFlowEntries(ECFlowRuleStore.java:788)
at org.onosproject.store.flow.impl.ECFlowRuleStore.getFlowEntries(ECFlowRuleStore.java:422)
at org.onosproject.net.flow.impl.FlowRuleManager.getFlowEntries(FlowRuleManager.java:301)
at org.onosproject.cli.net.FlowsListCommand.getSortedFlows(FlowsListCommand.java:272)
at org.onosproject.cli.net.FlowsListCommand.doExecute(FlowsListCommand.java:143)
at org.onosproject.cli.AbstractShellCommand.execute(AbstractShellCommand.java:159)
at org.apache.karaf.shell.impl.action.command.ActionCommand.execute(ActionCommand.java:84)
at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:68)
at org.apache.karaf.shell.impl.console.osgi.secured.SecuredCommand.execute(SecuredCommand.java:86)
at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:599)
at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:526)
at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:415)
at org.apache.felix.gogo.runtime.Pipe.doCall(Pipe.java:416)
at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:229)
at org.apache.felix.gogo.runtime.Pipe.call(Pipe.java:59)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Error executing command: There is no master for available device of:0000b86a97071b4a
karaf@root >
 
karaf@root > roles 02:00:54
of:0000000000000001: master=onos-onos-classic-2, standbys=[ ]
of:0000b86a97071b4a: master=onos-onos-classic-2, standbys=[ onos-onos-classic-1 onos-onos-classic-0 ]
karaf@root >
 
karaf@root > nodes 02:03:04
id=onos-onos-classic-0, address=10.233.75.47:9876, state=READY, version=2.2.3.b1, updated=1h13m ago
id=onos-onos-classic-1, address=10.233.102.131:9876, state=READY, version=2.2.3.b1, updated=1h13m ago *
id=onos-onos-classic-2, address=10.233.71.49:9876, state=READY, version=2.2.3.b1, updated=1h13m ago
karaf@root >