Activation of onu fails with etcd/kafka/zookeper persistence enabled

Environment:

hardware: 3 baremetal k8s nodes (
Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
1tb ssd disk for each node
OS: Centos 7
k8s: 1.18.5
helm: 3.2.4
_K8s storageclas_s: openebs-jiva-default
Setup:
Persistence enabled for kafka, zookeeper, atomix, etcd in helm charts

Components list:

adtran-olt-device-manager-6c7454c58f-5vxfz   1/1     Running   0          19h
adtran-olt-voltha-adapter-58bdf6cbc7-l9k6b   1/1     Running   0          19h
consul-0                                     1/1     Running   0          20h
consul-1                                     1/1     Running   2          20h
consul-2                                     1/1     Running   0          20h
grpccli                                      1/1     Running   0          5d19h
kafka-topic-exporter-56d486c494-7tgs4        1/1     Running   0          19h
onos-atomix-0                                1/1     Running   1          20h
onos-atomix-1                                1/1     Running   0          20h
onos-atomix-2                                1/1     Running   0          20h
onos-onos-classic-0                          1/1     Running   0          20h
onos-onos-classic-1                          1/1     Running   0          20h
onos-onos-classic-2                          1/1     Running   0          20h
padm-core-7db9c9464b-d7mkm                   1/1     Running   7          19h
padm-diag-7fcf47cf99-f4mgk                   1/1     Running   7          19h
padmee-f5b776cb5-2xn69                       1/1     Running   0          19h
vap-grpc-9577f6bcb-m7krk                     1/1     Running   0          19h
vap-onos-kafka-68696f9f5f-8x77w              1/1     Running   0          19h
vap-padm-grpc-6dbbb555d-fgfnc                1/1     Running   0          19h
vap-voltha-kafka-745966cb9b-wn66t            1/1     Running   0          19h
voltha-adapter-openonu-0                     1/1     Running   0          19h
voltha-etcd-cluster-client-0                 1/1     Running   0          20h
voltha-etcd-cluster-client-1                 1/1     Running   0          20h
voltha-etcd-cluster-client-2                 1/1     Running   0          20h
voltha-kafka-0                               1/1     Running   1          20h
voltha-kafka-1                               1/1     Running   2          20h
voltha-kafka-2                               1/1     Running   2          20h
voltha-kafka-zookeeper-0                     1/1     Running   0          20h
voltha-kafka-zookeeper-1                     1/1     Running   0          20h
voltha-kafka-zookeeper-2                     1/1     Running   0          20h
voltha-ofagent-86f4755c59-4pgn5              1/1     Running   0          19h
voltha-rw-core-7d4c5cc94c-lv7b5              1/1     Running   0          19h 

Test procedure:

Triggering of add olt with an active onu attached to it

Test results:

• data path not working
• ONOS flows are pushed and have right status:

• deviceId=of:0000002445306b5f, flowRuleCount=5
  id=1e000002e6a1891, state=ADDED, bytes=0, packets=0, duration=0, liveType=UNKNOWN, priority=10000, tableId=0, appId=org.opencord.olt, selector=[IN_PORT:1048576, ETH_TYPE:lldp], treatment=DefaultTrafficTreatment{immediate=[OUTPUT:CONTROLLER], deferred=[], transition=None, meter=[], cleared=false, StatTrigger=null, metadata=null}
  id=1e0000054d4a21b, state=ADDED, bytes=0, packets=0, duration=0, liveType=UNKNOWN, priority=1000, tableId=0, appId=org.opencord.olt, selector=[IN_PORT:16, VLAN_VID:Any], treatment=DefaultTrafficTreatment{immediate=[], deferred=[], transition=TABLE:1, meter=[METER:1], cleared=false, StatTrigger=null, metadata=METADATA:1000004100100000/0}
  id=1e000006a310aa3, state=ADDED, bytes=0, packets=0, duration=0, liveType=UNKNOWN, priority=1000, tableId=0, appId=org.opencord.olt, selector=[IN_PORT:1048576, VLAN_VID:129], treatment=DefaultTrafficTreatment{immediate=[VLAN_POP], deferred=[], transition=TABLE:1, meter=[METER:2], cleared=false, StatTrigger=null, metadata=METADATA:1000004100000010/0}
  id=1e000001a6056d1, state=ADDED, bytes=0, packets=0, duration=0, liveType=UNKNOWN, priority=1000, tableId=1, appId=org.opencord.olt, selector=[IN_PORT:16, VLAN_VID:Any], treatment=DefaultTrafficTreatment{immediate=[VLAN_PUSH:vlan, VLAN_ID:129, OUTPUT:1048576], deferred=[], transition=None, meter=[METER:1], cleared=false, StatTrigger=null, metadata=METADATA:4100000000/0}
  id=1e00000ee98aa40, state=ADDED, bytes=0, packets=0, duration=0, liveType=UNKNOWN, priority=1000, tableId=1, appId=org.opencord.olt, selector=[IN_PORT:1048576, METADATA:10, VLAN_VID:Any], treatment=DefaultTrafficTreatment{immediate=[OUTPUT:16], deferred=[], transition=None, meter=[METER:2], cleared=false, StatTrigger=null, metadata=METADATA:4100000000/0}  

• ONU has wrong status in voltha:

•  kubectl exec grpccli -i -n voltha -- voltctl device list
  ID TYPE ROOT PARENTID SERIALNUMBER ADMINSTATE OPERSTATUS CONNECTSTATUS REASON
  6b1b814e-6942-4d32-92c2-41e65f5ca6a3 adtranolt true c12877f2-940e-42d5-8d0c-0837b299910f LBADTN2020PT559 ENABLED ACTIVE REACHABLE
  cd8dbec0-1375-4576-92f7-ba0c7fc26d2c brcm_openomci_onu false 6b1b814e-6942-4d32-92c2-41e65f5ca6a3 SCOM00001c03 ENABLED ACTIVE REACHABLE tech-profile-config-download-success 

• logs are attached to the story

 It is expected that the reason for this problem is investigated and a corresponding bug is created and resolved.