ONOS does not come up due multiple Readiness probe failed

 
 
 
But now we have a new issue, the pre-running job is failing due ONOS does not come up:
Wait for ONOS Config loader to complete
I tried a re-run twice. Both failed due ONOS issue.
Look into artifacts in onos-pods-describe.txt I see:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19m default-scheduler Successfully assigned infra/voltha-infra-onos-classic-0 to node-3
Normal Pulled 19m kubelet Container image "tutum/dnsutils:latest" already present on machine
Normal Created 19m kubelet Created container onos-classic-init
Normal Started 19m kubelet Started container onos-classic-init
Normal Pulling 19m kubelet Pulling image "voltha/voltha-onos:5.0.3"
Normal Pulled 19m kubelet Successfully pulled image "voltha/voltha-onos:5.0.3" in 1.430453937s
Normal Created 19m kubelet Created container onos-classic
Normal Started 19m kubelet Started container onos-classic
Warning Unhealthy 14m (x18 over 18m) kubelet Readiness probe failed:
Cluster is not yet ready
Warning Unhealthy 14m kubelet Liveness probe failed:
Cluster is not yet ready
Warning Unhealthy 4m9s (x29 over 10m) kubelet (combined from similar events): Readiness probe failed: {"id":"onos-0","ip":"10.42.2.200","tcpPort":9876,"status":"ACTIVE","lastUpdate":"1666940781703","humanReadableLastUpdate":"2m38s ago"}
Cluster is not yet ready
I'm not a ONOS guy, maybe we should contact Gustavo or someone else...