voltha-scale: misc errors and job failed due to timeout (tt-subscribers)

https://jenkins.opencord.org/view/voltha-scale-measurements/job/voltha-scale-measurements-master-1-64-63-tt-subscribers/277/consoleFull

 
Timeout set to expire in 1 hr 0 min
 

sleep calls buried in the logs should be replaced with a polling loop rather than adding fixed time to every job run by the system – result of this will adds hours/days of idle resources over a single year.

 
20:53:40 Wait for VOLTHA Stack voltha1 to start*20:53:40* [Pipeline] sh*20:53:42* + set +x*20:53:42* bbsim-sadis-server-585b867bcc-z8r4g 1/1 Running 0 8s*20:53:42* bbsim0-7f86758688-k92m6 0/1 ContainerCreating 0 1s*20:53:42* voltha1-voltha-adapter-openolt-594db6b86b-d7466 0/1 ContainerCreating 0 5s*20:53:42* voltha1-voltha-adapter-openonu-85b79d865d-pc6r2 0/1 ContainerCreating 0 5s*20:53:42* voltha1-voltha-ofagent-669b7c585b-n95wd 0/1 ContainerCreating 0 5s*20:53:42* voltha1-voltha-rw-core-554657d9f-vqnnk 0/1 Running 0 5s*21:03:07* Cancelling nested steps due to timeout*21:03:07* Sending interrupt signal to process*21:03:11* Terminated*21:03:11* script returned exit code 143
21:03:18 + kubectl cp :/var/log/freeradius/radius.log /var/jenkins_home/workspace/voltha-scale-measurements-master-1-64-63-tt-subscribers/logs/radius.log*21:03:18* error: filespec must match the canonical format: [[namespace/]pod:]file/path
 
21:03:19 + kubectl exec -t bbsim0-7f86758688-k92m6 – bbsimctl onu list*21:03:19* time="2022-12-01T02:03:19Z" level=fatal msg="could not get OLT: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:50070: connect: connection refused\""21:03:19 command terminated with exit code 1
 

Interrupted here
21:06:42 + sleep 60*21:07:51* + python scripts/sizing.py -o /var/jenkins_home/workspace/voltha-scale-measurements-master-1-64-63-tt-subscribers/plots*21:07:51* CPU usage query: {'query': 'sum by(pod) (rate(container_cpu_usage_seconds_total

[5m]))', 'start': 1669859863.1703663, 'end': 1669860463.1703663, 'step': 30}21:07:51 Downloading CPU info from: http://127.0.0.1:31301/api/v1/query_range?query=sum+by%28pod%29+%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%22default%22%2Ccontainer%21%3D%22%22%2Ccontainer%21%3D%22POD%22%7D%5B5m%5D%29%29&start=1669859863.1703663&end=1669860463.1703663&step=3021:07:51 Traceback (most recent call last):21:07:51 File "scripts/sizing.py", line 405, in <module>21:07:51 main(args.address, args.output, args.since, args.namespace, args.rate, args.step)21:07:51 File "scripts/sizing.py", line 82, in main*21:07:51* plot_cpu_consumption(containers,**