root@sbat1:~# kubectl logs --namespace voltha voltha-etcd-cluster-95s9dfd7sd
2019-06-28 14:15:43.617618 I | pkg/flags: recognized and used environment variable ETCD_AUTO_COMPACTION_RETENTION=12
2019-06-28 14:15:43.617784 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_PORT_19999_TCP_PROTO=tcp
2019-06-28 14:15:43.617807 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_PORT_19999_TCP_ADDR=10.233.27.85
2019-06-28 14:15:43.617826 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_SERVICE_PORT_HTTP_ETCD_RESTORE_PORT=19999
2019-06-28 14:15:43.620902 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_PORT_19999_TCP=tcp://10.233.27.85:19999
2019-06-28 14:15:43.620912 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_SERVICE_PORT=19999
2019-06-28 14:15:43.620916 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_PORT_19999_TCP_PORT=19999
2019-06-28 14:15:43.620933 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_PORT=tcp://10.233.27.85:19999
2019-06-28 14:15:43.620942 W | pkg/flags: unrecognized environment variable ETCD_RESTORE_OPERATOR_SERVICE_HOST=10.233.27.85
2019-06-28 14:15:43.620964 I | etcdmain: etcd Version: 3.2.18
2019-06-28 14:15:43.620972 I | etcdmain: Git SHA: eddf599c6
2019-06-28 14:15:43.620976 I | etcdmain: Go Version: go1.8.7
2019-06-28 14:15:43.620980 I | etcdmain: Go OS/Arch: linux/amd64
2019-06-28 14:15:43.620985 I | etcdmain: setting maximum number of CPUs to 40, total number of available CPUs is 40
2019-06-28 14:15:43.621098 I | embed: listening for peers on http://0.0.0.0:2380
2019-06-28 14:15:43.621147 I | embed: listening for client requests on 0.0.0.0:2379
2019-06-28 14:15:43.647677 I | pkg/netutil: resolving voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2380 to 10.233.70.143:2380
2019-06-28 14:15:43.648494 I | pkg/netutil: resolving voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2380 to 10.233.70.143:2380
2019-06-28 14:15:43.648570 I | etcdserver: name = voltha-etcd-cluster-95s9dfd7sd
2019-06-28 14:15:43.648580 I | etcdserver: data dir = /var/etcd/data
2019-06-28 14:15:43.648586 I | etcdserver: member dir = /var/etcd/data/member
2019-06-28 14:15:43.648591 I | etcdserver: heartbeat = 100ms
2019-06-28 14:15:43.648599 I | etcdserver: election = 1000ms
2019-06-28 14:15:43.648605 I | etcdserver: snapshot count = 100000
2019-06-28 14:15:43.648624 I | etcdserver: advertise client URLs = http://voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2379
2019-06-28 14:15:43.648634 I | etcdserver: initial advertise peer URLs = http://voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2380
2019-06-28 14:15:43.648646 I | etcdserver: initial cluster = voltha-etcd-cluster-95s9dfd7sd=http://voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2380
2019-06-28 14:15:43.687727 I | etcdserver: starting member 1104aa4f65b2d088 in cluster af918d3b181db91f
2019-06-28 14:15:43.687769 I | raft: 1104aa4f65b2d088 became follower at term 0
2019-06-28 14:15:43.687784 I | raft: newRaft 1104aa4f65b2d088 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2019-06-28 14:15:43.687792 I | raft: 1104aa4f65b2d088 became follower at term 1
2019-06-28 14:15:43.732462 W | auth: simple token is not cryptographically signed
2019-06-28 14:15:43.766081 I | etcdserver: starting server... [version: 3.2.18, cluster version: to_be_decided]
2019-06-28 14:15:43.766252 I | etcdserver: 1104aa4f65b2d088 as single-node; fast-forwarding 9 ticks (election ticks 10)
2019-06-28 14:15:43.766745 I | etcdserver/membership: added member 1104aa4f65b2d088 [http://voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2380] to cluster af918d3b181db91f
2019-06-28 14:15:44.088475 I | raft: 1104aa4f65b2d088 is starting a new election at term 1
2019-06-28 14:15:44.088514 I | raft: 1104aa4f65b2d088 became candidate at term 2
2019-06-28 14:15:44.088570 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 2
2019-06-28 14:15:44.088636 I | raft: 1104aa4f65b2d088 became leader at term 2
2019-06-28 14:15:44.088655 I | raft: raft.node: 1104aa4f65b2d088 elected leader 1104aa4f65b2d088 at term 2
2019-06-28 14:15:44.088915 I | etcdserver: setting up the initial cluster version to 3.2
2019-06-28 14:15:44.090890 N | etcdserver/membership: set the initial cluster version to 3.2
2019-06-28 14:15:44.090964 I | etcdserver/api: enabled capabilities for version 3.2
2019-06-28 14:15:44.090994 I | embed: ready to serve client requests
2019-06-28 14:15:44.091134 I | etcdserver: published {Name:voltha-etcd-cluster-95s9dfd7sd ClientURLs:[http://voltha-etcd-cluster-95s9dfd7sd.voltha-etcd-cluster.voltha.svc:2379]} to cluster af918d3b181db91f
2019-06-28 14:15:44.091452 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
2019-06-28 14:15:49.821156 I | etcdserver/membership: added member 95ed36424bd1dcf5 [http://voltha-etcd-cluster-sqzg6bv54f.voltha-etcd-cluster.voltha.svc:2380] to cluster af918d3b181db91f
2019-06-28 14:15:49.821208 I | rafthttp: starting peer 95ed36424bd1dcf5...
2019-06-28 14:15:49.821245 I | rafthttp: started HTTP pipelining with peer 95ed36424bd1dcf5
2019-06-28 14:15:49.822573 I | rafthttp: started streaming with peer 95ed36424bd1dcf5 (writer)
2019-06-28 14:15:49.825069 I | rafthttp: started streaming with peer 95ed36424bd1dcf5 (writer)
2019-06-28 14:15:49.826079 I | rafthttp: started peer 95ed36424bd1dcf5
2019-06-28 14:15:49.826167 I | rafthttp: started streaming with peer 95ed36424bd1dcf5 (stream MsgApp v2 reader)
2019-06-28 14:15:49.826293 I | rafthttp: added peer 95ed36424bd1dcf5
2019-06-28 14:15:49.826376 I | rafthttp: started streaming with peer 95ed36424bd1dcf5 (stream Message reader)
2019-06-28 14:15:51.088367 W | raft: 1104aa4f65b2d088 stepped down to follower since quorum is not active
2019-06-28 14:15:51.088426 I | raft: 1104aa4f65b2d088 became follower at term 2
2019-06-28 14:15:51.088440 I | raft: raft.node: 1104aa4f65b2d088 lost leader 1104aa4f65b2d088 at term 2
2019-06-28 14:15:52.488411 I | raft: 1104aa4f65b2d088 is starting a new election at term 2
2019-06-28 14:15:52.488466 I | raft: 1104aa4f65b2d088 became candidate at term 3
2019-06-28 14:15:52.488481 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 3
2019-06-28 14:15:52.488497 I | raft: 1104aa4f65b2d088 [logterm: 2, index: 5] sent MsgVote request to 95ed36424bd1dcf5 at term 3
2019-06-28 14:15:54.388352 I | raft: 1104aa4f65b2d088 is starting a new election at term 3
2019-06-28 14:15:54.388415 I | raft: 1104aa4f65b2d088 became candidate at term 4
2019-06-28 14:15:54.388433 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 4
2019-06-28 14:15:54.388449 I | raft: 1104aa4f65b2d088 [logterm: 2, index: 5] sent MsgVote request to 95ed36424bd1dcf5 at term 4
2019-06-28 14:15:54.826611 W | rafthttp: health check for peer 95ed36424bd1dcf5 could not connect: dial tcp: lookup voltha-etcd-cluster-sqzg6bv54f.voltha-etcd-cluster.voltha.svc on 169.254.25.10:53: no such host
2019-06-28 14:15:56.288498 I | raft: 1104aa4f65b2d088 is starting a new election at term 4
2019-06-28 14:15:56.288560 I | raft: 1104aa4f65b2d088 became candidate at term 5
2019-06-28 14:15:56.288580 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 5
2019-06-28 14:15:56.288600 I | raft: 1104aa4f65b2d088 [logterm: 2, index: 5] sent MsgVote request to 95ed36424bd1dcf5 at term 5
2019-06-28 14:15:57.388448 I | raft: 1104aa4f65b2d088 is starting a new election at term 5
2019-06-28 14:15:57.388503 I | raft: 1104aa4f65b2d088 became candidate at term 6
2019-06-28 14:15:57.388519 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 6
2019-06-28 14:15:57.388536 I | raft: 1104aa4f65b2d088 [logterm: 2, index: 5] sent MsgVote request to 95ed36424bd1dcf5 at term 6
WARNING: 2019/06/28 14:15:57 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2019-06-28 14:15:58.688359 I | raft: 1104aa4f65b2d088 is starting a new election at term 6
2019-06-28 14:15:58.688417 I | raft: 1104aa4f65b2d088 became candidate at term 7
2019-06-28 14:15:58.688436 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 7
2019-06-28 14:15:58.688454 I | raft: 1104aa4f65b2d088 [logterm: 2, index: 5] sent MsgVote request to 95ed36424bd1dcf5 at term 7
2019-06-28 14:15:59.090458 I | rafthttp: peer 95ed36424bd1dcf5 became active
2019-06-28 14:15:59.090500 I | rafthttp: established a TCP streaming connection with peer 95ed36424bd1dcf5 (stream Message reader)
2019-06-28 14:15:59.090530 I | rafthttp: established a TCP streaming connection with peer 95ed36424bd1dcf5 (stream MsgApp v2 reader)
2019-06-28 14:15:59.092803 I | rafthttp: established a TCP streaming connection with peer 95ed36424bd1dcf5 (stream Message writer)
2019-06-28 14:15:59.092894 I | rafthttp: established a TCP streaming connection with peer 95ed36424bd1dcf5 (stream MsgApp v2 writer)
2019-06-28 14:15:59.828166 W | rafthttp: health check for peer 95ed36424bd1dcf5 could not connect: dial tcp 10.233.79.100:2380: getsockopt: connection refused
2019-06-28 14:15:59.985047 W | etcdserver: timed out waiting for read index response
2019-06-28 14:16:00.388345 I | raft: 1104aa4f65b2d088 is starting a new election at term 7
2019-06-28 14:16:00.388399 I | raft: 1104aa4f65b2d088 became candidate at term 8
2019-06-28 14:16:00.388414 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 1104aa4f65b2d088 at term 8
2019-06-28 14:16:00.388429 I | raft: 1104aa4f65b2d088 [logterm: 2, index: 5] sent MsgVote request to 95ed36424bd1dcf5 at term 8
2019-06-28 14:16:00.408059 I | raft: 1104aa4f65b2d088 received MsgVoteResp from 95ed36424bd1dcf5 at term 8
2019-06-28 14:16:00.408079 I | raft: 1104aa4f65b2d088 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
2019-06-28 14:16:00.408099 I | raft: 1104aa4f65b2d088 became leader at term 8
2019-06-28 14:16:00.408114 I | raft: raft.node: 1104aa4f65b2d088 elected leader 1104aa4f65b2d088 at term 8
2019-06-28 14:16:06.024121 I | etcdserver/membership: added member 1d5479adc43e3139 [http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380] to cluster af918d3b181db91f
2019-06-28 14:16:06.024172 I | rafthttp: starting peer 1d5479adc43e3139...
2019-06-28 14:16:06.024253 I | rafthttp: started HTTP pipelining with peer 1d5479adc43e3139
2019-06-28 14:16:06.024678 I | rafthttp: started streaming with peer 1d5479adc43e3139 (writer)
2019-06-28 14:16:06.025536 I | rafthttp: started streaming with peer 1d5479adc43e3139 (writer)
2019-06-28 14:16:06.029298 I | rafthttp: started peer 1d5479adc43e3139
2019-06-28 14:16:06.029335 I | rafthttp: added peer 1d5479adc43e3139
2019-06-28 14:16:06.029367 I | rafthttp: started streaming with peer 1d5479adc43e3139 (stream Message reader)
2019-06-28 14:16:06.029472 I | rafthttp: started streaming with peer 1d5479adc43e3139 (stream MsgApp v2 reader)
2019-06-28 14:16:06.985267 W | etcdserver: timed out waiting for read index response
2019-06-28 14:16:08.439329 W | etcdserver: failed to reach the peerURL(http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380) of member 1d5479adc43e3139 (Get http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380/version: dial tcp: lookup voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc on 169.254.25.10:53: no such host)
2019-06-28 14:16:08.439369 W | etcdserver: cannot get the version of member 1d5479adc43e3139 (Get http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380/version: dial tcp: lookup voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc on 169.254.25.10:53: no such host)
2019-06-28 14:16:11.029546 W | rafthttp: health check for peer 1d5479adc43e3139 could not connect: dial tcp: lookup voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc on 169.254.25.10:53: no such host
2019-06-28 14:16:12.444861 W | etcdserver: failed to reach the peerURL(http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380) of member 1d5479adc43e3139 (Get http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380/version: dial tcp 10.233.83.98:2380: getsockopt: connection refused)
2019-06-28 14:16:12.444911 W | etcdserver: cannot get the version of member 1d5479adc43e3139 (Get http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380/version: dial tcp 10.233.83.98:2380: getsockopt: connection refused)
2019-06-28 14:16:14.527697 I | rafthttp: peer 1d5479adc43e3139 became active
2019-06-28 14:16:14.527748 I | rafthttp: established a TCP streaming connection with peer 1d5479adc43e3139 (stream MsgApp v2 reader)
2019-06-28 14:16:14.527831 I | rafthttp: established a TCP streaming connection with peer 1d5479adc43e3139 (stream Message reader)
2019-06-28 14:16:14.527904 I | rafthttp: established a TCP streaming connection with peer 1d5479adc43e3139 (stream Message writer)
2019-06-28 14:16:14.527975 I | rafthttp: established a TCP streaming connection with peer 1d5479adc43e3139 (stream MsgApp v2 writer)
2019-06-28 14:16:16.029883 W | rafthttp: health check for peer 1d5479adc43e3139 could not connect: dial tcp 10.233.83.98:2380: getsockopt: connection refused
2019-06-28 14:44:40.995041 W | etcdserver: apply entries took too long [112.213954ms for 1 entries]
2019-06-28 14:44:40.995102 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 14:44:53.583025 W | etcdserver: apply entries took too long [134.075341ms for 1 entries]
2019-06-28 14:44:53.583061 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 14:49:25.629396 W | etcdserver: apply entries took too long [182.205347ms for 1 entries]
2019-06-28 14:49:25.629434 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 14:59:30.581773 W | etcdserver: apply entries took too long [141.450736ms for 1 entries]
2019-06-28 14:59:30.581805 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 14:59:43.977708 W | etcdserver: apply entries took too long [109.598481ms for 1 entries]
2019-06-28 14:59:43.977752 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:00:46.981158 W | etcdserver: apply entries took too long [109.766711ms for 1 entries]
2019-06-28 15:00:46.981187 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:07:25.622218 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 33.694091ms)
2019-06-28 15:07:25.622276 W | etcdserver: server is likely overloaded
2019-06-28 15:07:25.622294 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 33.796708ms)
2019-06-28 15:07:25.622321 W | etcdserver: server is likely overloaded
2019-06-28 15:08:47.798476 W | etcdserver: apply entries took too long [146.54236ms for 1 entries]
2019-06-28 15:08:47.798515 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:09:45.571777 W | etcdserver: apply entries took too long [123.880952ms for 1 entries]
2019-06-28 15:09:45.571807 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:11:43.909400 W | etcdserver: apply entries took too long [144.06309ms for 1 entries]
2019-06-28 15:11:43.909442 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:11:46.921245 W | etcdserver: apply entries took too long [141.861725ms for 1 entries]
2019-06-28 15:11:46.921279 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:19:58.762393 W | etcdserver: apply entries took too long [109.844001ms for 1 entries]
2019-06-28 15:19:58.762435 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:21:25.917931 W | etcdserver: apply entries took too long [156.040684ms for 1 entries]
2019-06-28 15:21:25.917960 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:21:28.945879 W | etcdserver: apply entries took too long [154.965541ms for 1 entries]
2019-06-28 15:21:28.946268 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:28:10.519825 W | etcdserver: apply entries took too long [106.257977ms for 1 entries]
2019-06-28 15:28:10.519844 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:30:19.968279 W | etcdserver: apply entries took too long [163.168529ms for 1 entries]
2019-06-28 15:30:19.968318 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:32:31.927351 W | etcdserver: apply entries took too long [114.515785ms for 1 entries]
2019-06-28 15:32:31.927382 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:44:55.953388 W | etcdserver: apply entries took too long [143.125881ms for 1 entries]
2019-06-28 15:44:55.953430 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:49:35.647733 W | etcdserver: apply entries took too long [140.009722ms for 1 entries]
2019-06-28 15:49:35.647780 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:54:40.906774 W | etcdserver: apply entries took too long [155.184482ms for 1 entries]
2019-06-28 15:54:40.906815 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 15:56:28.841301 W | etcdserver: apply entries took too long [103.87295ms for 1 entries]
2019-06-28 15:56:28.841348 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 16:03:10.127127 W | etcdserver: apply entries took too long [151.961253ms for 1 entries]
2019-06-28 16:03:10.127163 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 16:07:40.581260 W | etcdserver: apply entries took too long [109.189185ms for 1 entries]
2019-06-28 16:07:40.581293 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 16:13:06.003601 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 15.139376ms)
2019-06-28 16:13:06.003631 W | etcdserver: server is likely overloaded
2019-06-28 16:13:06.003639 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 15.186652ms)
2019-06-28 16:13:06.003644 W | etcdserver: server is likely overloaded
2019-06-28 16:31:20.556468 W | etcdserver: apply entries took too long [104.598942ms for 1 entries]
2019-06-28 16:31:20.556521 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 16:50:31.949619 W | etcdserver: apply entries took too long [153.922257ms for 1 entries]
2019-06-28 16:50:31.949659 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 17:05:27.092553 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 3.34701ms)
2019-06-28 17:05:27.092583 W | etcdserver: server is likely overloaded
2019-06-28 17:05:27.092590 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 3.391937ms)
2019-06-28 17:05:27.092595 W | etcdserver: server is likely overloaded
2019-06-28 17:14:46.852930 W | etcdserver: apply entries took too long [140.659333ms for 1 entries]
2019-06-28 17:14:46.852973 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 17:24:00.589715 W | etcdserver: apply entries took too long [156.327413ms for 1 entries]
2019-06-28 17:24:00.589773 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 17:30:01.719538 W | etcdserver: apply entries took too long [151.195015ms for 1 entries]
2019-06-28 17:30:01.719567 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 17:45:40.993799 W | etcdserver: apply entries took too long [160.616889ms for 1 entries]
2019-06-28 17:45:40.993854 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 17:55:01.847917 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 59.457573ms)
2019-06-28 17:55:01.847945 W | etcdserver: server is likely overloaded
2019-06-28 17:55:01.847952 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 59.500994ms)
2019-06-28 17:55:01.847956 W | etcdserver: server is likely overloaded
2019-06-28 18:04:37.774841 W | etcdserver: apply entries took too long [102.551945ms for 1 entries]
2019-06-28 18:04:37.774905 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 18:11:06.176602 W | etcdserver: apply entries took too long [150.220029ms for 1 entries]
2019-06-28 18:11:06.176656 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 18:12:40.558072 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 69.521916ms)
2019-06-28 18:12:40.558111 W | etcdserver: server is likely overloaded
2019-06-28 18:12:40.558119 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 69.648562ms)
2019-06-28 18:12:40.558124 W | etcdserver: server is likely overloaded
2019-06-28 18:24:16.827579 W | etcdserver: apply entries took too long [167.008807ms for 1 entries]
2019-06-28 18:24:16.827621 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 18:26:08.586777 W | etcdserver: apply entries took too long [103.709848ms for 1 entries]
2019-06-28 18:26:08.586806 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 18:35:05.603618 W | etcdserver: apply entries took too long [150.343973ms for 1 entries]
2019-06-28 18:35:05.603666 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 18:51:25.637760 W | etcdserver: apply entries took too long [155.485451ms for 1 entries]
2019-06-28 18:51:25.637809 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 18:59:35.527076 W | etcdserver: apply entries took too long [142.724346ms for 1 entries]
2019-06-28 18:59:35.527117 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:00:46.750460 W | etcdserver: apply entries took too long [108.064723ms for 1 entries]
2019-06-28 19:00:46.750483 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:07:46.795950 W | etcdserver: apply entries took too long [106.586013ms for 1 entries]
2019-06-28 19:07:46.796003 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:09:13.775325 W | etcdserver: apply entries took too long [137.78778ms for 1 entries]
2019-06-28 19:09:13.775377 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:13:10.709742 W | etcdserver: apply entries took too long [109.310584ms for 1 entries]
2019-06-28 19:13:10.709783 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:14:03.613310 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 24.873102ms)
2019-06-28 19:14:03.613347 W | etcdserver: server is likely overloaded
2019-06-28 19:14:03.613358 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 24.92832ms)
2019-06-28 19:14:03.613364 W | etcdserver: server is likely overloaded
2019-06-28 19:35:04.623800 I | etcdserver: start to snapshot (applied: 100001, lastsnap: 0)
2019-06-28 19:35:04.652062 I | etcdserver: saved snapshot at index 100001
2019-06-28 19:35:04.652648 I | etcdserver: compacted raft log at 95001
2019-06-28 19:39:37.272082 W | etcdserver: apply entries took too long [103.112648ms for 1 entries]
2019-06-28 19:39:37.272122 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:50:25.108236 W | etcdserver: apply entries took too long [100.402321ms for 1 entries]
2019-06-28 19:50:25.108269 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 19:50:43.820039 W | etcdserver: apply entries took too long [131.909314ms for 1 entries]
2019-06-28 19:50:43.820084 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:11:15.624443 W | etcdserver: apply entries took too long [145.070178ms for 1 entries]
2019-06-28 20:11:15.624471 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:16:07.750678 W | etcdserver: apply entries took too long [134.867755ms for 1 entries]
2019-06-28 20:16:07.750731 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:19:06.517503 W | etcdserver: apply entries took too long [141.904788ms for 1 entries]
2019-06-28 20:19:06.517526 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:21:04.788869 W | etcdserver: apply entries took too long [137.857535ms for 1 entries]
2019-06-28 20:21:04.788915 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:22:40.604596 W | etcdserver: apply entries took too long [116.973376ms for 1 entries]
2019-06-28 20:22:40.604621 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:35:55.677268 W | etcdserver: apply entries took too long [164.95705ms for 1 entries]
2019-06-28 20:35:55.677340 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:41:58.713450 W | etcdserver: apply entries took too long [138.474545ms for 1 entries]
2019-06-28 20:41:58.713509 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:44:01.748600 W | etcdserver: apply entries took too long [166.047771ms for 1 entries]
2019-06-28 20:44:01.748654 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:45:28.704068 W | etcdserver: apply entries took too long [130.581544ms for 1 entries]
2019-06-28 20:45:28.704114 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:45:37.912166 W | etcdserver: apply entries took too long [146.822282ms for 1 entries]
2019-06-28 20:45:37.912203 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:46:25.687673 W | etcdserver: apply entries took too long [119.443545ms for 1 entries]
2019-06-28 20:46:25.687700 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 20:46:43.697252 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 8.776798ms)
2019-06-28 20:46:43.697309 W | etcdserver: server is likely overloaded
2019-06-28 20:46:43.697327 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 8.865038ms)
2019-06-28 20:46:43.697340 W | etcdserver: server is likely overloaded
2019-06-28 20:59:35.175244 W | etcdserver: apply entries took too long [140.405858ms for 1 entries]
2019-06-28 20:59:35.175284 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:06:10.648911 W | etcdserver: apply entries took too long [134.949185ms for 1 entries]
2019-06-28 21:06:10.648951 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:06:10.697713 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 9.28843ms)
2019-06-28 21:06:10.697737 W | etcdserver: server is likely overloaded
2019-06-28 21:06:10.697749 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 9.339089ms)
2019-06-28 21:06:10.697762 W | etcdserver: server is likely overloaded
2019-06-28 21:07:10.091256 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 2.845065ms)
2019-06-28 21:07:10.091293 W | etcdserver: server is likely overloaded
2019-06-28 21:07:10.091303 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 2.905579ms)
2019-06-28 21:07:10.091309 W | etcdserver: server is likely overloaded
2019-06-28 21:14:50.625598 W | etcdserver: apply entries took too long [108.573683ms for 1 entries]
2019-06-28 21:14:50.625629 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:23:29.324995 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 36.510077ms)
2019-06-28 21:23:29.325043 W | etcdserver: server is likely overloaded
2019-06-28 21:23:29.325056 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 36.587427ms)
2019-06-28 21:23:29.325065 W | etcdserver: server is likely overloaded
2019-06-28 21:25:52.329575 W | etcdserver: apply entries took too long [104.349708ms for 1 entries]
2019-06-28 21:25:52.329602 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:25:55.721522 W | etcdserver: apply entries took too long [100.063741ms for 1 entries]
2019-06-28 21:25:55.721563 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:46:14.445747 W | etcdserver: apply entries took too long [103.503997ms for 1 entries]
2019-06-28 21:46:14.445782 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:46:15.593716 W | etcdserver: apply entries took too long [117.41539ms for 1 entries]
2019-06-28 21:46:15.593757 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 21:48:13.773128 W | etcdserver: apply entries took too long [152.722434ms for 1 entries]
2019-06-28 21:48:13.773157 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 22:08:57.393317 W | etcdserver: apply entries took too long [103.31914ms for 1 entries]
2019-06-28 22:08:57.393356 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 22:11:22.768575 W | etcdserver: apply entries took too long [128.846515ms for 1 entries]
2019-06-28 22:11:22.768621 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 22:12:53.979296 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 90.825498ms)
2019-06-28 22:12:53.979342 W | etcdserver: server is likely overloaded
2019-06-28 22:12:53.979357 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 90.913929ms)
2019-06-28 22:12:53.979365 W | etcdserver: server is likely overloaded
2019-06-28 22:33:15.612481 W | etcdserver: apply entries took too long [171.578414ms for 1 entries]
2019-06-28 22:33:15.612518 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 22:38:45.542628 W | etcdserver: apply entries took too long [100.585456ms for 1 entries]
2019-06-28 22:38:45.542658 W | etcdserver: avoid queries with large range/delete range!
2019-06-28 22:46:00.644165 W | rafthttp: lost the TCP streaming connection with peer 1d5479adc43e3139 (stream Message reader)
2019-06-28 22:46:00.644222 E | rafthttp: failed to read 1d5479adc43e3139 on stream Message (unexpected EOF)
2019-06-28 22:46:00.644233 I | rafthttp: peer 1d5479adc43e3139 became inactive
2019-06-28 22:46:00.644249 W | rafthttp: lost the TCP streaming connection with peer 1d5479adc43e3139 (stream MsgApp v2 reader)
2019-06-28 22:46:00.889062 W | rafthttp: lost the TCP streaming connection with peer 1d5479adc43e3139 (stream Message writer)
2019-06-28 22:46:02.691642 W | rafthttp: lost the TCP streaming connection with peer 1d5479adc43e3139 (stream MsgApp v2 writer)
2019-06-28 22:46:04.175395 W | etcdserver: failed to reach the peerURL(http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380) of member 1d5479adc43e3139 (Get http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380/version: dial tcp 10.233.83.98:2380: i/o timeout)
2019-06-28 22:46:04.175427 W | etcdserver: cannot get the version of member 1d5479adc43e3139 (Get http://voltha-etcd-cluster-pnn6nzvz5b.voltha-etcd-cluster.voltha.svc:2380/version: dial tcp 10.233.83.98:2380: i/o timeout)