-
Type: Bug
-
Status: To Do (View Workflow)
-
Priority: Medium
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: Future
-
Component/s: VOLTHA
-
Labels:None
-
Story Points:3
-
Epic Link:
We hit an issue on the pod where VOLTHA got into a state where it was unable to save/store the ONU device. We ran the test cases (which does a series of creating/deleting the OLT device through XOS and doing control plane functionality tests in between) a number of times (15-25 times). After some time, the ONU device was stopped being discovered and we observed that VOLTHA was not seeing that device anymore and it got into a stuck loop.
Exception in VOLTHA:
20190104T185427.843 DEBUG MainThread local_handler.ListReachableLogicalDevices {'instance_id': 'vcore-0_1546623561', 'vcore_id': '0001', 'request': , 'event': 'grpc-request'} 20190104T185427.845 ERROR ThreadPoolExecutor-0_3 grpc_utils.in_thread_wrapper {'exception': 'Traceback (most recent call last):\n File "/voltha/common/utils/grpc_utils.py", line 100, in in_thread_wrapper\n result = f.result()\n File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 462, in result\n return self.__get_result()\n File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 414, in __get_result\n raise exception_type, self._exception, self._traceback\nKeyError: \'key id=00011afbec830d0a not found\'', 'e': KeyError('key id=00011afbec830d0a not found',), 'vcore_id': '0001', 'args': (<voltha.core.local_handler.LocalHandler object at 0x7f9b959e1f50>, , <grpc._server._Context object at 0x7f9b1034d550>), 'instance_id': 'vcore-0_1546623561', 'kw': {}, 'func': <function ListReachableLogicalDevices at 0x7f9b8fcba2a8>} 20190104T185427.846 ERROR ThreadPoolExecutor-0_3 _server._call_behavior Exception calling application: 'key id=00011afbec830d0a not found' Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line 390, in _call_behavior return behavior(argument, context), True File "/voltha/common/utils/grpc_utils.py", line 100, in in_thread_wrapper result = f.result() File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 462, in result return self.__get_result() File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 414, in __get_result raise exception_type, self._exception, self._traceback KeyError: 'key id=00011afbec830d0a not found'
VOLTHA CLI:
(logical device 000100000ac004ce) devices(logical device 000100000ac004ce) devicesDevices:+------------------+---------+------+-------------+-------------+----------------+-------------------+| id | type | root | admin_state | oper_status | connect_status | host_and_port |+------------------+---------+------+-------------+-------------+----------------+-------------------+| 00011afbec830d0a | openolt | True | ENABLED | ACTIVATING | UNREACHABLE | 10.192.4.206:9191 |+------------------+---------+------+-------------+-------------+----------------+-------------------+(logical device 000100000ac004ce) portsLogical device ports:table empty(logical device 000100000ac004ce) \(logical device 000100000ac004ce)(logical device 000100000ac004ce) portsLogical device ports:table empty(logical device 000100000ac004ce) devicesEXCEPTION of type '_Rendezvous' occured with message: '<_Rendezvous of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Device '00011afbec830d0a' not found" debug_error_string = "{"created":"@1546628041.702371510","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Device '00011afbec830d0a' not found","grpc_status":5}"
NOTE: All the functionality tests are passing. It just appears that after some load on the system (deleting/re-creating OLT), voltha can get into this state.
# | Subject | Branch | Project | Status | CR | V |
---|---|---|---|---|---|---|
12351,2 | SEBA-393 adding bug to known-issues | master | docs | Status: MERGED | +2 | +1 |