Uploaded image for project: 'SEBA'
  1. SEBA
  2. SEBA-393

voltha unable to save device after some time

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: To Do (View Workflow)
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Future
    • Component/s: VOLTHA
    • Labels:
      None

      Description

      We hit an issue on the pod where VOLTHA got into a state where it was unable to save/store the ONU device. We ran the test cases (which does a series of creating/deleting the OLT device through XOS and doing control plane functionality tests in between) a number of times (15-25 times). After some time, the ONU device was stopped being discovered and we observed that VOLTHA was not seeing that device anymore and it got into a stuck loop.

       

      Exception in VOLTHA:

       

      20190104T185427.843 DEBUG MainThread local_handler.ListReachableLogicalDevices {'instance_id': 'vcore-0_1546623561', 'vcore_id': '0001', 'request': , 'event': 'grpc-request'} 20190104T185427.845 ERROR ThreadPoolExecutor-0_3 grpc_utils.in_thread_wrapper {'exception': 'Traceback (most recent call last):\n File "/voltha/common/utils/grpc_utils.py", line 100, in in_thread_wrapper\n result = f.result()\n File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 462, in result\n return self.__get_result()\n File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 414, in __get_result\n raise exception_type, self._exception, self._traceback\nKeyError: \'key id=00011afbec830d0a not found\'', 'e': KeyError('key id=00011afbec830d0a not found',), 'vcore_id': '0001', 'args': (<voltha.core.local_handler.LocalHandler object at 0x7f9b959e1f50>, , <grpc._server._Context object at 0x7f9b1034d550>), 'instance_id': 'vcore-0_1546623561', 'kw': {}, 'func': <function ListReachableLogicalDevices at 0x7f9b8fcba2a8>} 20190104T185427.846 ERROR ThreadPoolExecutor-0_3 _server._call_behavior Exception calling application: 'key id=00011afbec830d0a not found' Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line 390, in _call_behavior return behavior(argument, context), True File "/voltha/common/utils/grpc_utils.py", line 100, in in_thread_wrapper result = f.result() File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 462, in result return self.__get_result() File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 414, in __get_result raise exception_type, self._exception, self._traceback KeyError: 'key id=00011afbec830d0a not found'
      

      VOLTHA CLI:

       

      (logical device 000100000ac004ce) devices(logical device 000100000ac004ce) devicesDevices:+------------------+---------+------+-------------+-------------+----------------+-------------------+|               id |    type | root | admin_state | oper_status | connect_status |     host_and_port |+------------------+---------+------+-------------+-------------+----------------+-------------------+| 00011afbec830d0a | openolt | True |     ENABLED |  ACTIVATING |    UNREACHABLE | 10.192.4.206:9191 |+------------------+---------+------+-------------+-------------+----------------+-------------------+(logical device 000100000ac004ce) portsLogical device ports:table empty(logical device 000100000ac004ce) \(logical device 000100000ac004ce)(logical device 000100000ac004ce) portsLogical device ports:table empty(logical device 000100000ac004ce) devicesEXCEPTION of type '_Rendezvous' occured with message: '<_Rendezvous of RPC that terminated with: status = StatusCode.NOT_FOUND details = "Device '00011afbec830d0a' not found" debug_error_string = "{"created":"@1546628041.702371510","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Device '00011afbec830d0a' not found","grpc_status":5}"
      

      NOTE: All the functionality tests are passing. It just appears that after some load on the system (deleting/re-creating OLT), voltha can get into this state. 
       

       

        Attachments

        # Subject Branch Project Status CR V

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            kailash Kailash Khalasi
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Gerrit Reviews

                There are no open Gerrit changes