Uploaded image for project: 'VOLTHA'
  1. VOLTHA
  2. VOL-3419

OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows

    XMLWordPrintable

    Details

      Description

      We have observed that the OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows.

      Few ideas are discussed in document 
      https://docs.google.com/document/d/1Q_VAfBg-kSR-ATS5JgC57rZboQ-aeNhSm4tINOiFcCE/edit#heading=h.4x5eh1mw3ynt to make OpenOltFlowManager per PON port that could limit the lock to a PON port rather than the whole device and could provide performance improvements.
      Also the openolt adapter code needs to be reviewed for any redundant locks or re-work data structures that are stored on the KV store which could potentially give performance improvements.

        Attachments

        # Subject Branch Project Status CR V
        20138,8 This is a WIP patch VOL-3419: OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows The patch addresses the following - Create OpenOltFlowMgr per PON port (instead of one instance for the whole OLT device earlier) - Create a separate OpenOltGroupMgr - currently one instance for the whole OLT device - Re-organize the DeviceHandler and OpenOltFlowMgr modules to seggregate exported and unpexported functions - Remove redundant global lock around getting ONU-ID in DeviceHandler module as there exists a separate per-pon-port lock in ResourceManager module which suffices the required synchronization - Remove redundant locks in OpenOltFlowMgr module to serialize FlowDelete before FlowAdd - Rename divideAndAddFlow to processAddFlow. "divideAndAddFlow" was used in 1.x voltha days and had a different meaning and the name seems to have been blindly ported to 2.x adapter and does not make sense anymore - More enhancements, performance improvements and code-reorg to follow in subsequent patches master voltha-openolt-adapter Status: ABANDONED 0 +1
        20145,9 VOL-3419: Replicate voltha flows in openolt agent Pass the following additional information in the Flow message for agent to replicate the flow - voltha flow_id - symmetric voltha_flow_id (where applicable) - flag to indicate whether the flow is to be replicated - when flow is to be replicated, the pbit to gemport map - Bump major version to 4.0.0 since this change is NOT backward compatible master voltha-protos Status: MERGED +2 +1
        20373,1 VOL-3419: Update the tools image to use grpc c++ 1.31.1. The latest grpc c++ version provides mechanisms to fine tune thread pool settings in 'sync' mode of grpc API execution. This version of grpc will be used to acheive some high performance when agent takes up more task later to replicate the flows instead of adapter doing it for the agent. master openolt-test Status: MERGED +2 +1
        20385,13 VOL-3419: Replicate voltha flows in openolt agent - Use the flow_id, symmetric_flow_id, replication_flag and pbit_to_gemport_map coming in Flow proto messge to replicate the flow as needed - Use the CreateQueues and RemoveQueues messages to setup and remove gem_ports - Use latest gRPC version 1.31.1 which allows fine tuning of gRPC threadpools which allows for greating performance. - Performance numbers when tested with openolt-scale-tester has been better with using latest gRPC and threadpool tuning when compared to earlier. It is to be noted that the performance is better even with openolt-agent replicating the flow now. - Scale tests with 512 subscribers have been successfull with BAL3.4.7.5 version - Use openolt proto version 4.0.0 - Use openolt-test (for unit test) image version 2.0.1 (which bundles latest gRPC version 1.31.1) - These changes are NOT backward compatible and openolt-agent will have a major version bump to 3.0.0 master openolt Status: MERGED +2 +1
        20399,7 VOL-3419: Replicate voltha flows in openolt agent - The scale-tester-app will adhere to new openolt.proto interface (version 4.0.2) and will pass necessary information for openolt-agent to replicate the flows. - upgrade to voltha-lib-go version 4.0.0 master openolt-scale-tester Status: MERGED +2 +1
        20423,4 VOL-3419: OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows The patch addresses the following - Create OpenOltFlowMgr per PON port (instead of one instance for the whole OLT device earlier) - Create a separate OpenOltGroupMgr - currently one instance for the whole OLT device - Remove redundant global lock around getting ONU-ID in DeviceHandler module as there exists a separate per-pon-port lock in ResourceManager module which suffices the required synchronization - Remove redundant locks in OpenOltFlowMgr module to serialize FlowDelete before FlowAdd - Rename divideAndAddFlow to processAddFlow. "divideAndAddFlow" was used in 1.x voltha days and had a different meaning and the name seems to have been blindly ported to 2.x adapter and does not make sense anymore master voltha-openolt-adapter Status: MERGED +2 +1
        20494,2 Cherry-picking VOL-3419 and releasing 2.5.6 voltha-2.5 voltha-openolt-adapter Status: MERGED +2 +1
        20505,16 VOL-3419: OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows - Pass information to agent to do the flow replication - Consolidate various locks in the adapter and remove reduntant locks - use voltha-proto version 4.0.2 and voltha-lib-go version 4.0.0 - Bump adapter version to 3.0.0 master voltha-openolt-adapter Status: MERGED +2 +1
        20998,9 [VOL-3419] Supporting new flow format. Releasing version 1.3.0 master bbsim Status: MERGED +2 +1
        21121,2 VOL-3419: - Make iScheduler and iScheduler public - Bump version to 3.3.0 master voltha-lib-go Status: MERGED +2 +1

          Activity

            People

            Assignee:
            ggowdra Girish Gowdra
            Reporter:
            ggowdra Girish Gowdra
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes