[VOL-3419] OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows - OpenCORD JIRA

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Medium
Resolution: Fixed
Affects Version/s: VOLTHA v2.5
Fix Version/s: VOLTHA v2.6
Component/s: openolt-adapter, openolt-agent
Labels:
None

Story Points:
5
Epic Link:
Maintenance

Description

We have observed that the OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows.

Few ideas are discussed in document
https://docs.google.com/document/d/1Q_VAfBg-kSR-ATS5JgC57rZboQ-aeNhSm4tINOiFcCE/edit#heading=h.4x5eh1mw3ynt to make OpenOltFlowManager per PON port that could limit the lock to a PON port rather than the whole device and could provide performance improvements.
Also the openolt adapter code needs to be reviewed for any redundant locks or re-work data structures that are stored on the KV store which could potentially give performance improvements.

Attachments

Options
- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

Gerrit Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Girish Gowdra

Reporter:: Girish Gowdra

Watchers:: 1 Start watching this issue

Dates

Created:: 2020/08/16 5:38 PM

Updated:: 2020/10/26 8:47 AM

Resolved:: 2020/10/26 8:47 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 10 closed Gerrit changes

Hide There are 10 closed Gerrit changes

This is a WIP patch VOL-3419: OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows The patch addresses the following - Create OpenOltFlowMgr per PON port (instead of one instance for the whole OLT device earlier) - Create a separate OpenOltGroupMgr - currently one instance for the whole OLT device - Re-organize the DeviceHandler and OpenOltFlowMgr modules to seggregate exported and unpexported functions - Remove redundant global lock around getting ONU-ID in DeviceHandler module as there exists a separate per-pon-port lock in ResourceManager module which suffices the required synchronization - Remove redundant locks in OpenOltFlowMgr module to serialize FlowDelete before FlowAdd - Rename divideAndAddFlow to processAddFlow. "divideAndAddFlow" was used in 1.x voltha days and had a different meaning and the name seems to have been blindly ported to 2.x adapter and does not make sense anymore - More enhancements, performance improvements and code-reorg to follow in subsequent patches: Gerrit Review:

VOL-3419: Replicate voltha flows in openolt agent Pass the following additional information in the Flow message for agent to replicate the flow - voltha flow_id - symmetric voltha_flow_id (where applicable) - flag to indicate whether the flow is to be replicated - when flow is to be replicated, the pbit to gemport map - Bump major version to 4.0.0 since this change is NOT backward compatible: Gerrit Review:

VOL-3419: Update the tools image to use grpc c++ 1.31.1. The latest grpc c++ version provides mechanisms to fine tune thread pool settings in 'sync' mode of grpc API execution. This version of grpc will be used to acheive some high performance when agent takes up more task later to replicate the flows instead of adapter doing it for the agent.: Gerrit Review:

VOL-3419: Replicate voltha flows in openolt agent - Use the flow_id, symmetric_flow_id, replication_flag and pbit_to_gemport_map coming in Flow proto messge to replicate the flow as needed - Use the CreateQueues and RemoveQueues messages to setup and remove gem_ports - Use latest gRPC version 1.31.1 which allows fine tuning of gRPC threadpools which allows for greating performance. - Performance numbers when tested with openolt-scale-tester has been better with using latest gRPC and threadpool tuning when compared to earlier. It is to be noted that the performance is better even with openolt-agent replicating the flow now. - Scale tests with 512 subscribers have been successfull with BAL3.4.7.5 version - Use openolt proto version 4.0.0 - Use openolt-test (for unit test) image version 2.0.1 (which bundles latest gRPC version 1.31.1) - These changes are NOT backward compatible and openolt-agent will have a major version bump to 3.0.0: Gerrit Review:

VOL-3419: Replicate voltha flows in openolt agent - The scale-tester-app will adhere to new openolt.proto interface (version 4.0.2) and will pass necessary information for openolt-agent to replicate the flows. - upgrade to voltha-lib-go version 4.0.0: Gerrit Review:

VOL-3419: OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows The patch addresses the following - Create OpenOltFlowMgr per PON port (instead of one instance for the whole OLT device earlier) - Create a separate OpenOltGroupMgr - currently one instance for the whole OLT device - Remove redundant global lock around getting ONU-ID in DeviceHandler module as there exists a separate per-pon-port lock in ResourceManager module which suffices the required synchronization - Remove redundant locks in OpenOltFlowMgr module to serialize FlowDelete before FlowAdd - Rename divideAndAddFlow to processAddFlow. "divideAndAddFlow" was used in 1.x voltha days and had a different meaning and the name seems to have been blindly ported to 2.x adapter and does not make sense anymore: Gerrit Review:

Cherry-picking VOL-3419 and releasing 2.5.6: Gerrit Review:

VOL-3419: OpenOLT adapter at scale constantly takes more that 10 seconds to react to flows - Pass information to agent to do the flow replication - Consolidate various locks in the adapter and remove reduntant locks - use voltha-proto version 4.0.2 and voltha-lib-go version 4.0.0 - Bump adapter version to 3.0.0: Gerrit Review:

[VOL-3419] Supporting new flow format. Releasing version 1.3.0: Gerrit Review:

VOL-3419: - Make iScheduler and iScheduler public - Bump version to 3.3.0: Gerrit Review: