-
Type: Story
-
Status: Resolved (View Workflow)
-
Priority: Blocker
-
Resolution: Done
-
Affects Version/s: None
-
Fix Version/s: VOLTHA v2.6
-
Component/s: openonu-adapter
-
Labels:None
-
Story Points:2
-
Epic Link:
https://jenkins.opencord.org/view/openonu-go/job/voltha-openonu-go-scale-measurements-master-2-16-32-dt-subscribers/76/
failing ONUs (hanging in device reason "starting-openomci"):
- onu-adapter receives RPC adopt_device() by rw_core
- onu-adapter receives onu_indication_request-msg by olt-adapter
- three seconds after adopt_device() call has been received, onu-adapter receives RPC reconcile_device() by rw_core
"reconcile-device" call, it happens if openonu adapter restarts. https://jenkins.opencord.org/view/openonu-go/job/voltha-openonu-go-scale-measurements-master-2-16-32-dt-subscribers/76/artifact/logs/pods.txt (check the restarts column).
Concurrent map r/w issue
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: fatal error: concurrent map read and map write
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]:
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: goroutine 80844 [running]:
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.throw(0x1263443, 0x21)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0018b5828 sp=0xc0018b57f8 pc=0x4397b2
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.mapaccess1_faststr(0x106cd40, 0xc00061cdb0, 0xc00115c3c0, 0x24, 0xc002886b38)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/map_faststr.go:21 +0x44f fp=0xc0018b5898 sp=0xc0018b5828 pc=0x41d01f
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: test.internal/openadapter/internal/pkg/onuadaptercore.(*OpenONUAC).getDeviceHandler(0xc0003e80f0, 0xc00115c3c0, 0x24, 0xc0011ff001, 0xc002863708)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/internal/pkg/onuadaptercore/openonu.go:163 +0x243 fp=0xc0018b5960 sp=0xc0018b5898 pc=0xeae553
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: test.internal/openadapter/internal/pkg/onuadaptercore.(*OpenONUAC).Process_inter_adapter_message(0xc0003e80f0, 0xc0011fee70, 0x1d, 0xc0011feff0)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/internal/pkg/onuadaptercore/openonu.go:228 +0x22b fp=0xc0018b59e0 sp=0xc0018b5960 pc=0xeaef5b
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: github.com/opencord/voltha-lib-go/v3/pkg/adapters/common.(*RequestHandlerProxy).Process_inter_adapter_message(0xc0006589c0, 0xc0033acf80, 0x4, 0x4, 0x0, 0x0, 0x0)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/adapters/common/request_handler.go:553 +0x4b6 fp=0xc0018b5a90 sp=0xc0018b59e0 pc=0xc5b9b6
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.call64(0xc0007f2480, 0xc0027c2840, 0xc0024dd300, 0x2000000038)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/asm_amd64.s:540 +0x3b fp=0xc0018b5ae0 sp=0xc0018b5a90 pc=0x464c7b
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: reflect.Value.call(0x120c680, 0xc0006589c0, 0x4613, 0x1229bfb, 0x4, 0xc0033ad020, 0x1, 0x1, 0xc0018b5d70, 0x44f15c, ...)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/reflect/value.go:460 +0x5f6 fp=0xc0018b5d00 sp=0xc0018b5ae0 pc=0x4b6046
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: reflect.Value.Call(0x120c680, 0xc0006589c0, 0x4613, 0xc0033ad020, 0x1, 0x1, 0xc0006589c0, 0x4613, 0xc0004f6000)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/reflect/value.go:321 +0xb4 fp=0xc0018b5d80 sp=0xc0018b5d00 pc=0x4b5804
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: github.com/opencord/voltha-lib-go/v3/pkg/kafka.CallFuncByName(0x120c680, 0xc0006589c0, 0xc00115c360, 0x1d, 0xc0018b5f90, 0x1, 0x1, 0x4, 0x4, 0x40fd15, ...)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/kafka/kafka_inter_container_library.go:685 +0x264 fp=0xc0018b5e38 sp=0xc0018b5d80 pc=0xc37dd4
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: github.com/opencord/voltha-lib-go/v3/pkg/kafka.(*interContainerProxy).handleMessage(0xc00043a0b0, 0xc000914fc0, 0x120c680, 0xc0006589c0)
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/kafka/kafka_inter_container_library.go:744 +0x145d fp=0xc0018b5fc0 sp=0xc0018b5e38 pc=0xc39a1d
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.goexit()
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0018b5fc8 sp=0xc0018b5fc0 pc=0x466911
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: created by github.com/opencord/voltha-lib-go/v3/pkg/kafka.(*interContainerProxy).waitForMessages
default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/kafka/kafka_inter_container_library.go:823 +0x75