Uploaded image for project: 'VOLTHA'
  1. VOLTHA
  2. VOL-3637

Openonu go adapter restart due to concurrent map access

    XMLWordPrintable

    Details

      Description

      https://jenkins.opencord.org/view/openonu-go/job/voltha-openonu-go-scale-measurements-master-2-16-32-dt-subscribers/76/
      failing ONUs (hanging in device reason "starting-openomci"):

      • onu-adapter receives RPC adopt_device() by rw_core
      • onu-adapter receives onu_indication_request-msg by olt-adapter
      • three seconds after adopt_device() call has been received, onu-adapter receives RPC reconcile_device() by rw_core

      "reconcile-device" call, it happens if openonu adapter restarts. https://jenkins.opencord.org/view/openonu-go/job/voltha-openonu-go-scale-measurements-master-2-16-32-dt-subscribers/76/artifact/logs/pods.txt (check the restarts column).
      Concurrent map r/w issue
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: fatal error: concurrent map read and map write
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]:
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: goroutine 80844 [running]:
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.throw(0x1263443, 0x21)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0018b5828 sp=0xc0018b57f8 pc=0x4397b2
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.mapaccess1_faststr(0x106cd40, 0xc00061cdb0, 0xc00115c3c0, 0x24, 0xc002886b38)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/map_faststr.go:21 +0x44f fp=0xc0018b5898 sp=0xc0018b5828 pc=0x41d01f
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: test.internal/openadapter/internal/pkg/onuadaptercore.(*OpenONUAC).getDeviceHandler(0xc0003e80f0, 0xc00115c3c0, 0x24, 0xc0011ff001, 0xc002863708)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/internal/pkg/onuadaptercore/openonu.go:163 +0x243 fp=0xc0018b5960 sp=0xc0018b5898 pc=0xeae553
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: test.internal/openadapter/internal/pkg/onuadaptercore.(*OpenONUAC).Process_inter_adapter_message(0xc0003e80f0, 0xc0011fee70, 0x1d, 0xc0011feff0)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/internal/pkg/onuadaptercore/openonu.go:228 +0x22b fp=0xc0018b59e0 sp=0xc0018b5960 pc=0xeaef5b
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: github.com/opencord/voltha-lib-go/v3/pkg/adapters/common.(*RequestHandlerProxy).Process_inter_adapter_message(0xc0006589c0, 0xc0033acf80, 0x4, 0x4, 0x0, 0x0, 0x0)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/adapters/common/request_handler.go:553 +0x4b6 fp=0xc0018b5a90 sp=0xc0018b59e0 pc=0xc5b9b6
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.call64(0xc0007f2480, 0xc0027c2840, 0xc0024dd300, 0x2000000038)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/asm_amd64.s:540 +0x3b fp=0xc0018b5ae0 sp=0xc0018b5a90 pc=0x464c7b
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: reflect.Value.call(0x120c680, 0xc0006589c0, 0x4613, 0x1229bfb, 0x4, 0xc0033ad020, 0x1, 0x1, 0xc0018b5d70, 0x44f15c, ...)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/reflect/value.go:460 +0x5f6 fp=0xc0018b5d00 sp=0xc0018b5ae0 pc=0x4b6046
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: reflect.Value.Call(0x120c680, 0xc0006589c0, 0x4613, 0xc0033ad020, 0x1, 0x1, 0xc0006589c0, 0x4613, 0xc0004f6000)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/reflect/value.go:321 +0xb4 fp=0xc0018b5d80 sp=0xc0018b5d00 pc=0x4b5804
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: github.com/opencord/voltha-lib-go/v3/pkg/kafka.CallFuncByName(0x120c680, 0xc0006589c0, 0xc00115c360, 0x1d, 0xc0018b5f90, 0x1, 0x1, 0x4, 0x4, 0x40fd15, ...)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/kafka/kafka_inter_container_library.go:685 +0x264 fp=0xc0018b5e38 sp=0xc0018b5d80 pc=0xc37dd4
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: github.com/opencord/voltha-lib-go/v3/pkg/kafka.(*interContainerProxy).handleMessage(0xc00043a0b0, 0xc000914fc0, 0x120c680, 0xc0006589c0)
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/kafka/kafka_inter_container_library.go:744 +0x145d fp=0xc0018b5fc0 sp=0xc0018b5e38 pc=0xc39a1d
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: runtime.goexit()
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0018b5fc8 sp=0xc0018b5fc0 pc=0x466911
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: created by github.com/opencord/voltha-lib-go/v3/pkg/kafka.(*interContainerProxy).waitForMessages
      default/open-onu-voltha-adapter-openonu-6df5b98dfc-7x4g5[adapter-open-onu]: /go/src/github.com/opencord/voltha-openonu-adapter-go/vendor/github.com/opencord/voltha-lib-go/v3/pkg/kafka/kafka_inter_container_library.go:823 +0x75

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

            Assignee:
            ggowdra Girish Gowdra
            Reporter:
            andrea Andrea Campanella
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes