BBSIM Panic on reboot: send on closed channel

time="2023-08-07T13:04:59.310680139Z" level=debug msg="Stopped handling ONU Indication Channel" module=ONU onuID=26 onuSN=BBSM0001041a stream="&{0xc0061f80c0}"
time="2023-08-07T13:04:59.364778109Z" level=debug msg="Send ONU Re-Discovery" IntfId=0 OnuId=11 OnuSn=BBSM0001000b module=ONU
time="2023-08-07T13:04:59.364910479Z" level=debug msg="Changing ONU OperState from down to up" IntfId=0 OnuId=11 OnuSn=BBSM0001000b module=ONU
time="2023-08-07T13:04:59.364957702Z" level=debug msg="Changing ONU InternalState from disabled to initialized" IntfId=0 OnuId=11 OnuSn=BBSM0001000b module=ONU
time="2023-08-07T13:04:59.364989774Z" level=debug msg="Changing ONU InternalState from initialized to discovered" IntfId=0 OnuId=11 OnuSn=BBSM0001000b module=ONU
time="2023-08-07T13:04:59.365059577Z" level=debug msg="Starting ONU Indication Channel" module=ONU onuID=11 onuSN=BBSM0001000b ponPort=0 stream="&{0xc0061f80c0}"
time="2023-08-07T13:04:59.365136353Z" level=debug msg="ONU message handling canceled via channel close" module=ONU onuID=11 onuSN=BBSM0001000b
time="2023-08-07T13:04:59.365156373Z" level=debug msg="Stopped handling ONU Indication Channel" module=ONU onuID=11 onuSN=BBSM0001000b stream="&{0xc0061f80c0}"
time="2023-08-07T13:05:03.14772372Z" level=debug msg=power-on-onu IntfId=0 OnuId=11 SerialNumber=BBSM0001000b module=ONU
panic: send on closed channel

goroutine 50185 [running]:
github.com/opencord/bbsim/internal/bbsim/alarmsim.SimulateOnuAlarm(0xc00b915698, 0xb, 0xc0060ae5a0, 0x0, 0xc000741250)
/app/internal/bbsim/alarmsim/alarmsim.go:228 +0x99
github.com/opencord/bbsim/internal/bbsim/devices.(*Onu).HandlePowerOnONU(0xc00049fce0, 0xc000000005, 0xc00b915b80)
/app/internal/bbsim/devices/onu.go:673 +0x1a2
github.com/opencord/bbsim/internal/bbsim/devices.(*Onu).Reboot(0xc00049fce0, 0x2540be400, 0x1a52cb8, 0x0)
/app/internal/bbsim/devices/onu.go:1372 +0x4dd
github.com/opencord/bbsim/internal/bbsim/devices.(*Onu).handleOmciRequest.func1(0xc00049fce0)
/app/internal/bbsim/devices/onu.go:946 +0x52
created by github.com/opencord/bbsim/internal/bbsim/devices.(*Onu).handleOmciRequest
/app/internal/bbsim/devices/onu.go:945 +0x6d04

ROOT CAUSE ANALYSIS

This is a scenario where we are rebooting an ONU and immediately within 5 seconds we are also rebooting the OLT associated with the ONU using voltctl device reboot. On ONU reboot we have a sleep of 5 seconds sleep (a wait to insure the cleanup) & on OLT reboot we close all the channels, but after 5 seconds ONU resumes and tries to send a ONU re-discovery alarm(SimulateOnuAlarm function) on a closed channel. Therefore the panic happen.

WORKAROUNDS

None

SOLUTIONS

1. On OLT reboot(OLT is set to 'Deleted' state), any request to handle ONU add/discoveries (ONU Power ON/Re-Discovery) can be ignored, also after the OLT reboot(Initialized), OLT and ONUs will itself be discovered. DON'T serve ONU request when the OLT is in DELETED state. NOTE: we don't want to miss any discovery here.