Uploaded image for project: 'VOLTHA'
  1. VOLTHA
  2. VOL-1983

robot system tests continues after failure, even if failure means additional tests will fail or are meaningless

    XMLWordPrintable

    Details

      Description

      In the following example, the test fails authentication but continues to authorize the subscriber. these tests are really a single use case where each step builds on the other. Would it be better to fail everything after the first fail?

      robot -v ONOS_REST_PORT:8181 -v ONOS_SSH_PORT:8101 --exclude notready --critical sanity -v num_onus:1 sanity.robot
      ==============================================================================
      Sanity :: Creates bbsim olt/onu and validates activataion Assumes voltha-go...
      ==============================================================================
      Activate Device BBSIM OLT/ONU :: Validate deployment -> create and... | PASS |
      ------------------------------------------------------------------------------
      Validate OLT Connected to ONOS :: Verifies the BBSIM-OLT device is... | PASS |
      ------------------------------------------------------------------------------
      Check EAPOL Flows in ONOS :: Validates eapol flows for the onu are... | PASS |
      ------------------------------------------------------------------------------
      Validate ONU Authenticated in ONOS :: Validates onu is AUTHORIZED ... | FAIL |
      Keyword 'Verify Number of AAA-Users' failed after retrying for 1 minute 30 seconds. The last error was: ' 0' does not contain '1'
      ------------------------------------------------------------------------------
      Add Subscriber-Access in ONOS :: Through the olt-app in ONOS, exec... | PASS |
      ------------------------------------------------------------------------------
      Validate DHCP Assignment in ONOS :: After IP Flows are pushed to t... | PASS |
      ------------------------------------------------------------------------------
      Delete Device and Verify :: Disable -> Delete devices via voltctl ... | FAIL |
      {{Keyword 'Validate Device Removed' failed after retrying for 1 minute. The last error was: [ 7b34f014f750be39b2019740 ] contains value '7b34
      ------------------------------------------------------------------------------
      Sanity :: Creates bbsim olt/onu and validates activataion Assumes ... | FAIL |
      7 critical tests, 5 passed, 2 failed
      7 tests total, 5 passed, 2 failed

        Attachments

        # Subject Branch Project Status CR V

          Activity

            People

            Assignee:
            dbainbri David Bainbridge
            Reporter:
            dbainbri David Bainbridge
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes