Dr. Ambar Sarkar, Chief Verification Technologist, Paradigm Works Inc.
Have you been frustrated by how long it takes to run the initial phases of a test before it gets to those that really matter? And it gets worse when you need to run a number of these tests. I often see projects where about 80% of the test execution time is used up in these initial phases.
While you do need to execute these initial phases, repeating them for each test does not contribute to any additional verification. Ideally, you would like to run the initial phases for a group of related tests just once, just to a point where you are ready to send real traffic or transactions. The phases that follow should be executed once per test, with the environment rolled back to an appropriate state before the next test starts executing.
The diagram below shows how you may want chain two testsTEST1 and TEST2, where you want to use the same configuration for both.
VMM1.2 supports this model of test execution for implicitly phased environments.
First the steps.
Step 1. Tag a test as concatenable and specify the rewind-to phase.?
If you are using implicit phasing of your environment, this is done simply by using the macro that announces the tests as concatenable, along with the phase to roll_back to.
class TEST1 extends vmm_test;
//Macro to indicate which phase to rollback to
`vmm_test_concatenate(configure_test)
…
endclass
class TEST2 extends vmm_test;
//Macro to indicate which phase to rollback to
`vmm_test_concatenate(configure_test)
…
endclass
Step 2. Invoke it.
This is done by simply specifying the command line parameters:
//Command line arguments to run the example
./simv +vmm_test=TEST1+TEST2
You also have the option of specifying all tests in a file or just chain all the tests.
So what are the gotchas?
The underlying VMM implementation (and you can read the gory details of how this is done using the three timelines: pre, top, and post in the VMM user guide) takes care of synchronizing all the test phases so that the tests can be effectively chained. However, the user still has to implement any application specific cleanup in between tests to restore the environment. This seems fair enough, and the cleanup phase is the perfect phase to implement that.
However, there is one limitation when considering chained tests. You cannot use vmm_test::set_config() in concatenated tests, which is the place to perform class-factory overrides for the environment components. This is not a limitation of the implementation however, but just how class factory overrides work. Note that you can still use class factory to override your traffic scenarios, just not the environment components.
First, a test can only run after the environment is created. Second, a class factory override for the environment can work only if it is called before the environment is created. This means that after the first test is executed, you cannot override the environment setting using class factory override, unless you are willing to rebuild the environment from scratch in between. Clearly, this rebuild per test run will negate any benefits of chaining. Since vmm_test::set_config is explicitly used to reconfigure environments using the class factory method, it is not allowed when tests are chained.
This article is the 5th in the Verification in the trenches series. Hope you found this article useful. If you would like to hear about any other related topic, please comment or drop me a line at ambar.sarkar@paradigm-works.com. Also, if you are starting out fresh, please check out the free VMM1.2 environment generator at http://resourceworks.paradigm-works.com/svftg/vmm .
ShareThis
Posted in
