Evaluation of Variability Management Approaches
Variability Management (VM) was not a topic of my research interest until I assumed the leadership of two projects in the area of software product lines when I moved to Ireland a few years ago. During my initial observation of the area, I noticed that VM is one of the most researched areas in software product lines. Of course, one reason is that VM is one of the most important activities of software product-line engineering (SPLE). I noticed that software product line community had allocated huge amount of effort to develop various approaches to dealing with variability related challenges during the last two decade. Several dozens of VM approaches have been reported. However, most of the reported approaches appeared to have not been evaluated to support the claims made about those approaches. Hence, we decided to review the status of evaluation of reported VM approaches and to synthesize the available evidence about the effects of the reported approaches. We carried out a systematic literature review of the VM approaches in SPLE reported from 1990s until December 2007. Based on the review of the finally selected 97 papers, we discovered several interesting findings. For example, only a small number of the reviewed approaches had been evaluated using rigorous scientific methods. We found that a significant quality deficiencies in various aspects of the used quality assessment criteria. The synthesis of the available evidence showed that all studies, except one, reported only positive effects.
Our findings have shown that a large majority of the reported VM approaches have not been sufficiently evaluated using scientifically rigorous methods. The available evidence is sparse and the quality of the presented evidence is quite low. The findings highlight the areas in need of improvement, i.e., rigorous evaluation of VM approaches. While I had an observation that VM approaches generally lacked rigorous evaluation, now I am glad that we have found the evidence to support our observation through a systematic review. We are hoping that our findings convince the researchers in this area to pay more attention to systematically and rigorously evaluate the VM approaches and tools to support the technology transfer efforts. Our paper has been published in the Journal of Information and Software Technology this year.