Research Methods for Software Engineering Students

The Software Engineering community is increasingly recognising the value of empirical evidence to support research and practice. Empirical evidence provides a means to evaluate the utility of promising research areas and to help practitioners to make informed technology adoption decisions. Hence, there is an increasing need for providing software engineering researchers and practitioners with appropriate knowledge and training in different methods and techniques to design, execute, analyze, and report empirical research. Software Engineering degree programs (both undergraduate and postgraduate) are expected to have high quality courses on Research Methods for imparting appropriate knowledge and skills of designing, connecting, analysing, and reporting empirical studies. There are an increasing number of books and articles on research methods, however, many Software Engineering degree programs appear to find it difficult to come up with a high quality course that can impart the knowledge, understanding, and skills of carrying out both quantitive and qualitative research a wide variety of topics in Software Engineering. Based on several tutorials We are also working on designing and running a course on research methods for our Software Engineering students. 

Key goals for a short course on empirical research methods in Software Engineering:

  1. Understand various concepts and terminology underpinning empirical research
  2. Gain detailed knowledge about different empirical research methods and their strengths and limitations.
  3. How to evaluate and select an appropriate empirical research method for a certain research question.
  4. How to design, execute, analyze, and published empirical studies to assess and/or guide software engineering research.
  5. How to identify and assess empirical evidence to support software engineering technology evaluation and selection.
  6. Appreciate the ethical issues involved in empirical studies involving humans.
  7. Comprehend the challenges involved in conducting empirical research and strategies to deal with them.

Apart from covering key aspects of statistics and data analysis tools like “R”, a suitable course on empirical methods for Software Engineering is expected to cover the following topics.

Introduction and key components

  1. Why empirical research?
  2. Empirical approaches
    1. Descriptive
    2. Rational
    3. Experimental/Analytical
  3. Empirical research in software engineering
  4. General concepts and terminology
  5. Generic process of empirical research
  6. Main data collection techniques
  7. Example studies

Research methods

  1. Systematic literature review
  2. Case studies
  3. Action research
  4. Surveys
  5. Controlled experiments

Critical but usually overlooked aspects

  1. Ethical issues in empirical research
  2. Challenges of conducting empirical research
  3. Discussion and summary

Sample Material

Following are some of the research articles that can provide students with a good understanding of the key areas of empirical research methods for Software Engineering:

[1] B.A. Kitchenham, et al., Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, , 2002. 28(8): pp. 721-734.

[2] A. Jedlitschka and D. Pfahl, Reporting Guidelines for Controlled Experiments in Software Engineering, Proceedings of the International Symposium on Empirical Software Engineering, 2005.

[3] B. Kitchenham, et al., Evaluating Guidelines for Empirical Software Engineering Studies, Proceedings of the International Symposium on Empirical Software Engineering, 2006.

[4] B. Kitchenham and S.L. Pfleeger, Personal Opinion Surveys, in F. Shull, J. Singer & D. Sjøberg (Eds.), Guide to Advanced Empirical Software Engineering, Springer-Verlag, 2008, pp. 63-92.

[5] D. Perry, S.E. Sim, and S. Easterbrook, Case Studies for Software Engineers, Proc. of the 26th Int’l Conference on Software Engineering, 2004.

[6] B. Kitchenham, L. Pickard, and S.L. Pfleeger, Case Studies for Method and Tool Evaluation, IEEE Software, 1995. 12(4): pp. 52-62.

[7] J. Singer and N.G. Vinson, Ethical Issues in Empirical Studies of Software Engineering, IEEE Transactions on Software Engineering, 2002. 28(12): pp. 1171-1180.

[8] T.C. Lethbridge, S.E. Sim, and J. Singer, Studying Software Engineers: Data Collection Techniques for Software Field Studies, Empirical Software Engineering, 2005. 10: pp. 311-341.

[9] P. Brereton, et al., Lessons from applying the systematic literature review process within the software engineering domain, Journal of Systems and Software, 2007. 80: pp. 571-583.

[10]. D. Sjøberg, T. Dybå and M. Jørgensen, The Future of Empirical Methods in Software Engineering Research, 29th International Conference on Software Engineering (ICSE’07), Minneapolis, Minnesota, USA, 20-26 May, Future of Software Engineering (FoSE’07), in L. Briand and A. Wolf (Eds.), IEEE Computer Society Press, 2007, pp. 358-378.

[11] T. Dybå, B. Kitchenham and M. Jørgensen, Evidence-based Software Engineering for Practitioners, IEEE Software, 2005, 22(1): pp. 58-65.

The following book has several good chapters written by experts of different topics of empirical Software Engineering

Shull, J. Singer and D. Sjøberg (Eds.), Guide to Advanced Empirical Software Engineering, Springer-Verlag, London, 2008.