Research Article Open Access

Comparing Approaches for Quality Evaluation of Software Engineering Experiments: An Empirical Study on Software Product Line Experiments

Viviane R. Furtado1, Henrique Vignando1, Victor França 1 and Edson OliveiraJr1
  • 1 State University of Maringá, Brazil


The Software Engineering (SE) research area must provide results of a certain quality for the sake of value. High quality research results may ensure experience and knowledge, which are essential for the technology to be transferred to the industry. One of the means to obtain such quality results is experimentation. Experimentation is a scientific method that aims to provide evidence of a theory over real-world observations establishing a cause-effect relation. Well conducted, auditable and repeatable experiments are vital for scientific evolution and novelty. Quality evaluation of controlled experiments and quasi-experiments in SE has been recently discussed in the literature as researchers desire to assess whether such experiments have improved by reporting information that enables the experiments to be replicated and the reader can understand the experiment and validate results. Thus, this work empirically compares four approaches for quality evaluation of SE experiments in the context of Software Product Lines (SPL). In addition, we are interested on verifying the quality of reporting experiments in a well-discussed reuse technique as SPL. The Pearson technique supported the correlation between pairs of evaluation approaches. In addition, the T-Test and Mann-Whitney-Wilcoxon U test were applied to the samples to verify whether there was a difference in the quality of experiments when using an experimental template. Preliminary results show a strong positive correlation between them, the hypothesis tests confirmed there is such a difference in quality when using experimental template and the SPL experiments report more the planning phase than the analysis and interpretation phase. Based on our results, we provide initial evidence two approaches are the best to reporting SPL experiments.

Journal of Computer Science
Volume 15 No. 10, 2019, 1396-1429


Submitted On: 7 December 2018 Published On: 25 April 2019

How to Cite: Furtado, V. R., Vignando, H., França , V. & OliveiraJr, E. (2019). Comparing Approaches for Quality Evaluation of Software Engineering Experiments: An Empirical Study on Software Product Line Experiments. Journal of Computer Science, 15(10), 1396-1429.

  • 3 Citations



  • Experiments
  • Quasi-Experiments
  • Quality Evaluation of Experiments
  • Software Product Line