A Map of Threats to Validity of Systematic Literature Reviews in Software Engineering

Citations Over TimeTop 10% of 2016 papers

Abstract

Context: The assessment of Threats to Validity (TTVs) is critical to secure the quality of empirical studies in Software Engineering (SE). In the recent decade, Systematic Literature Review (SLR) was becoming an increasingly important empirical research method in SE. One of the mechanisms of insuring the level of scientific value in the findings of an SLR is to rigorously assess its validity. Hence, it is necessary to realize the status quo and issues of TTVs of SLRs in SE. Objective: This study aims to investigate the-state-of-the-practice of TTVs of the SLRs published in SE, and further support SE researchers to improve the assessment and strategies against TTVs in order to increase the quality of SLRs in SE. Method: We conducted a tertiary study by reviewing the SLRs in SE that report the assessment of TTVs. Results: We identified 316 SLRs published from 2004 to the first half of 2015, in which TTVs are discussed. The issues associated to TTVs were also summarized and categorized. Conclusion: The common TTVs related to SLR research, such as internal validity and reliability, were thoroughly discussed in most SLRs. The threats to construct validity and external validity drew less attention. Moreover, there are few strategies and tactics being reported to cope with the various TTVs.

Related Papers

• → Internal, External, and Ecological Validity in Research Design, Conduct, and Evaluation(2018)305 cited
• → The internal validity obsession(2022)3 cited
• → The Quality of Accounting Research and Internal, External, and Construct Validity(1999)1 cited
• → Validity: Definitions and Applications to Psychiatric Research(2002)13 cited
• → Validity in Experimentation(2010)1 cited