TUTORIAL SESSION

Regression Testing: Theoretical Underpinnings, Practical Techniques, and Empirical Insights

Assc. Prof. Gregory M. Kapfhammer
Allegheny College, USA
gkapfham@allegheny.edu

Duration

Three Hours

Abstract

According to a federal study published in 2002, software defects cost the economy of the United States of America an estimated $59.5 billion each year. In response to this crisis, researchers and practitioners develop, use, and evaluate testing methods that analyze computer software in order to establish a confidence in its correctness and identify lurking faults. Regression testing, or the process of repeatedly running a test suite whenever the computer software changes, is one important approach to improving software quality and reducing the costs of defective software.
This tutorial will explain how to effectively employ key regression testing techniques such as test suite selection, reduction, and prioritization. After intuitively highlighting the theoretical and conceptual underpinnings of regression testing, the tutorial will enable attendees to apply practical testing methods, as implemented in readily available free and open source software, to real-world software applications. This hands-on and relevant tutorial will also help audience members use strategies and frameworks to empirically evaluate the efficiency and effectiveness of different regression testing methods.

Objectives

In our modern world, software is ... everywhere! Regrettably, many software applications contain serious defects that lead to substantial economic costs. Regression testing, or the repeated execution of a test suite when a computer program changes, is one of the best-known methods for finding defects in the software that is now an everyday part of our lives. While certain regression testing techniques now see limited use in industry, many useful ideas have not transitioned into practice. Both the explosive growth in regression testing research and the lack of a comprehensive reference text often leaves both scholars and practitioners scrambling to find the best technique for solving a new problem. Using a clear, intuitive, and hands-on method of instruction, the tutorial will allow attendees to develop and hone their skills in an important, yet often neglected, field of study. SEA 2011 delegates who opt to attend this tutorial will be equipped to use regression testing to better develop high quality software and ultimately researchers.

Timeline

• Introduction to Regression Testing (30 minutes)


• Regression Testing Techniques (100 minutes)
  
  • Model of Regression Testing
  
  
  • Test Adequacy Criteria
  
  
  • Test Suite Execution*
  
  
  • Test Coverage Monitoring*
  
  
  • Reducing and Prioritizing Test Suites*
    
    
    
    1. Greedy Methods
    
    
    2. Search-based Techniques
    
    
    3. Additional Approaches
    
    
    
    
  • Test Suite Selection
  
  
  • Affiliated Regression Testing Methods*
  
  


• Empirical Evaluation of Regression Testing Techniques* (30 minutes)


• Conclusions and Future Directions (15 minutes)


• Further Resources (5 minutes)

*Denotes a section that includes hands-on activities where attendees with laptops will use free and open source software to learn how to apply a specific regression testing technique or evaluation method. If permitted by the conference organizers, the instructor will furnish laptops to interested attendees so that they can easily participate in all of the tutorial's exercises.

Target Audience

A participant should have a basic knowledge of the theory and/or practice of software engineering in any application domain (e.g., databases, graphics, medical informatics, or e-Commerce). Since the presenters will use easy-to-understand diagrams, examples, and hands-on activities, attendees do not need to have a background in the theory and practice of regression testing in order to benefit from the tutorial. The presenters will also give all participants color versions of all tutorial resources (e.g., presentation slides and instruction manuals) and agree to communicate with all interested attendees after the presentation, thus ensuring that active learning and collaboration will continue beyond the meeting dates of the 15th IASTED International Conference on Software Engineering and Applications.

Presenters

• Assc. Prof. Gregory M. Kapfhammer – Allegheny College, USA
• Jonathan Miller Kauffman – Allegheny College, USA

Qualifications of the Instructor(s)

Dr. Gregory M. Kapfhammer is an Associate Professor in the Department of Computer Science at Allegheny College. As an educator, Dr. Kapfhammer develops and teaches courses in the areas of software engineering, software testing, data management, and distributed systems. He is also involved in the teaching of the department's introductory courses, junior seminar, senior seminar, and a research adviser for undergraduate theses. Kapfhammer's expertise in teaching was recently recognized when Allegheny College presented him with the Thoburn Award for Excellence in Teaching. Dr. Kapfhammer's interactive and engaging teaching style will carry over into the tutorial as he seeks to actively involve the attendees in learning more about the topic of regression testing.
As an experimental computer scientist, Dr. Kapfhammer conducts research in the areas of software engineering, software testing and analysis, and computer software systems. In collaboration with a diverse and highly skilled group of students and colleagues, Kapfhammer pursues research that results in award-winning and frequently-cited research papers and useful free and open source software. The paper "A Family of Test Adequacy Criteria for Database-Driven Applications," co-authored by Kapfhammer and Dr. Mary Lou Soffa, was selected to receive the ACM SIGSOFT Distinguished Paper Award at the 2003 ACM SIGSOFT Foundations of Software Engineering Conference (ESEC/FSE 2003). Furthermore, Kapfhammer's paper "Empirically Studying the Role of Selection Operators During Search-Based Test Suite Prioritization," co-authored with Alexander P. Conrad and Dr. Robert S. Roos, was recognized as the best paper in the Search-Based Software Engineering Track of the 2010 ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO 2010). Three examples of Kapfhammer's free and open source software testing tools and experimentation frameworks include:

• RAISE: Methods for Reducing and Prioritizing Test Suites
  http://raise.googlecode.com/


• GELATIONS: Test Prioritization Techniques Using Genetic Algorithms
  http://gelations.googlecode.com/


• Proteja: Techniques for Test Suite Management
  http://proteja.googlecode.com/


• Modificare: Framework for Experiments in Regression Testing
  http://modificare.googlecode.com/

Dr. Kapfhammer has published nearly forty refereed research papers in top conferences, workshops, journals, and books (e.g., ACM International Symposium on Software Testing and Analysis, ACM International Symposium on the Foundations of Software Engineering, ACM/IEEE International Conference on Automated Software Engineering, International Workshop on the Automation of Software Test, and the International Workshop on Regression Testing). Among Kapfhammer's publications is the "Regression Testing" chapter in the 2010 edition of the Taylor and Francis Encyclopedia of Software Engineering. On nearly one hundred separate occasions, Dr. Kapfhammer has served the research community as a reviewer for conferences and journals, technical program committee member, conference program co-chair, workshop co-chair, and review panelist at the U.S. National Science Foundation. In the past, Dr. Kapfhammer held the position of Program Co-Chair for the 2nd and 3rd International Testing: Academic and Industrial Conference - Practice and Research Techniques (TAIC PART 2008 and 2009) and the 6th International Workshop on the Automation of Software Test (AST 2010). It is also important to note that Kapfhammer, in conjunction with Dr. Shin Yoo (University College London), will serve as the Program Co-Chair for the forthcoming International Workshop on Regression Testing (Regression 2012) .

Jonathan Miller Kauffman is a student and research assistant in the Department of Computer Science at Allegheny College. In the past Kauffman has worked in the area of computer science education and he is currently conducting regression testing research. Jonathan Miller Kauffman recently co-authored a fast abstract entitled "Empirically Identifying the Best Genetic Algorithm for Covering Array Generation," which was presented at the Third International Symposium on Search Based Software Engineering (SSBSE 2011). Kauffman also works as a teaching assistant in Allegheny College's introductory computer science courses. Jonathan Miller Kauffman is serving as the Web Chair for the Second International Workshop on Regression Testing (Regression 2012).