MSR 2024
Mon 15 - Tue 16 April 2024 Lisbon, Portugal
co-located with ICSE 2024

Continuous Integration (CI) is a popular software development practice that allows developers to quickly verify modifications to their projects. To cope with the ever-increasing demand for faster software releases, CI acceleration approaches have been proposed to expedite the feedback that CI provides. However, adoption of CI acceleration is not without risks. For example, CI acceleration techniques may mislabel change sets (e.g., a build labeled as failing that passes in an unaccelerated setting or vice versa) or produce results that are inconsistent with an unaccelerated build (e.g., the underlying reasons for failure differ accelerated and unaccelerated builds). These inconsistencies call into question the trustworthiness of CI acceleration products.

In this paper, we propose an approach inspired by mutation testing to systematically evaluate the trustworthiness of CI acceleration. We apply our approach to a deterministic solution that uses program analysis to accelerate CI and uncover issues that hinder its trustworthiness. Our analysis consists of three parts. We first study how often the same build in accelerated and unaccelerated CI settings produce different mutation testing outcomes. We call mutants with different outcomes in the two settings "gap mutants’’. Next, we study the code locations where gap mutants appear. Finally, we inspect gap mutants to understand why acceleration causes them to survive. Our analysis of a commercial-grade CI acceleration approach and ten thriving open-source projects uncovers 2,237 gap mutants. We find that: (1) the gap in mutation outcomes between accelerated and unaccelerated settings varies from 0.11%–23.50% of the studied mutants; (2) 88.95% of gap mutants can be mapped to specific source code functions and classes using the dependency representation of the studied CI acceleration product; and (3) 69% of gap mutants survive CI acceleration due to deterministic reasons that can be classified into six fault patterns. Our results show that even deterministic CI acceleration solutions suffer from trustworthiness limitations, and highlight the ways in which trustworthiness could be improved in a pragmatic manner.

Tue 16 Apr

Displayed time zone: Lisbon change

11:00 - 12:30
Process automation & DevOps and Tutorial ITechnical Papers / Tutorials at Almada Negreiros
Chair(s): Tom Mens University of Mons, Ayushi Rastogi University of Groningen, The Netherlands
Learning to Predict and Improve Build Successes in Package Ecosystems
Technical Papers
Harshitha Menon Lawrence Livermore National Lab, Daniel Nichols University of Maryland, College Park, Abhinav Bhatele University of Maryland, College Park, Todd Gamblin Lawrence Livermore National Laboratory
The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI Builds
Technical Papers
Ajiromola Kola-Olawuyi University of Waterloo, Nimmi Rashinika Weeraddana University of Waterloo, Mei Nagappan University of Waterloo
A Mutation-Guided Assessment of Acceleration Approaches for Continuous Integration: An Empirical Study of YourBase
Technical Papers
Zhili Zeng University of Waterloo, Tao Xiao Nara Institute of Science and Technology, Maxime Lamothe Polytechnique Montreal, Hideaki Hata Shinshu University, Shane McIntosh University of Waterloo
Cohort Studies for Mining Software Repositories
Nyyti Saarimäki Tampere University, Sira Vegas Universidad Politecnica de Madrid, Valentina Lenarduzzi University of Oulu, Davide Taibi University of Oulu and Tampere University , Mikel Robredo University of Oulu