Technical PapersMSR 2024

Accepted Papers

Call for Papers

The International Conference on Mining Software Repositories (MSR) is the premier conference for data science (DS), machine learning (ML), and artificial intelligence (AI) in software engineering. There are vast amounts of data available in software-related repositories, such as source control systems, defect trackers, code review repositories, app stores, archived communications between project personnel, question-and-answer sites, CI build servers, package registries, and run-time telemetry. The MSR conference invites significant research contributions in which software data plays a central role. MSR research track submissions using data from software repositories, either solely or combined with data from other sources, can take many forms, including: studies applying existing DS/ML/AI techniques to better understand the practice of software engineering, software users, and software behavior; empirically-validated applications of existing or novel DS/ML/AI-based techniques to improve software development and support the maintenance of software systems; and cross-cutting concerns around the engineering of DS/ML/AI-enabled software systems.

The 21st International Conference on Mining Software Repositories will be held on April 15-16, 2024, in Lisbon, Portugal.

Evaluation Criteria

We invite both full (maximum ten pages, plus two additional pages of references) as well as short (four pages, plus references) papers to the Research Track. Full papers are expected to describe new techniques and/or novel research results, to have a high degree of technical rigor, and to be evaluated scientifically. Short papers are expected to discuss controversial issues in the field, or present interesting or thought-provoking ideas that are not yet fully developed. Submissions will be evaluated according to the following criteria:

• Soundness: This aspect pertains to how well the paper’s contributions — whether they involve new methodologies, applications of existing techniques to unfamiliar problems, empirical studies, or other research methods — address the research questions posed and are backed by thorough application of relevant research procedures. For short papers, the expectation is for more limited evaluations given their narrower scope.
• Relevance: The extent to which the paper successfully argues or illustrates that its contributions help bridge a significant knowledge gap or tackle a crucial practical issue within the field of software engineering.
• Novelty: How original the paper’s contributions are in comparison to existing knowledge or how significantly they contribute to the current body of knowledge. Note that this doesn’t discourage well-motivated replication studies.
• Presentation: How well-structured and clear the paper’s argumentation is, how clearly the contributions are articulated, the legibility of figures and tables, and the adequacy of English language usage. All papers should comply with the formatting instructions provided.
• Replicability: The extent to which the paper’s claims can be independently verified through available replication packages and/or sufficient information included in the paper to understand how data was obtained, analyzed, and interpreted, or how a proposed technique works. All submissions are expected to adhere to the Open Science policy below.

Junior PC

Following two successful editions of the MSR Shadow PC in 2021 and 2022 (see also this paper and this presentation for more context), and the success of the Junior PC in MSR 2023, MSR 2024 will once again integrate the junior reviewers into the main technical track program committee!

The main goal remains unchanged: to train the next generation of MSR (and, more broadly, SE) reviewers and program committee members, in response to a widely-recognized challenge of scaling peer review capacity as the research community and volume of submissions grows over time. As with the previous Shadow and Junior PC, the primary audience for the Junior PC is early-career researchers (PhD students, postdocs, new faculty members, and industry practitioners) who are keen to get more involved in the academic peer-review process but have not yet served on a technical research track program committee at big international SE conferences (e.g., ICSE, ESEC/FSE, ASE, MSR, ICSME, SANER).

Prior to the MSR submission deadline, all PC members, including the junior reviewers, will receive guidance on review quality, confidentiality, and ethics standards, how to write good reviews, and how to participate in discussions (see ACM reviewers’ responsibilities). Junior reviewers will then serve alongside regular PC members on the main technical track PC, participating fully in the review process, including author responses and PC discussions to reach consensus. In addition, Junior PC members will receive feedback on how to improve their reviews throughout the process.

All submissions to the MSR research track will be reviewed jointly by both regular and junior PC members, as part of the same process. We expect that each paper will receive three reviews from regular PC members and two additional reviews from Junior PC members. The final decisions will be made by consensus among all reviewers, as always. Based on our experience with the MSR Shadow and Junior PC, we expect that the addition of junior reviewers to each paper will increase the overall quality of reviews the authors receive, since junior reviewers will typically have a deep understanding of recent topics, and can thus provide deep technical feedback on the subject.

Submission Process

All authors should use the official “ACM Primary Article Template”, as can be obtained from the ACM Proceedings Template page. LaTeX users should use the sigconf option, as well as the review (to produce line numbers for easy reference by the reviewers) and anonymous (omitting author names) options. To that end, the following LaTeX code can be placed at the start of the LaTeX document:

\documentclass[sigconf,review,anonymous]{acmart}
\acmConference[MSR 2024]{21st International Conference on Mining Software Repositories}{April 2024}{Lisbon, Portugal}

Submissions to the Research Track can be made via the submission site by the submission deadline. However, we encourage authors to submit at least the paper abstract and author details well in advance of the deadline, to leave enough time to properly enter conflicts of interest for anonymous reviewing. All submissions must adhere to the following requirements:

• All submissions must not exceed 10 pages for the main text, inclusive of all figures, tables, appendices, etc. Two more pages containing only references are permitted. All submissions must be in PDF. Accepted papers will be allowed one extra page for the main text of the camera-ready version. The page limit is strict, and it will not be possible to purchase additional pages at any point in the process (including after acceptance).
• Submissions must strictly conform to the ACM conference proceedings formatting instructions specified above. Alterations of spacing, font size, and other changes that deviate from the instructions may result in desk rejection without further review.
• By submitting to MSR, authors acknowledge that they are aware of and agree to be bound by the ACM Policy and Procedures on Plagiarism and the IEEE Plagiarism FAQ. Papers submitted to MSR 2024 must not have been published elsewhere and must not be under review or submitted for review elsewhere whilst under consideration for MSR 2024. Contravention of this concurrent submission policy will be deemed a serious breach of scientific ethics, and appropriate action will be taken in all such cases. To check for double submission and plagiarism issues, the chairs reserve the right to (1) share the list of submissions with the PC Chairs of other conferences with overlapping review periods and (2) use external plagiarism detection software, under contract to the ACM or IEEE, to detect violations of these policies.
• By submitting your article to an ACM Publication, you are hereby acknowledging that you and your co-authors are subject to all ACM Publications Policies, including ACM’s new Publications Policy on Research Involving Human Participants and Subjects. Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.
• Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process for your accepted paper. ACM has been involved in ORCID from the start and ICSE has recently made a commitment to collect ORCID IDs from all of the published authors. We are committed to improve author discoverability, ensure proper attribution and contribute to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.
• The MSR 2024 Technical Track will employ a double-anonymous review process. Thus, no submission may reveal its authors’ identities. The authors must make every effort to honor the double-anonymous review process. In particular:
  • Authors’ names must be omitted from the submission.
  • All references to the author’s prior work should be in the third person.
  • While authors have the right to upload preprints on ArXiV or similar sites, they must avoid specifying that the manuscript was submitted to MSR 2024.
  • During review, authors should not publicly use the submission title. We recommend using a different paper title for any pre-print in arxiv or similar websites.
• Further advice, guidance, and explanation about the double-anonymous review process can be found in the Q&A page from ICSEs.
• By submitting to MSR 2024, authors acknowledge that they conform to the authorship policy of the ACM, and the authorship policy of the IEEE.

Submissions should also include a supporting statement on the data availability, per the Open Science policy below.

Any submission that does not comply with these requirements is likely to be desk rejected by the PC Chairs without further review.

Authors will have a chance to see the reviews and respond to reviewer comments before any decision about the submission is made.

Upon notification of acceptance, all authors of accepted papers will be asked to fill a copyright form and will receive further instructions for preparing the camera-ready version of their papers. At least one author of each paper is expected to register and present the paper at the MSR 2024 conference. All accepted contributions will be published in the electronic proceedings of the conference.

A selection of the best papers will be invited to an Empirical Software Engineering (EMSE) Special Issue. The authors of accepted papers that show outstanding contributions to the FOSS community will have a chance to self-nominate their paper for the MSR FOSS Impact Award.

Open Science Policy

The MSR conference actively supports the adoption of open science principles. Indeed, we consider replicability as an explicit evaluation criterion. We expect all contributing authors to disclose the (anonymized and curated) data to increase reproducibility, replicability, and/or recoverability of the studies, provided that there are no ethical, legal, technical, economic, or sensible barriers preventing the disclosure. Please provide a supporting statement on the data availability in your submitted papers, including an argument for why (some of) the data cannot be made available, if that is the case.

Specifically, we expect all contributing authors to disclose:

• the source code of relevant software used or proposed in the paper, including that used to retrieve and analyze data.
• the data used in the paper (e.g., evaluation data, anonymized survey data, etc.)
• instructions for other researchers describing how to reproduce or replicate the results.

Fostering artifacts as open data and open source should be done as:

• Archived on preserved digital repositories such as zenodo.org, figshare.com, www.softwareheritage.org, osf.io, or institutional repositories. GitHub, GitLab, and similar services for version control systems do not offer properly archived and preserved data. Personal or institutional websites, consumer cloud storage such as Dropbox, or services such as Academia.edu and Researchgate.net may not provide properly archived and preserved data and may increase the risk of violating anonymity if used at submission time.
• Data should be released under a recognized open data license such as the CC0 dedication or the CC-BY 4.0 license when publishing the data.
• Software should be released under an open source license.
• Different open licenses, if mandated by institutions or regulations, are also permitted.

We encourage authors to make artifacts available upon submission (either privately or publicly) and upon acceptance (publicly).

We recognize that anonymising artifacts such as source code is more difficult than preserving anonymity in a paper. We ask authors to take a best effort approach to not reveal their identities. We will also ask reviewers to avoid trying to identify authors by looking at commit histories and other such information that is not easily anonymized. Authors wanting to share GitHub repositories may also look into using https://anonymous.4open.science/, which is an open source tool that helps you to quickly double-anonymize your repository.

For additional information on creating open artifacts and open access pre- and post-prints, please see this ICSE 2023 page.

Submission Link

Papers must be submitted through HotCRP: https://msr2024-technical.hotcrp.com

Important Dates

• Abstract Deadline: November 14, 2023 AoE
• Paper Deadline: November 17, 2023 AoE
• Author Response Period: December 19 – 22, 2023 AoE
• Author Notification: January 12, 2024 AoE
• Camera Ready Deadline: January 28, 2024 AoE

Accepted Papers and Attendance Expectation

Accepted papers will be permitted an additional page of content to allow authors to incorporate review feedback. Therefore, the page limit for published papers will be 11 pages for full papers (or 5 pages, for short papers), plus 2 pages which may only contain references.

• The official publication date is the date the proceedings are made available in the ACM or IEEE Digital Libraries. This date may be up to two weeks prior to the first day of the ICSE 2024 conference week. The official publication date affects the deadline for any patent filings related to published work.
• Purchases of additional pages in the proceedings are not allowed.

After acceptance, the list of paper authors can not be changed under any circumstances and the list of authors on camera-ready papers must be identical to those on submitted papers. After acceptance paper titles can not be changed except by permission of the Program Co-Chairs, and only then when referees recommended a change for clarity or accuracy with paper content.

If a submission is accepted, at least one author of the paper is required to register for MSR 2024 and present the paper.