MSR 2024
Mon 15 - Tue 16 April 2024 Lisbon, Portugal
co-located with ICSE 2024
Mon 15 Apr 2024 14:10 - 14:15 at Almada Negreiros - Mining Challenge Chair(s): Preetha Chatterjee, Fabio Palomba

The release of large language models (LLMs) like ChatGPT has revolutionized software development. Numerous studies are exploring the generated response quality of ChatGPT, the effectiveness of different prompting techniques, and its performance in programming contests, among other aspects. However, there is limited information regarding the practical usage of ChatGPT by software developers. This data mining challenge focuses on DevGPT, a curated dataset of developer-ChatGPT conversations encompassing prompts with ChatGPT’s responses, including code snippets. Our paper leverages this dataset to investigate (RQ1) whether ChatGPT generates Python & Java code with quality issues; (RQ2) whether ChatGPT-generated code is merged into a repository, and, if it does, to what extent developers change them; and (RQ3) what are the main use cases for ChatGPT besides code generation. We found that ChatGPT-generated code suffers from using undefined/unused variables and improper documentation. They are also suffering from improper resources and exception management-related security issues. Our results show that ChatGPT-generated codes are hardly merged, and they are significantly modified before merging. Instead, based on an analysis of developers’ discussions and the developer-ChatGPT chats, we found that developers use this model for every stage of software development and leverage it to learn about new frameworks and development kits.

Presentation Slides (MSR-Challenge'24.pptx)4.63MiB

Mon 15 Apr

Displayed time zone: Lisbon change

14:00 - 15:30
Mining ChallengeMining Challenge at Almada Negreiros
Chair(s): Preetha Chatterjee Drexel University, USA, Fabio Palomba University of Salerno
14:00
5m
Talk
ChatGPT Chats Decoded: Uncovering Prompt Patterns for Superior Solutions in Software Development Lifecycle
Mining Challenge
Liangxuan Wu Huazhong University of Science and Technology, Yanjie Zhao Huazhong University of Science and Technology, Xinyi Hou Huazhong University of Science and Technology, Tianming Liu Monash Univerisity, Haoyu Wang Huazhong University of Science and Technology
14:05
5m
Talk
Write me this Code: An Analysis of ChatGPT Quality for Producing Source Code
Mining Challenge
Konstantinos Moratis Electrical and Computer Engineering Dept., Aristotle University of Thessaloniki, Themistoklis Diamantopoulos Electrical and Computer Engineering Dept, Aristotle University of Thessaloniki, Dimitrios-Nikitas Nastos Electrical and Computer Engineering Dept., Aristotle University of Thessaloniki, Andreas Symeonidis Aristotle University of Thessaloniki
Pre-print
14:10
5m
Talk
Quality Assessment of ChatGPT Generated Code and their Use by Developers
Mining Challenge
Mohammed Latif Siddiq University of Notre Dame, Lindsay Roney University of Notre Dame, Jiahao Zhang , Joanna C. S. Santos University of Notre Dame
Pre-print Media Attached File Attached
14:15
5m
Talk
Analyzing Developer Use of ChatGPT Generated Code in Open Source GitHub Projects
Mining Challenge
Balreet Grewal University of Alberta, Wentao Lu University of Alberta, Sarah Nadi New York University Abu Dhabi, University of Alberta, Cor-Paul Bezemer University of Alberta
Pre-print
14:20
5m
Talk
How I Learned to Stop Worrying and Love ChatGPT
Mining Challenge
Piotr Przymus Nicolaus Copernicus University in Toruń, Poland, Mikołaj Fejzer Nicolaus Copernicus University in Toruń, Jakub Narębski Nicolaus Copernicus University in Toruń, Krzysztof Stencel University of Warsaw
Pre-print
14:25
5m
Talk
Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation.
Mining Challenge
Kailun Jin York University, Chung-Yu Wang York University, Hung Viet Pham York University, Hadi Hemmati York University
Pre-print
14:30
5m
Talk
The role of library versions in Developer-ChatGPT conversations
Mining Challenge
Rachna Raj Concordia University, Diego Costa Concordia University, Canada
Pre-print
14:35
5m
Talk
AI Writes, We Analyze: The ChatGPT Python Code Saga
Mining Challenge
Md Fazle Rabbi Idaho State University, Arifa Islam Champa Idaho State University, Minhaz F. Zibran Idaho State University, Md Rakibul Islam Lamar University
DOI Pre-print
14:40
5m
Talk
ChatGPT in Action: Analyzing Its Use in Software Development
Mining Challenge
Arifa Islam Champa Idaho State University, Md Fazle Rabbi Idaho State University, Costain Nachuma Idaho State University, Minhaz F. Zibran Idaho State University
DOI Pre-print
14:45
5m
Talk
Chatting with AI: Deciphering Developer Conversations with ChatGPT
Mining Challenge
Suad Mohamed Belmont University, Abdullah Parvin Belmont University, Esteban Parra Belmont University
14:50
5m
Talk
Does Generative AI Generate Smells Related to Container Orchestration?: An Exploratory Study with Kubernetes Manifests
Mining Challenge
Yue Zhang Auburn University, Rachel Meredith Auburn University, Wilson Reaves Auburn University, Julia Coriolano Federal University of Pernambuco, Muhammad Ali Babar School of Computer Science, The University of Adelaide, Akond Rahman Auburn University
Pre-print
14:55
5m
Talk
On the Taxonomy of Developers' Discussion Topics with ChatGPT
Mining Challenge
Ertugrul Sagdic Lamar University, Arda Bayram Lamar University, Md Rakibul Islam Lamar University
15:00
5m
Talk
How to refactor this code? An exploratory study on developer-ChatGPT refactoring conversations
Mining Challenge
Eman Abdullah AlOmar Stevens Institute of Technology, AnushKrishna Venkatakrishnan Rochester Institute of Technology, USA, Mohamed Wiem Mkaouer University of Michigan - Flint, Christian Newman , Ali Ouni ETS Montreal, University of Quebec
15:05
5m
Talk
Analyzing Developer-ChatGPT Conversations for Software Refactoring: An Exploratory Study
Mining Challenge
Omkar Sandip Chavan Rochester Institute of Technology, Divya Dilip Hinge Rochester Institute of Technology, Soham Sanjay Deo Rochester Institute of Technology, Yaxuan (Olivia) Wang Rochester Institute of Technology, Mohamed Wiem Mkaouer University of Michigan - Flint
15:10
5m
Talk
How Do Software Developers Use ChatGPT? An Exploratory Study on GitHub Pull Requests
Mining Challenge
Moataz Chouchen ETS, Narjes Bessghaier ETS Montreal, University of Quebec, Mahi Begoug , Ali Ouni ETS Montreal, University of Quebec, Eman Abdullah AlOmar Stevens Institute of Technology, Mohamed Wiem Mkaouer University of Michigan - Flint
15:15
5m
Talk
Investigating the Utility of ChatGPT in the Issue Tracking System: An Exploratory Study
Mining Challenge
Joy Krishan Das University of Saskatchewan, Saikat Mondal University of Saskatchewan, Chanchal K. Roy University of Saskatchewan, Canada
Pre-print
15:20
5m
Talk
Enhancing User Interaction in ChatGPT: Characterizing and Consolidating Multiple Prompts for Issue Resolution
Mining Challenge
Saikat Mondal University of Saskatchewan, Suborno Deb Bappon Department of Computer Science, University of Saskatchewan, Canada, Chanchal K. Roy University of Saskatchewan, Canada
Pre-print