DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks
Jupyter notebooks have become indispensable tools for data analysis and processing in various domains. However, despite their widespread use, there is a notable research gap in understanding and analyzing the contents and code metrics of these notebooks. This gap is primarily attributed to the absence of datasets that encompass both Jupyter notebooks and extracted their code metrics. To address this limitation, we introduce DistilKaggle, a unique dataset specifically curated to facilitate research on code metrics in Jupyter notebooks, utilizing the Kaggle repository as a prime source. Through an extensive study, we identify thirty-four code metrics that significantly impact Jupyter notebook code quality. These features such as \textit{lines of code cell, mean number of words in markdown cells, performance tier of developer, etc.} are crucial for understanding and improving the overall effectiveness of computational notebooks. The DistilKaggle dataset is derived from a vast collection of notebooks, and we present two distinct datasets: (i) \textbf{Code Cells and markdown Cells Dataset} which is presented in two CSV files, allowing for easy integration into researchers’ workflows as dataframes. It provides a granular view of the content structure within 542,051 Jupyter notebooks, enabling detailed analysis of code and markdown cells; and (ii) \textbf{Notebook Code Metrics Dataset} focused on the identified code metrics of notebooks. Researchers can leverage this dataset to access Jupyter notebooks with specific code quality characteristics, surpassing the limitations of filters available on the Kaggle website. Furthermore, the reproducibility of the notebooks in our dataset is ensured through the code cells and markdown cells datasets, offering a reliable foundation for researchers to build upon. Given the substantial size of our datasets, less than 5 GB, it becomes an invaluable resource for the research community, surpassing the capabilities of individual Kaggle users to collect such extensive data. For accessibility and transparency, both the datasets https://doi.org/10.5281/zenodo.10317389 and the code https://github.com/theablemo/DistilKaggle utilized in crafting this dataset are publicly available.
Tue 16 AprDisplayed time zone: Lisbon change
11:00 - 12:30 | Software Evolution & AnalysisTechnical Papers / Data and Tool Showcase Track / Industry Track at Grande Auditório Chair(s): Vladimir Kovalenko JetBrains Research | ||
11:00 12mTalk | Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study Technical Papers Rosalia Tufano Università della Svizzera Italiana, Antonio Mastropaolo Università della Svizzera italiana, Federica Pepe University of Sannio, Ozren Dabic Software Institute, Università della Svizzera italiana (USI), Switzerland, Massimiliano Di Penta University of Sannio, Italy, Gabriele Bavota Software Institute @ Università della Svizzera Italiana | ||
11:12 12mTalk | DRMiner: A Tool For Identifying And Analyzing Refactorings In Dockerfile Technical Papers Emna Ksontini University of Michigan - Dearborn, Aycha Abid Oakland University, Rania Khalsi University of Michigan - Flint, Marouane Kessentini University of Michigan - Flint | ||
11:24 12mTalk | A Large-Scale Empirical Study of Open Source License Usage: Practices and Challenges Technical Papers Jiaqi Wu Zhejiang University, Lingfeng Bao Zhejiang University, Xiaohu Yang Zhejiang University, Xin Xia Huawei Technologies, Xing Hu Zhejiang University | ||
11:36 12mTalk | Analyzing the Evolution and Maintenance of ML Models on Hugging Face Technical Papers Joel Castaño Fernández Universitat Politècnica de Catalunya, Silverio Martínez-Fernández UPC-BarcelonaTech, Xavier Franch Universitat Politècnica de Catalunya, Justus Bogner Vrije Universiteit Amsterdam Link to publication Pre-print | ||
11:48 12mTalk | On the Anatomy of Real-World R Code for Static Analysis Technical Papers Florian Sihler Ulm University, Lukas Pietzschmann Ulm University, Raphael Straub Ulm University, Matthias Tichy Ulm University, Germany, Andor Diera Ulm University, Abdelhalim Dahou GESIS Leibniz Institute for the Social Sciences Pre-print File Attached | ||
12:00 6mTalk | Encoding Version History Context for Better Code Representation Technical Papers Huy Nguyen The University of Melbourne, Christoph Treude Singapore Management University, Patanamon Thongtanunam University of Melbourne Pre-print | ||
12:06 4mTalk | CodeLL: A Lifelong Learning Dataset to Support the Co-Evolution of Data and Language Models of Code Data and Tool Showcase Track Martin Weyssow DIRO, Université de Montréal, Claudio Di Sipio University of L'Aquila, Davide Di Ruscio University of L'Aquila, Houari Sahraoui DIRO, Université de Montréal | ||
12:10 4mTalk | Bidirectional Paper-Repository Tracing in Software Engineering Data and Tool Showcase Track Daniel Garijo , Miguel Arroyo Universidad Politécnica de Madrid, Esteban González Guardia Universidad Politécnica de Madrid, Christoph Treude Singapore Management University, Nicola Tarocco CERN | ||
12:14 4mTalk | DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks Data and Tool Showcase Track Mojtaba Mostafavi Department of Computer Engineering of Sharif University of Technology, Arash Asgari Department of Computer Engineering of Sharif University of Technology, Mohammad Abolnejadian Department of Computer Engineering of Sharif University of Technology, Abbas Heydarnoori Bowling Green State University | ||
12:18 5mTalk | Estimating Usage of Open Source Projects Industry Track |