Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools

Citations Over TimeTop 1% of 2024 papers

Abstract

Generating unit tests is a crucial task in software development, demanding substantial time and effort from programmers. The advent of Large Language Models (LLMs) introduces a novel avenue for unit test script generation. This research aims to experimentally investigate the effectiveness of LLMs, specifically exemplified by ChatGPT, for generating unit test scripts for Python programs, and how the generated test cases compare with those generated by an existing unit test generator (Pynguin). For experiments, we consider three types of code units: 1) Procedural scripts, 2) Function-based modular code, and 3) Class-based code. The generated test cases are evaluated based on criteria such as coverage, correctness, and readability. Our results show that ChatGPT's performance is comparable with Pynguin in terms of coverage, though for some cases its performance is superior to Pynguin. We also find that about a third of assertions generated by ChatGPT for some categories were incorrect. Our results also show that there is minimal overlap in missed statements between ChatGPT and Pynguin, thus, suggesting that a combination of both tools may enhance unit test generation performance. Finally, in our experiments, prompt engineering improved ChatGPT's performance, achieving a much higher coverage.*These authors contributed equally.

Related Papers

• → Analysis of Professors’ Experiences with Generative AI and the Concerns of Classroom Use : Application of the Concerns-Based Adoption Model (CBAM)(2023)8 cited
• → Are There Any Unit Tests? An Empirical Study on Unit Testing in Open Source Python Projects(2017)20 cited
• → MIPSUnit(2017)4 cited
• CAE listening and speaking skills(1996)
• The Limitation of Generative Grammar at the Present Period(2000)