Are we ready to embrace generative AI for software Q&A?

• Main Authors: XU, Bowen, NGUYEN, Thanh-Dat, LE-CONG, Thanh, HOANG, Thong, LIU, Jiakun, KIM, Kisub, GONG, Chen, NIU, Changan, WANG, Chenyu, David LO, LO, David
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: ChatGPT; Generative AI; large language model; Software Q&A; Stack Overflow; Artificial Intelligence and Robotics; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/8489; https://ink.library.smu.edu.sg/context/sis_research/article/9492/viewcontent/EmbraceGAI_Software_av.pdf
• Institution: Singapore Management University

Stack Overflow, the world's largest software Q&A (SQA) website, is facing a significant traffic drop due to the emergence of generative AI techniques. ChatGPT is banned by Stack Overflow after only 6 days from its release. The main reason provided by the official Stack Overflow is that the answers generated by ChatGPT are of low quality. To verify this, we conduct a comparative evaluation of human-written and ChatGPT-generated answers. Our methodology employs both automatic comparison and a manual study. Our results suggest that human-written and ChatGPT-generated answers are semantically similar, however, human-written answers outperform ChatGPT-generated ones consistently across multiple aspects, specifically by 10% on the overall score. We release the data, analysis scripts, and detailed results at https://github.com/maxxbw54/GAI4SQA.