Programmers are still needed: ChatGPT failed to understand more than half of 517 questions from Stack Overflow
Researchers from Purdue University in the United States have found that 52% of all ChatGPT answers in programming are incorrect. This was reported by TechRadar.
The researchers analyzed 517 questions from Stack Overflow, asked to the free ChatGPT 3.5 version, and compared the answers with those given by humans.
Experts checked artificial intelligence errors and found that 54% of them occurred because ChatGPT misunderstood the context. 36% of the answers had factual inaccuracies, and 28% had logical errors. Another 12% of answers contained terminological errors.
In some answers, ChatGPT made several mistakes, for example, the answer contained a logical fallacy (28%) due to misunderstanding the context (54%).
The researchers also criticize ChatGPT for overly long and complex answers that contain more details than necessary. This leads to potential confusion and distraction. However, according to a survey of 12 programmers, one-third of them preferred ChatGPT’s clear, textbook-like answers.
“Since ChatGPT produces a large number of incorrect answers, our results emphasize the necessity of caution and awareness regarding the usage of ChatGPT answers in programming tasks,” the authors of the study conclude.