How (in)accurate is generative AI?

A new study, Magesh et al. (2024), finds that Generative AI is mostly wrong.

➤ Highlights

👉 A study from May tested four platforms, focusing on legal tech. 

(Excuse the typo in the pdf, the study is from May, not March)

👉 The researchers sent prompts with legal questions and measured the accuracy of the results, sorting them into false, incomplete, or accurate answers.

👉 They found that the answers are mostly wrong, false or incomplete – see the image attached to this post 

➤ Reflections

💡 Generative AI is mostly wrong

💡Platforms vary on how accurate they are 

💡Make sure you inquire about accuracy when you choose platforms, whether for personal use or when procuring platforms for your business.

💡 Don’t believe vendors who tell you that their chatbots are error-free! 

➤ Comments welcome! What do people think of this finding?