52% of GenAI's answers are wrong – that's where you come in.
Use AI as a copilot but don't let it take the yoke.
A new study from Purdue University1 measures the accuracy, consistency, and comprehensiveness of ChatGPT’s responses. It reveals insights into where the AI rubber meets humanity’s road.
Researchers posed over 500 questions to ChatGPT2 and compared them to top-rated human answers in the Stack Overflow tech community. They confirmed what serious GenAI users know: its answers can be magical, informative, and surprising.3
And wrong. 52% of AI’s answers were “incorrect.”
What does the Purdue study say about the working relationship between humans and AI?
Let’s take a deeper look.
The yin and yang of AI and treating AI as a copilot
Those who use GenAI regularly won’t be surprised to hear that AI is magical, powerful, and wrong — all at once. But the study reveals that AI’s “wrongness” doesn’t stem from technical errors, bias, or hallucination; it’s often due to a lack of understanding of the underlying question.
In this way, interaction with AI is human-like. This is why my mini-course on using GenAI reminds students to treat AI like you’re the captain of a plane. AI is merely a copilot. Rookies. AI isn’t ready to take the yoke.
So when you use AI, remember who’s in charge: you.
Don’t delegate responsibility to AI, ever.
Another very human problem: bullsh*t detection and laziness.
In the Purdue study, AI’s answers were persuasive, comprehensive, and positive in tone.
Great!
Not so fast. Unfortunately, AI’s answers were found to be so persuasive that 39% of participants overlooked errors.
In other words, human-AI interaction has another familiar flaw: bullsh*t.
Are we being duped by AI’s bullshit, or are we just lazy?
A study from MIT suggests that 68% of us are lazy. It found that even when participants were incentivized with a monetary award for quality responses, 68% of participants submitted ChatGPT’s output straight out of the tool without modification. The tasks were complicated and subtle, like writing a return-to-office memo, a press release, or a technical brief. Still, most participants used AI’s answers straight out of the box.
Once again, the wise way to use AI is to think. Yes, use its prodigious powers to create images that convey your ideas, summarize complicated decisions, perform initial research, and conquer the blank page by generating outlines.
But always treat AI as a copilot, a rookie, or an advisor — never use its output without making it your own.
Don’t let AI take the yoke.
AI is a copilot. Its input is helpful but raw.
You are the captain.
This is where the AI rubber meets humanity’s road.
Just don’t hand it the yoke.
Want to Learn the GenAI Growth Mindset? Join Me!
Want to learn how to make AI your copilot in 2024? Not sure where to start?
Join my mini-course, The Generative AI Growth Mindset, to benefit hospice charity this holiday season. This one-hour presentation is based on a series I teach for Modal Learning. 100% of the $49 fee, net of the 12% administration fee for the Maven e-learning platform, will be donated to Kaplan House Hospice in Danvers, Massachusetts.2
Join my next cohort on January 8th to join the GenAI party and help the hospice mission this holiday season at the same time.
A Two-Minute Beauty Break from
One of my favorite writers on Substack is basketball legend Kareem Abdul-Jabbar, and I’m from Boston.
This week, Kareem shared this 2-minute gem. He wrote, “You’re about to witness a phenomenon you will never forget. You’ll also learn a new word: murmuration. Mostly, you’ll see a flock of thousands of starlings creating art in the sky. Of course, they don’t know that they are, which makes it all the more fascinating.”
It’s well worth two minutes.
Who Answers It Better? An In-Depth Analysis of ChatGPT and Stack Overflow Answers to Software Engineering Questions. Kabir, Kou, Udo-Imeh, Zhang. Purdue University
The questions were technical in nature, but all community-sourced (2)
Compared to human-written, top-vote-getting responses on the community site Stack Overflow.
My enormous apologies to Purdue alumni and fans for mis-spelling "Perdue" in my original post. Arghhh!!!!!!