Human students were able to outperform ChatGPT in most of the exam questions in this particular field.
ADVERTISEMENTArtificial intelligence (AI) has the world in its thrall, with daily updates about new capabilities inspiring awe and fear in equal measure.
In the wake of the release of ChatGPT - the fastest growing technology platform ever - there has been talk of AI coming for the jobs of writers, artists, musicians, and coders, to name just a few.
But one profession may be safe for the time being. According to a study from Brigham Young University (BYU) in the US, human accountants still outperform ChatGPT at accounting exams.
The study comprised 327 co-authors from 186 different educational institutions across 14 countries, all-in-all contributing 25,181 classroom accounting exam questions.
The questions covered accounting information systems (AIS), auditing, financial accounting, managerial accounting, and tax. They varied in difficulty and type, including true v false, multiple choice, and short answers.
And ChatGPT - having previously passed law exams among others - unsurprisingly performed well. Its overall average was 47.4 per cent, doing better than human students on 11.3 per cent of questions. It did especially well on AIS and auditing.
But the students outperformed the machine overall. Their average scores came to 76.7 per cent, and they were better at questions on tax, financial, and managerial assessments, possibly because ChatGPT struggled with the mathematical processes required for the latter type.
In terms of question type, ChatGPT did better on true v false questions (68.7 per cent correct), and multiple choice (59.5 per cent), but it struggled with short answer questions.
A ‘game changer’ for teaching and learning
The study’s lead author, BYU professor of accounting David Wood, wanted to recruit as many professors as possible to see how AI did against real students, to help inform the debate about how AI language models should factor into education.
“When this technology first came out, everyone was worried that students could now use it to cheat,” said Wood.
“But opportunities to cheat have always existed. So for us, we’re trying to focus on what we can do with this technology now that we couldn’t do before to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening”.
The researchers say that while it still has work to do in the realm of accounting, it’s a game changer that will change the way everyone teaches and learns for the better.
“It’s an opportunity to reflect on whether we are teaching value-added information or not,” said study coauthor and fellow BYU accounting professor Melissa Larson.
“This is a disruption, and we need to assess where we go from here. Of course, I’m still going to have teaching assistants, but this is going to force us to use them in different ways”.
The study also highlighted some other interesting findings. For one, ChatGPT doesn’t always recognise when it is doing maths, and makes nonsensical errors such as adding two numbers in a subtraction problem, or dividing numbers incorrectly. It also provides explanations for its answers, even if they are incorrect.
ChatGPT is built on OpenAI’s GPT3.5 deep learning language model. OpenAI has since released GPT4, which surpasses GPT3.5 in capabilities. The researchers expect the technology to improve exponentially on the accounting questions posed in their study.
What they find most promising is how the chatbot can help improve teaching and learning, including the ability to design and test assignments, or perhaps be used for drafting portions of a project.
“It’s not perfect; you’re not going to be using it for everything,” said Jessica Wood, a student at BYU. “Trying to learn solely by using ChatGPT is a fool’s errand”.
