As I’ve been struggling a bit with a section of my book proposal, namely the “comparable literature” I thought why not make use of ChatGPT to help me do some research. No, in case you’re wondering, I will not let ChatGPT write my book. I still intend to write every word, formulate every sentence, by myself. No, what I wanted to do was an intelligent search of comparable literature on the subject of global public policy making, to help me narrow down, track down, and read the relevant literature in the subject, then to compare my upcoming book with these, then know how my work will fit into the canon of existing works by other authors.
So, I gave ChatGPT 3.5 (the free version) the following prompt:
“You are writing a book proposal to an academic publisher on the subject of global public policy making in the context of the Internet revolution of the past 25 years. You intend to have your book covering both the academic underpinnings and spice it up with your own real world experiences using anecdotes of what happened in different situations. As part of the book proposal you are supposed to come up with a list of existing literature in the field. The task is to show a list of existing literature in the field.”
I got a list of ten books with promising titles. Then, for each of the titles presented, I asked the following prompt:
“What are the main lines of argument in the book”
This gave me nice summaries of the contents of the books. They all read very well, and helped my understanding of recent literature.
But, true to my academic training, I wlso wanted to find these books in a library or bookstore or on Amazon, to actually read them, and not merely to rely on these summaries which were only intended as initial pointers. So I asked another prompt:
“Please give me full bibliographic data on the book.”
And then I found to my dismay, after checking the provided bibliographical data, that three out of ten references did not really exist. They were invented by ChatGPT. I confronted the AI with that, for example:
“There is no record of a book titled “Global Governance And Public Policy In The 21st Century: The Impact Of New Technologies” by Sarbani Ghosh and David T. Mitchell. Is it possible that it does not exist or that the information provided is incorrect?”
And ChatGPT replied:
“I apologize for the mistake in providing that citation. It seems that the book “Global Governance And Public Policy In The 21st Century: The Impact Of New Technologies” by Sarbani Ghosh and David T. Mitchell does not exist. I will make sure to remove it from the list of available literature. Thank you for bringing this to my attention.”
Which is really a shame. Imagine a failure rate of 30 percent. 3 out of ten answers were incorrect. That means only 70 percent is right. In academia, that would be a dismal ‘C’ grade. This left me pretty gobsmacked. Everyone is shouting out how wonderful the future with AI, artificial intelligence, would be. However, I am not convinced. On the contrary. Disappointed by the failure of this particular AI, ChatGPT 3.5, to come up with a reliable set of answers to a few simple questions.
What are your thoughts? Leave a comment.
