Should researchers really be worried about ChatGPT?

The Internet is abuzz with discussions about ChatGPT (Generative Pretrained Transformer), with many pundits expressing concern about the potential disruption this and other artificial intelligence (AI) tools might cause to academia.

The first example I came across showed up in my Twitter feed. The tweet extolled the virtues of the AI tool in drafting a university’s vision statement. To be honest, I found the text it had churned out rather bland, uninspiring and non-committal, the kind of statement that could apply to just about any institution.

And so I was rather skeptical about its potential to make student assignments redundant, despite stories that ChatGPT had successfully passed medical and legal exams. Therefore, I decided to put it to the test. It's surprisingly easy to interact with the tool: type your question or paste your text into the box provided on the ChatGPT website and you'll see your answer appear in real time! (You will need to register first, but that's a quick process.)

An aid to scientific writing?

I recently completed a biography of an historical personality, and so I decided to quiz ChatGPT about him. I specifically asked the tool to compose a 500 word biography. The result was less than impressive. Not only was the biography well short of the brief (307 words), contents-wise it only got part of the history correct.

When I challenged the AI response by pointing out both of these shortcomings, it nicely apologised (admittedly a cool feature) and churned out another 287 words that were more or less correct and added to its previous attempt. This led me to question the AI tool about its potential use as a scientific writing aid:

ChatGPT is a powerful language model that can generate human-like text, but it is important to understand its limitations when using it for scientific writing. While ChatGPT can generate text that is grammatically correct and coherent, it may not always produce scientifically accurate or factually correct information. Additionally, it is not able to independently conduct research or provide original insights, so it should not be used as a substitute for the scientific method or your own research. Therefore, if you are planning to use the text generated by ChatGPT for scientific publication, it is important to thoroughly fact-check and verify all information before using it. Additionally, it is important to cite any sources used by the model to avoid plagiarism. In summary, ChatGPT can be a useful tool for generating text that can be used as a starting point for scientific writing, but it should not be relied upon as the sole source of information and should always be verified and fact-checked before use.

ChatGPT is a powerful language model that can generate human-like text, but it is important to understand its limitations when using it for scientific writing.

This response invited additional questions, such as whether ChatGPT could be asked to write text that actually included references:

Yes, I can include references in my responses if I have the information and it is appropriate for the question or topic being discussed. However, it’s important to note that the references I provide may not be from peer-reviewed sources and should be independently verified.

Limitations

When I probed deeper, trying to understand the tool’s limitations, it explained,

ChatGPT is a powerful language model that can generate human-like text, but it has several limitations when it comes to scientific writing.

• Lack of domain-specific knowledge: ChatGPT is trained on a wide range of text, but it may not have the same level of expertise in a specific scientific field as a human expert.
• Limited ability to follow a specific writing style or format: ChatGPT can generate text in different styles, but it may not be able to perfectly mimic a specific writing style or format that is required for scientific writing.
• Limited understanding of scientific jargon: ChatGPT may not understand scientific terms and jargon as well as a human expert in the field.
• Limited ability to critically evaluate sources: ChatGPT can generate text based on a provided source, but it does not have the ability to critically evaluate the credibility or relevance of that source for a specific scientific study.
• Limited ability to understand ethical issues: ChatGPT can generate text on ethical issues, but it does not have the ability to understand the nuances and complexity of these issues as a human expert would.

These are some of the limitations of ChatGPT for scientific writing. It is important to keep in mind that ChatGPT is a tool that can assist with writing, but it should not be used as a replacement for human expertise and critical thinking.

Indeed, and more to the point,

While ChatGPT can assist with language-related tasks, such as grammar and vocabulary, it is not capable of understanding the nuances and complexities of scientific research. It is important ... to develop [your] own writing skills and to understand the key concepts and findings of [your] research. Additionally, using ChatGPT or other AI-based writing tools to generate content for a scientific paper is considered as Plagiarism. Scientific papers are expected to be original works, and the use of AI-generated text could be considered a violation of academic integrity.

In the case of students whose first language is not English, it is advisable to provide them with additional support and resources such as language tutors or editing services to help them improve their writing skills in English and to effectively communicate their findings to the scientific community.

Ethics

Interestingly, this response raised two ethics issues, plagiarism and whether one should rely on external language support. As regards the former, ChatGPT responded that it

… is designed to avoid plagiarism in its responses. It is trained on a diverse range of text and is able to generate unique responses based on the input it receives. However, it is possible that it may generate similar responses to text that it has seen during its training if the input is very similar. Additionally, ChatGPT is a machine learning model, and like all machine learning models, it can make errors. 

… while it helpfully added that language support could be provided, although with a number of provisos:

ChatGPT can assist non-English students with language-related tasks, such as grammar and vocabulary. However, it should not be used as a replacement for professional language instruction or practice. ChatGPT can help non-English students by providing suggestions for alternative word choices, grammar corrections and also to provide them with a better understanding of sentence structure and idiomatic expressions. However, it is important to note that language proficiency is a complex skill that requires regular practice and feedback. ChatGPT can be a helpful tool for language learning, but it should be used in conjunction with other resources and instruction.

Student assessments

So, all of those responses sounded encouraging. Yet, what about illegitimate use of the AI tool by students when completing their assignments? 

There are several ways to detect if text has been generated by ChatGPT or another language model. Some of these methods include:

• Style and tone: Generated text often lacks the unique writing style and tone of a human author. This can make it easy to detect that the text was not written by a human.
• Coherence: Generated text may be grammatically correct, but it may lack coherence and may not make sense in context.
• Factual accuracy: Generated text may contain factual inaccuracies or inconsistencies, which can be a sign that the text was generated by a machine.
• Repetition: Generated text may repeat the same phrases or ideas multiple times, which can be a sign that the text was generated by a machine.
• Unnatural or robotic language: Generated text may have an unnatural or robotic quality to it, lacking the natural variations and nuances that are present in human-written text.
• Plagiarism: Generated text may be plagiarized from other sources. You can use plagiarism-detection tools to check if the text matches any existing texts on the internet

It’s important to note that these methods are not foolproof, and it may be difficult to definitively say whether text was generated by a machine or not. However, by using a combination of these methods, it is possible to identify text that was likely generated by a machine with a high degree of accuracy.

It appears that not all is lost. Human intelligence is still required to make progress in science, although ChatGPT and similar tools could potentially aid in the discovery process.

My verdict

That brings me back to my own trials of its performance. I tested whether it could detect its own text (affirmative) but when I fed it a published scientific abstract, it recognised the abstract as too technical. In fact, when I asked it to rewrite that abstract more concisely, the result was a non-technical and rather bland paragraph that lacked the kind of detail one would expect to see in a scientific paper.

It appears that not all is lost. Human intelligence is still required to make progress in science, although ChatGPT and similar tools could potentially aid in the discovery process.

But don't let that deter you from trying it out. I found that the AI tool is quite good at converting my research to poetry or rephrasing a reference letter in the style of a Donald Trump rally (not that I would ever do that to any of my students, of course...).

Richard de Grijs is an academic and journal editor with more than 25 years of experience in the physical sciences. In March 2018, he joined Macquarie University (Sydney, Australia) as Associate Dean (Global Engagement). Richard is an AuthorAID Steward and INASP Associate. He presently serves as Managing Editor of the News section of the AuthorAID website.