Cox, Hunt, Hill conduct first research on AI-generated questions and human-generated items in nursing

There are few limits when it comes to Artificial Intelligence utilization; be it in smartphones, chatbots, social media websites, security and navigations systems, web searches, translating software, cyber security, medicine, and writing, AI is permeating most segments of society. 

Add nursing education to AI’s next frontier, specifically practice questions for nursing’s National Council Licensure Examination (NCLEX). 

Three MGH Institute nursing professors recently won a Journal of Nursing Education award for being the first to delve into the topic. Rachel Cox Simms, DNP, FNP-BC, Karen Hunt, PhD, RN, RD, CNE, and Rebecca Hill, PhD, DNP, CNE, FAAN, were named winners of the 2024 Christine A. Tanner Scholarly Writing Award for their article, “Comparative Analysis of NCLEX-RN Questions: A Duel Between ChatGPT and Human Expertise.”

The award recognizes a piece of scholarly writing that has the potential to widely impact nursing education, and the work by the trio fit the bill, as they looked at how well NCLEX practice questions crafted by AI would compare to NCLEX practice questions crafted by actual nurses. 

It was the first research comparing AI-generated questions and human-generated items in nursing.

Practice questions are often created by nursing faculty who want to prepare their students for the NCLEX examination and future nursing practice. But producing high-quality NCLEX-style questions is a laborious and time-consuming process. Enter AI. 

“Our research was a study on the use of ChatGPT to create NCLEX-style questions, which is a form of generative artificial intelligence,” said Cox Simms, an expert in AI. 

“We were wondering, could this technology help nurse educators in that endeavor?” said Hunt. 

The answer is an unequivocal yes.

“We used a comparative analysis where we took some questions that had been tried and true on some of our exams, historically, across a few different courses that were developed by nurse educators. We looked for questions that were testing what we were looking for, and that the students who got the questions correctly did so because they knew the material.”

“We used ChatGPT to create NCLEX style questions and then compared the questions created by generative AI to those compared created by humans,” added Cox Simms. 

The NCLEX questions, created in mere seconds, were sent alongside questions created by nurses to faculty across the 58 accredited nursing programs in New England. 

According to the study, “Participants were unaware of the origin of the questions (i.e., ChatGPT or human). The evaluation centered around five crucial dimensions: (1) grammar; (2) clarity; (3) clinical relevance; (4) terminology; and (5) difficulty.”

“Our results were so interesting because AI really did produce some very high-quality questions in literally seconds that in some cases were statistically superior, rated as superior by nurse educators, and preferred by nurse educators,” marveled Hunt. 

The award committee saw considerable value in this contribution, writing: “The paper by Cox et al is clear, informative, relevant, and has significant potential to contribute to changes in nursing education —whether or not we are ready for them. The perspective taken by these authors might help us to actually get ready. Their delivery of information was clear and concise.”

“I was surprised by how quickly it just produced these questions that didn’t need to be tweaked,” said Hill. “That’s not how it will always be, but in a world where it can take an incredible amount of time to come up with a well-written NCLEX question, this is an amazing tool to utilize.” 

The study underscores the increasing ways in which AI can be used in the academic setting. 

“What's really exciting about this is that it demonstrates the power of generative AI to be a tool that nurse educators can use to create content that is very useful for their students,” said Cox Simms. “It’s going to save time and be more productive for them, and that’s just really exciting.”

Noted Hunt: “We're just discovering the opportunities for AI in academia and in nursing education in particular. The biggest takeaway is that as wonderful of a tool as AI is, we can't take away the need for expertise of our nurse educators. There were some AI questions that we didn't use because we didn't think they were good questions, and so that human involvement is still in the forefront.”

“The most salient piece is we have to be sure that as educators we're embracing AI because it's here, it's only going to get more sophisticated,” warned Hill. “If we don't embrace it, we’ll be light years behind everyone else who does embrace it.” 

Do you have a story the Office of Strategic Communications should know about? If so, let us know.