A team from the University of Missouri (USA) has developed a way to assess whether artificial intelligence (AI) actually finds meaningful ideas in people's words or simply sorts them into categories. The work was published in the International Journal of Qualitative Methods.

Researchers often analyze interviews, focus groups and open-ended survey responses to understand people's experiences and views. Identifying meaningful themes takes time and the ability to grasp context. To check whether AI handles this task without losing important details, specialists in journalism, data and nursing created a method for evaluating computer-generated themes - AURA (audit of correspondence and usefulness). It compares the algorithm's themes with themes identified by people and checks how useful they are for understanding the information.

The test used 32 interviews with instructors, practicing nurses and nursing students about their experience using generative AI (systems that create text on request). First the researchers identified the key themes themselves, then compared them with the computer model's themes.

The model found many of the same broad themes as people, but often lost important differences in meaning and context. A match did not mean the algorithm grasped the full meaning of a theme or organized the information in the best way. Sometimes the model merged several different themes into one broad one or, conversely, split one theme into several smaller ones.

"When people speak in interviews, they usually do not stick to one topic from beginning to end," explained John Robert Bautista, an assistant professor at the University of Missouri School of Nursing. "AI can help organize such conversations so that researchers can more easily see what people are talking about."

Although AI helped organize large volumes of information and find patterns, the team found little evidence that it consistently generates meaningful new ideas beyond those identified by human analysts.

"Our results show that the technology can help researchers manage large collections of interviews, focus groups and survey responses," said Temitope Eyinla, a doctoral student at the University of Missouri Institute for Data and Informatics. "But human expertise remains central to interpreting the results."

The authors note that the method could be useful in studying patient experiences, educational difficulties and problems at work, where understanding people's point of view is important. However, the work analyzed a relatively small number of interviews, so further checks are needed to understand how well AURA works across different subject areas and types of research.

Source: Tech Xplore