The easy access students have to generative artificial intelligence (AI) through large language models (LLMs), such as ChatGPT or Microsoft Copilot, to research and write their schoolwork worries many parents and educators — and probably many students as well. They're concerned that the traditional skills involved in learning, like taking notes, are being undermined by the passive copying and pasting of results from AI-infused internet searches.

Learning requires more effort than that, as new research shows. But LLMs can play a role. A study of over 400 students age 14 to 15 found that note-taking remains critical for students' reading comprehension and retention. It also found that AI-based LLMs can be useful tools, helping students pursue information to clarify and explore the material they are learning and put it into context.

Cambridge University Press & Assessment and Microsoft Research teamed up for one of the first classroom experiments to investigate how LLMs affect students' reading comprehension and retention.

The test came, when without advance warning, the students were questioned about the two excerpts they'd been asked to read. The questions were designed to see how well they understood and remembered the information.

Students participating in the study read segments on apartheid in South Africa and the Cuban missile crisis. Then they were randomized into one of two groups. One group was asked to study one text with an LLM (ChatGPT 3.5 turbo) and the other text by writing notes.

The second group was also asked to study one of the texts with the LLM, but, for the other text, they were instructed to combine using the LLM and making notes.

In all LLM conditions, students were given a brief tutorial and allowed to use the tool however they liked.

The test came three days later when, without advance warning, the students were questioned about the two excerpts they'd been asked to read. The questions were designed to see how well they understood and remembered the information. For example, they were asked, “What horrific event happened at the Soweto Youth Uprising in 1976?” and “Explain the role of the Soviet Union in the Cuban Missile Crisis.”

Students were asked how they felt about the work, including whether they enjoyed it or found it interesting after both the study and test sessions.

The results suggest that taking handwritten notes or taking notes in combination with using an LLM improve recall and understanding of new information better than using the LLM alone.

There was one area where LLMs excelled: students enjoyed using the programs to explore relevant topics beyond the text.

“Our study shows that students enjoyed using AI chatbots but note taking was more effective for learning outcomes,” the study's first author, Pia Kreijkes, a senior researcher at Cambridge University Press & Assessment, said. “Our findings can help guide the use of LLMs for learning. In particular, they indicate that students should take notes separately from using LLMs to avoid copying the LLM.”

Taking notes or taking notes in combination with using an LLM are better for recall and understanding than using the LLM alone.

There's also a risk when using AI-generated information of reproducing bot-associated “hallucinations” and misinformation. “[S]tudents should receive training and guidance on how to use LLMs to support active and constructive learning,” she added.

Teachers could also benefit from students' use of LLMs. Educators may be able to identify areas where teaching support is needed from students' interactions with the LLMs and adjust their teaching accordingly.

“I was struck by how many students used the LLM to try to deepen their understanding — asking about historical context, clarifying unfamiliar references, and exploring the significance of key events,” Jake Hofman, Senior Principal Researcher from Microsoft Research, said. “Rather than viewing traditional learning techniques, like note-taking, and newer generative-AI approaches as competing alternatives, we should view them as complementary.”

The study is published in Computers & Education.