A day doesn’t go by now when we don’t hear about about new AI platforms, models or crazes. In January it was all about Deep Seek, by February ChatGPT’s had released ‘Deep Research” (now also available to Plus subscribers). In March RLUK’s own conference included sessions on AI. 

In April social media was awash with images of AI generated dolls and AI Action Figures. These are all accompanied by hoopla from AI advocates and concern from AI sceptics who [quite rightly] worry about the ethical use of AI, copyright and its environmental impact. 

Is there some middle ground though as we reflect on what these models mean for research libraries and how we might effectively use them in our daily and ongoing work. Spoiler: Yes AI can be useful, but there are no RLUK AI Figures! 

Revisiting the Digital Shift Maturity Survey

Could these new models and platforms help in giving us new insights into work we have already done, and provide some ‘training wheels’ in understanding what can (and can’t be done)? RLUK’s Digital Shift Maturity survey seemed ideally suited for this exercise, and the initial analysis identified AI as one of the least mature of the Digital Shift principles.

The Digital Shift Maturity Survey was run in the summer and autumn of 2023 to gauge progress in digital transformation across its member libraries. The survey was pivotal in helping RLUK members reflect upon and benchmark their positions on the digital shift spectrum and 50% of RLUK member institutions completed it. The initial analysis produced a substantial volume of qualitative data with 43,000 words and 100 plus pages of free text responses.

In January and February 2025 RLUK revisited the survey responses using ChatGPT’s 4o model to learn more about using the models (and their limitations), to validate initial themes, identify gaps and take a fresh look at the data. This work was also done in the spirit of the survey which identified that “Engaging with AI as a Transformative Technology” as an area which was less mature.

The summary report is available on Zenodo at https://10.5281/zenodo.15854157 

Why ChatGPT (other LLMs are available)?

The choice of ChatGPT was pragmatic, it was one we had some initial ‘free’ experience with but for this analysis we used the GPT Plus subscription version which provides unrestricted access to the GPT-4o model as well as more control over the data. ChatGPT Plus users can opt-out of having their conversations used to train OpenAI’s models and can also upload files which became very important . This model is recognised for its versatility and capacity to handle structured document analyses, which proved invaluable for managing the rich, anonymised qualitative data available. 

Other models such as Claude by Anthropic, Copilot and Google’s Gemini could all potentially have been used but we don’t have subscriptions to these, or institutional licences so ChatGPT was a pragmatic choice, and one which we have acknowledged in the interests of transparency and ethics.

Methodology: Using ChatGPT Plus

ChatGPT Plus in addition to removing various limits of Model use, also enabled access to Projects. In ChatGPT projects “provide a new way to group files and chats for personal use, simplifying the management of work that involves multiple chats. Projects keep chats, files, and custom instructions in one place. Use them for ongoing work” 

DSM Survey Project

In the first instance I asked ChatGPT for some guidance on using it for qualitative survey analysis and had it adopt the persona of an “expert survey analyst”, this was reinforced in the individual question prompts, which were kept consistent. The instructions also gave ChatGPT guidance on using British spelling and idioms (by default it is American), the goal of the survey and wider context on its use. 

DSM Survey Instructions

Structuring the data: Thematic Scaffolding

SmartSurvey, our survey tool can export data in Word and Excel, and following ChatGPT’s recommendations our survey data was uploaded on a question by question basis, structured in Excel field with anonymised identifiers. ChatGPT had to include the identifiers to justify the quotes and themes and to “show its working”. More in-depth qualitative analysis could be applied to code themes but broad themes were felt sufficient as a comparator. This was the approach taken with UN Policy Documents in the working paper on “Using ChatGPT for thematic analysis” from the Bennett Institution in Public Policy, Cambridge.

The simple but structured and more transparent approach, coupled with clear instructions on preserving confidentiality and including respondent IDs, allowed ChatGPT to provide valuable thematic scaffolding (albeit with some repetition at times). The theming worked best when there was a rich pool of responses, for instance in “Skills, Scholarship, Stakeholders, Spaces”, and less so in the DSM principles which had much fewer responses but still clustered responses by theme.

Example DSM Prompt in ChatGPT

Once the prompt for the question was refined (through various iterations), it was applied consistently to the questions and their underlying Excel files. Once run, ChatGPT replied with a response reflecting the instructions.

ChatGPT Response

Value Add(s)

Beyond the analysis on a question by question basis, there was scope in the main thematic areas to ask ChatGPT to identify gaps in Skills etc which were not in the datasets it was working from. It generated a number of these, and these were also consistent with gaps like cybersecurity which the Digital Shift Working Group identified last June (2024).

DSM Survey Additional Gaps

The opportunity to identify potential gaps in survey responses  through tools like ChatGPT provides an interesting value-add feature – albeit one to be mindful of about the suggestions and recommendations it makes, and their provenance) on the work done to prep, upload, review, revise and re-publish our results.

Reflections and Caveats

Prompt engineering is too grandiose a term for the prompts used here, it was more prompt tinkering or iterating. I am sure there are more sophisticated prompts which could be applied, but this exercise was as much about building confidence and capability with AI platforms as it was revisiting the Digital Shift Maturity Survey. Making the time and space to help iterate the prompts, structure the data, develop workflows and learn about the models (and their capabilities and quirks) were all invaluable and could be more readily applied in the future.

There definitely needs to be a human in the loop (or wheel house to stretch an analogy) to continually sense-check the results against original responses and quotes, and akcnowledging ChatGPT’s standard disclaimer “ChatGPT can make mistakes. Check important info.” 

The thematic scaffolding it provided was very valuable, even if it some of it may be too granular, but themes could be merged. The opportunity to have ChatGPT explore gaps and further recommendations was also useful, again as a framework to ‘pick and mix’ from. The range of recommendations and gaps have been included as an appendix in the summary report.

The key lessons from this work are the importance to

• Make time to iterate and develop the prompt(s) and have an initial testing cycle which can be used to refine the prompt and in turn the response it generates, and then be consistent; and do the same with format of the data and file types
• Ask the AI platform itself for advice on to help refine the prompt, and where possible, provide global instructions, this can also include using British English to address spelling and language issues
• Use structured data: themes and responses improved when the data was structured, and improved again when survey IDs were included to ‘show working’, for instance in two columns in Excel
• Run the prompts on a Question by Question basis for analysis: results improved when the analysis was done question by question, using a consistent prompt set
• Have a reasonable amount of freetext response: more responses led to better overarching, and broader themes
• Keeping the human in the loop, in particular to review quotes, combine themes which may be too granular and provide a sense check

Conclusions and Next Steps

Was it a successful experiment? I would say yes, we have a revised summary report on Zenodo highlighting key themes and gaps which provides a mix of affirmation of the original analysis in addition to some value added insights. 

It was reassuring too that it wasn’t just a magic bullet which could turn survey results into clear themes and recommendations without human intervention and engagement through the entire lifecycle of the survey. My experience resonates with Turobov, Coyle & Harding’s broader paper on thematic analysis that

“a hybrid AI and human analysis model, where the AI tool does the heavy lifting of data coding and initial analysis, allowing researchers to engage more critically with the material and refine findings with their expert judgement”.  [Turobov, Coyle & Harding]

It also helped to build capability and confidence in using AI, exploring different models, and refining the prompts for more complex analysis and activities (with caveats).

The work done here will be used to complement and inform the analysis of future surveys, applying the lessons learned and to consider where AI may assist, but not replace the ‘human in the loop’.

Opening up Wider Community Discussion

This work, and this blog post invites further community discussion on the use and role of Gen AI in our daily work and activities and RLUK’s strategic role in this space. And this discussion isn’t limited to ChatGPT.  In a meeting earlier this year a colleague remarked that “Claude [from anthropic] is [their] most productive team member.” 

How are you using and engaging with AI platforms in your daily practice and work? Are there specific uses you use of them, specific platforms or models which you use?

References and Related Links

• RLUK’s Strategy 2020-25
• RLUK’s Digital Shift Manifesto
• Turobov, Coyle & Harding (2024) Using ChatGPT for Thematic Analysis [Working Paper] https://www.bennettinstitute.cam.ac.uk/wp-content/uploads/2024/05/Using-ChatGPT-for-analytics-WP.pdf