Designing a Dementia Caregiver-Centered Recommendation and Context-Aware AI Media Analytical Application: A Comprehensive System Requirements Analysis

Abstract

During technology-assisted reminiscence therapy sessions, caregivers need to quickly assess and select suitable multimedia content to elicit positive memories and emotions in the participating person or persons living with dementia. Especially when relying on online content, caregivers encounter difficulties in identifying dementia-friendly media items due to the large number of search results and the uncertainty about dementia-related selection criteria (e.g., visual and thematic composition). This study seeks to develop an AI-based recommendation and media analysis application for reminiscence therapy by presenting participatory co-design results and outlines caregiver-driven application requirements. Collaborating with n = 14 caregivers from three care facilities, we defined user needs based on our initial definition of the minimum viable components that emerged from previous studies and from the research undertaken as part of this study, including profiling, media search, media storage, media rating, AI-based media analysis, and a recommendation system. Our main findings result in system requirements that correspond to the practical implementation of dementia friendly, digital media-based reminiscence sessions in daily practice. These findings also aid future research and the design of similar applications, benefiting researchers and developers.

Keywords:

• Reminiscence therapy
• requirements analysis
• participatory design
• AI multimedia analysis
• context-aware deep learning
• recommendation system

1. Introduction

Caregivers in residential dementia care constantly strive to maintain the well-being of people living with dementia in their care facilities. To reach this – oftentimes challenging – goal, they offer a host of engaging activities daily. In this context, “reminiscence therapy” is a proven and frequently used type of intervention that involves the discussion of past events and experiences – ideally individual ones – with either a single person living with dementia or groups of people living with dementia. It has been shown to have a positive impact on the social, mental, and emotional well-being of people with dementia (Woods et al., 2018). Reminiscence therapy is ideally supported by props such as videos, music, pictures, and objects that hold significant meaning for the person with dementia and thus trigger personally meaningful memories (Woods et al., 2018).

In the past, conventional media (such as personalized photos, photo books, audio recordings, etc.) were primarily used in reminiscence therapy to provide positive emotional experiences to people living with dementia (Woods et al., 2018). In recent years, interactive forms of media are becoming more and more popular (Laird et al., 2018; Lazar et al., 2014; Subramaniam & Woods, 2016), since they offer the possibility to effectively elicit positive memories through multiple sensory channels as well as allow for easier access to extensive media content databases and thus covering a wider range of potential interests (Goodall et al., 2021). On the other hand, the search for relevant, well-suited content for people living with dementia can be particularly time-consuming and resource-intensive (Bejan et al., 2018; Lazar et al., 2014). In terms of gathering materials for reminiscence therapy, caregivers usually resort to individual biographical information (i.e., information about the person’s past and his or her interests) to select appropriate content tailored to the specific needs of the people living with dementia participating in the reminiscence session (Bermingham et al., 2013; Woods et al., 2018). This becomes even more difficult and tedious the less information and/or content is available regarding a person’s life history (Alarcão et al., 2022). Furthermore, reminiscence therapy is particularly challenging in group contexts, as this particular form must take into account the common interests and experiences of all participants to be effective and enjoyable for people living with dementia (Bermingham et al., 2013).

In summary, caregivers need to access content quickly and simply during reminiscence sessions to decide on and select content that is beneficial to the person with dementia (Bermingham et al., 2013).

To support a more modern and time-saving form of daily reminiscence therapy with quickly accessible as well as suitable dementia-friendly media, a technology-assisted solution must be developed that can also be easily integrated into daily care (Alarcão et al., 2022). An exemplary system combining an easily integrated daily care solution for rapid identification of physical reminiscence content with user-friendly technology was demonstrated by Paay et al. (2022). The system comprises sensors, a smartphone, and a smartwatch that enable individualized and context-aware reminiscence sessions. The sensors are used to identify biographically relevant objects for a person living with dementia and trigger a notification to the caregiver’s smartwatch when the person is in proximity to these objects. This approach facilitates the initiation of a spontaneous reminiscence therapy session tailored to the individual interests and needs of a person living with dementia. Previous solutions are often based on (a) the storage of biographical information – which means an additional effort for the caregivers, as they need to create biographies for all users and maintain and update them on an ongoing basis (Bejan et al., 2018; Schultz et al., 2021) – and (b) vast but unspecific media collections that aren’t necessarily dementia-friendly and thus potentially ineffective.

The use of an Artificial Intelligence (AI)-based media analysis framework can help caregivers efficiently find suitable content for people living with dementia by quickly filtering out unsuitable media, addressing a limitation of current system designs. Context-aware AI can be dynamically tailored to the (individual) needs of people with dementia without relying on a strict rule-based system, thereby addressing the issue that current system designs often utilize a pool of standard content without the possibility of adding new individual and personalized media to a reminiscence session.

In the context of this work, the term “media” always refers to images, audio, and videos. However, not all types of media can be analyzed with a single technology or algorithm. In contrast, media analysis itself is an encompassing term for the separate analysis of images, audio, and videos with different techniques. For our purposes, AI methods – especially appropriate deep learning techniques for analyzing picture (Wang et al., 2021), audio (Lau & Ajoodha, 2022) and video (Bekhet & Alghamdi, 2021) files – have proven to be effective as well as efficient in the past.

By incorporating domain- and user-specific information into the system’s knowledge, context-aware AI systems can provide more detailed insights into the operating domain and generate more valuable contextual predictions (Mehra, 2012; Brézillon, 1999). They offer a distinct advantage in the field of dementia care as they can utilize a wide range of people living with dementia-related information and biographical data, such as age, gender, situations, people, places, and hobbies, to provide more accurate and valuable results. This makes them a valuable analytical tool for improving the quality of life of people living with dementia by recommending and selecting useful media content for reminiscence sessions.

Since the manual identification of suitable content is very time-consuming and requires knowledge of the suitability of media for people living with dementia, an AI-based media analysis framework will be used to facilitate the selection of freely available internet media. Currently, suitable media content has to be manually identified and checked for suitability. Due to the abundance of online search results, this is a very time-consuming task that is usually avoided due to limited time resources of caregivers. In most cases, a pool of standard media content is used. Despite the extensive availability of digital content, high-quality, biography-oriented and individualized memory care remains a major challenge. Drawing upon the achievements of AI-based media analysis, as seen in current research, we are focusing our efforts on creating a specialized AI-based media analysis framework designed specifically for people living with dementia. The framework serves as a filter so that only dementia-friendly content is shown as a result of the media search. In addition, a recommendation system is planned to be integrated into the application for easier person-related media identification. In this way, the application can be used to support media-based reminiscence sessions without causing additional work for caregivers. Furthermore, this paper discusses the analysis of user requirements.

The DIDEM (Digital technologies for the care of people living with dementia (DIDEM¹) project aims to develop and evaluate novel approaches to the use of digital technologies in the care of people living with dementia, with a specific focus on improving the quality of life of both people living with dementia and their caregivers. What sets this project apart is its consistent pursuit of a participatory approach to technology development in collaboration with caregivers. One part of the project focuses on interactive media with a focus on sound and music for people living with dementia, another emphasizes virtual support and guidance for caregivers employing chatbot technologies, and the third aspect (to which this study belongs) relates to AI-supported approaches to support reminiscence work. In this part of the project, our aim is to develop a practical, context-aware AI-based media analysis and recommendation application for daily use in reminiscence therapy.

The purpose of this analysis is to understand the target group’s developmental needs and desires leading to the creation of a functional, practicable, and efficient system. We aim to answer the following research questions: (i) “How can an AI-based media application efficiently and effectively support caregivers in conducting reminiscence therapy?” and (ii) “What are the most common needs and wishes of the caregivers that should be taken into consideration for the final implementation?” Our contributions aim to offer suggestions to researchers and designers on how to develop AI-based media analysis and recommendation tools for people living with dementia from a user’s perspective.

In this paper, we report on the results of the participatory design process elaborating on requirements for the application from a dementia caregiver’s perspective to enhance the ease of use for caregivers (especially about media searches) to automatically suggest appropriate media for individual residents or groups. The focus is on planning a tablet-based application to support reminiscence therapy sessions.

This work consists of six sections, starting with related work in Section 2. The methods used in the co-design workshops with caregivers are described in Section 3. Section 4 then presents the requirements that emerged from the results. Finally, the discussion, conclusion, and future work are described in the Sections 5, 6, and 7.

2. Related work

This section presents several areas relevant to our work within the field of technology-supported reminiscence therapy.

2.1. Technology-assisted reminiscence work

The use of digital media, including images, sounds, and videos plays a crucial role in reminiscence therapy for people with dementia. One of the first specific projects was CIRCA (Computer Interactive Reminiscence and Conversation Aid), it initially started as a concept to assist people living with dementia in reminiscing (Alm et al., 2004). The project aimed to support reminiscence therapy with a computer-based multimedia system, and over the years, the project has undergone expansion and development (Astell et al., 2007, 2010, 2018; Purves et al., 2015). Since the publication of the CIRCA project, there has been an increased interest in designing interactive media experiences in residential dementia care. For instance, researchers have explored multi-modal interactions (Klein & Uhlig, 2016), the use of virtual environments based on gesture and motion control (Ang & Siriaraya, 2014; Bejan et al., 2018), the use of augmented reality (Hamilton et al., 2021), the use of pre-selected digital media (I-CARE) (Schultz et al., 2021) and personalized digital media (Sarne-Fleischmann et al., 2011), with the help of networked reminiscence therapy via videophones incorporating photo and video sharing (Kuwahara et al., 2006) and for the revival of autobiographical memories of people living with dementia (Bejan et al., 2020).

In recent years, the focus has shifted away from the simple presentation of multimedia content via a tablet computer (Astell et al., 2010; Gilson et al., 2019) towards the automation of reminiscence sessions. Welsh et al. (2018) developed the media application “Ticket to Talk” to facilitate conversations through general questions mixed with personal information from a resident profile. “Memento” is a context-aware system utilizing a smartwatch, smartphone, and sensors. It enables caregivers to conduct personalized reminiscence sessions by generating smartwatch notifications when people living with dementia approximate relevant objects (Paay et al., 2022). Other works describe recommendation systems to automate media identification and selection (Bejan et al., 2018; Steinert et al., 2022; Yang et al., 2013). Allalouf et al. (2020) present a music recommendation system that tailors playlists according to an individual’s age, place of birth, and popular songs from their younger years. The research conducted by Bermingham et al. (2013) presents an approach for an automatic recommendation system that generates media suggestions for group reminiscence therapy. To date, there is limited research available that discusses the utilisation of gathered information to provide suitable content recommendations for reminiscing purposes. In particular, the vast majority of solutions do not assess how people living with dementia react to the media (e.g., in a positive or negative emotional way) nor do they use this information to automatically adjust the media session.

Recent work states that enabling personalized experiences using multimedia content triggers memory recall (Abu Abu Hashim et al., 2015; Damianakis et al., 2009; Peeters et al., 2016; Ryan et al., 2020), reduces anxiety and depression (Davison et al., 2016), supports social interaction and communication (Abu Hashim et al., 2015; Karlsson et al., 2013; Laird et al., 2018; Ryan et al., 2020) and strengthens identity (Karlsson et al., 2017; Massimi et al., 2008). Furthermore, there is evidence that generic multimedia content can also evoke meaningful memories if tailored to individually meaningful “life themes” (Astell, 2009; Davis & Shenk, 2015; Subramaniam & Woods, 2012). A “digital twins” approach may also provide a technical solution for developing personalized systems for person-related dementia care, as Wickramasinghe et al. (2022) proposed.

2.2. Requirements analysis in the context of technology-assisted care for people living with dementia

Several studies have been conducted in the past to identify the requirements for various technical assistance systems within the context of dementia care (Berrett et al., 2022; Fikry, 2020; Hyry et al., 2011; Kerkhof et al., 2017; Meiland, Reinersmann, et al., 2012; Meiland, de Boer, et al., 2012; Nakamura et al., 2021). However, in the most cases, these technologies primarily target individuals with dementia and, in certain cases, caregivers who serve as proxies to convey the wishes and needs of those affected (de Jong et al., 2018; Meiland et al., 2014). In the project “Living Well with Anne” (de Jong et al., 2018) requirements were elicited through focus groups with caregivers. The authors proposed a user-centered requirements analysis for both people living with dementia and their caregivers.

In the context of multimedia-based reminiscence therapy, most systems are primarily directed towards caregivers, aiming to facilitate content work with reminiscence content and thus making it easier to prepare effective reminiscence sessions. In summary, the development and evaluation of multimedia systems for reminiscence therapy has shown to be effective in facilitating conversations with caregivers and eliciting personal memories. Alarcão et al. (2022) have conducted a user-centered study that prioritized the requirements of caregivers with a specific focus on their needs when working with people living with dementia by utilising biographically related reminiscence content. The study of Sarne-Fleischmann and Tractinsky (2008) investigates the effectiveness of a personalized multimedia system developed for individuals with disabilities and their caregivers. The system was designed using a user-centered co-design approach and subsequently evaluated through iterative processes.

2.3. Context-aware AI media analysis approaches in healthcare

The benefits of using AI in assistive dementia care have been underpinned by previous research (Varghese et al., 2021; Su et al., 2022; Qi et al., 2022). The study conducted by Leong (2017) falls within the realm of research focused on supportive and collaborative AI caregiving agents for people living with dementia and their caregivers. This work addresses the necessity of effective collaboration within dynamic and variable environments. Throughout their evaluation, they underscore the importance of contextual applications in tailoring and adapting a system to various scenarios. Enhancing (AI) systems with contextual information yields significant benefits in customizing them for specific scenarios or individuals (Baldauf et al., 2007), and it offers numerous benefits commonly associated with context-aware AI systems, including adaptability, customizability, and explainability (Brdiczka, 2022). These advantages are particularly relevant in healthcare (Gubert et al., 2020; Zon et al., 2023), especially in dementia care (Baumgarten & Mulvenna, 2010; Bharucha et al., 2009), because personal, biographical and situational information can serve as contextual system information, to improve a system’s outcome. This is confirmed from Si et al. (2007) and Du et al. (2008) by developing contextual reminding systems for increasing the independence of early stage people living with dementia and from Kikhia et al. (2009) with “MemoryLane,” a context incorporating life-logging system for independent elderly living.

Bricon-Souf and Newman (2007) present a contextual analysis framework designed for implementation in healthcare applications. In addition to presenting this framework, they also conduct a comparative analysis of existing contextual healthcare projects. In doing so, they shed light on common challenges in the domain, such as the creation of efficient systems that can effectively mediate human perspectives. The works described in (Pei et al., 2020) (integrate contextual parameter directly into the training phase), (Lam et al., 2019), and (Abdollahi et al., 2021) focus on integrating contextual information into learning systems in healthcare using different methods: Pei et al. (2020) integrate contextual parameter directly into the training phase for brain tumour segmentation. Lam et al. (2019) utilize a data augmentation procedure based on topic modeling as input for a transformer model and Abdollahi et al. (2021) apply ensemble neural networks for the detection of chronic diseases such as diabetes, cardiovascular conditions, or breast cancer. Ongenae et al. (2013) introduce a self-learning and ontology-based framework that enables context-sensitive applications in healthcare to adapt their behaviour at runtime, leading to higher user adoption.

Numerous studies within the healthcare field outline context-aware AI image analysis approaches. In most cases, they aim to detect diseases with picture-based approaches. Vercauteren et al. (2020) emphasize the critical importance of considering human factors and contextual information to effectively address the challenges faced in interventional imaging within clinical suites and operating rooms, particularly in the interpretation and utilization of surgical images. Chennamsetty et al. (2018) provide a detailed description of the integration of image information into an ensemble neural network to identify breast cancer using cancer images. Soydemir and Unay (2013) developed an approach for context-aware dementia diagnosis via brain imaging, by combining contextual parameters with image information.

Audio-related work (Nanni et al., 2021) was published in the research area of context-aware deep acoustic signal processing to analyze audio content by integrating contextual information into a deep ensemble learning system with signal images of bird calls, cat sounds, and environmental sounds. The sound categories are suitable for certain dementia-related life themes.

The work of Shih (2018) describes a survey on content-aware video analysis for examining sports videos. Biancalana et al. (2011) are presenting a context-aware movie recommendation system-based approach using signal processing and machine learning. The approaches described by the authors could be adapted for the selection of videos for people living with dementia.

2.4. Context-aware recommendation approaches in healthcare

Recent studies demonstrate that the integration of context into healthcare media recommendation systems is a promising approach (Gopalan et al., 2011; Kim et al., 2014; López-Nores et al., 2013; Schäfer et al., 2017) – which should also be applied within the realm of reminiscence therapy for people living with dementia.

3. Methods

Following an iterative, participatory design approach, we present the first phase of co-design focused on creating and exploring the concept of an AI-based media analysis and recommendation application. The concept of co-design encompasses the inventive collaboration between designers and non-designers, as they work together during the process of design development (Sanders & Stappers, 2008). Co-design provides a way of ensuring that the needs of users – in our case caregivers – are taken into account when developing new technologies and interventions (Rodgers, 2017). A total of three workshops were held between October and November 2022 with caregivers from three residential care facilities to jointly redesign the overall system based on the pre-defined categories, building on the system developed in a previous participatory project (Bejan et al., 2020): (1) profiling, (2) media search, (3) stored media, (4) and media recommendation. Given the growing importance of context-aware AI, the researchers also decided to discuss the potential relevance of context-aware media analysis for the system with the carergivers. We conducted one workshop in each of the three residential care facilities.

3.1. Participants

As professional caregivers are the primary user group when it comes to the implementation of reminiscence therapies and the use of multimedia reminiscence content, we decided to involve them in the development phases of the system. For this purpose, care facilities were recruited at the start of the project using purposive sampling. The current study refers to a series of participatory workshops that took place as part of the design phase. Participants in the study were selected through the gatekeepers of the care facilities. The voluntary nature of participation was the guiding principle for the selection of participants. A total of 14 professional caregivers from three residential care facilities participated in the workshops. The general inclusion criteria for participation in the co-design workshops were caring for people living with dementia in institutional long-term care and having a certain experience in delivering reminiscence therapy sessions. Participants were recruited through the residential care facilities cooperating in the project. Ethical approval (Approval number: 21-016) was gained from the Research Ethics Committee Deutsche Gesellschaft für Pflegewissenschaften (DGP). Informed consent was given for the whole project, so for the workshops described here, consent was only asked for in the sense of ongoing consent at the beginning of each workshop.

3.2. Low-fidelity paper prototyping

Based on the experience and knowledge gained in the multimedia reminiscence therapy project InterMem (Interactive Memories) (Bejan et al., 2019) and the follow-up project RemeMTI (Bejan et al., 2020), as well as on recent literature, a first low-fidelity paper prototype was created by the authors – the leading researcher and developer – prior to the workshop, so that the participants could get a better understanding of the technology to be developed. Therefore, the Miro Wireframe Library (Khusid, 2011) was used. Since co-creative processes involving AI can be challenging due to the complexity and “invisibility” of the design object (Bratteteig & Verne, 2018), it was decided to share and make information visible, which is relevant to this stage of development with the participants. The prototype consists of wireframes that were designed to make the three previous application areas of media search, profiling and stored media easier to understand for those involved in the design process, and to give them the opportunity to discuss design options using a concrete artifact. Paper prototypes were perceived as helpful in encouraging active participation from users because of the physical interaction with the mock-ups. As a supplement to the paper prototypes, user scenarios focusing on the three application areas were created prior to the workshop as artifacts to support negotiation processes to (a) establish a common understanding between researchers and caregivers about the design context, (b) to generate ideas for innovative features, (c) to organize and prioritize potential content, and (d) to identify weaknesses while exploring possible solutions.

3.3. Co-design meeting procedures

We conducted the workshops in the real-world environment of the user group. One workshop per institution was held, with one conducted online due to Covid-19 restrictions using Zoom and Miro digital whiteboard (Khusid, 2011). The workshop activities actively involved the caregivers in joint work on and revision of the paper prototypes and the scenario design. The aim was to empower caregivers of people living with dementia to become co-designers by including their perspectives, knowledge, and experiences in the design and development of the system resulting in user requirements. Each workshop was facilitated and led by a project staff member (who acted as a facilitator and researcher) with working experience in institutional care for people with dementia as well as experience in conducting co-design workshops with caregivers, while another trained researcher participated as an observer and note taker. In addition, the lead tech developer was involved in the workshop to answer questions about the technical feasibility of certain features and to record observations from a technical perspective.

Figure 1 shows the use of the paper-based mock-up during a co-design session. Each workshop was divided into two parts:

Figure 1. Co-design session with caregivers on AI-based recommendation and media analysis application.

• Part I: Exploration of user needs and goals using scenarios and paper-based mock-up;

• Part II: Elaboration of requirements for the media rating process with low-fidelity prototyping;

In the initial phase of the workshop, the idea of the system was introduced using the paper-based mock-up and the textual scenarios. The scenarios that described the functionalities of the application were read aloud during the workshops and then discussed, making sure all participants understood the idea. Following this, the application requirements were developed from focused discussions. Notes were taken of all suggestions made by the participants. The mock-up was organized into three key categories: (1) Profiling and recommendation, (2) Media Search, and (3) Media Storage.

The second part of the workshop emphasized the importance of contextual awareness and the process of media rating. The latter is generally perceived by caregivers as time-consuming and redundant documentation. Participants were provided with several design elements to be arranged on a paper-based frame while having the option to add their elements to the prototype.

During the co-design workshops, the wishes and needs of the participants were jointly identified, discussed, and transferred into decisions about the future system. The results were used to formulate the functional requirements and finally develop the prototype.

3.4. Data collection and analysis

The workshops were audio-recorded, transcribed, and then – along with the field notes – qualitatively analyzed using “thematic analysis” (Braun & Clarke, 2012). All data were reviewed independently by two researchers – the workshop facilitator and the note-taker. Due to the use of findings from previous projects, themes were not developed solely inductively through reflexive thematic analysis, as is usually recommended for this method of analysis. Rather, they were developed deductively, supplemented by inductive codes. The analysis process included familiarization with the data, coding the text to identify aspects relevant to the research question and to reduce the amount of data, identification of further inductive themes within the data, and review and validation of the identified themes. For clustering and rearranging, the data was visualized in Miro and shared with the application developer for further refinement. Several techniques were used to ensure rigour and reliability, including inter-rater reliability checks during the coding process. Each researcher conducted an initial coding of the data independently, identifying themes and patterns according to their interpretation. Following this individual coding phase, we met as a team to compare our findings and discuss any discrepancies. Through iterative discussion and consensus building, we refined our coding and clustering process and finalized the themes together. This approach helped to reduce potential bias and increase the credibility of our analysis.

The results of the workshops served as input for the developers to create the prototype, which will be further co-(re)designed with the caregivers in the second iteration.

4. Results

This section describes the system design proposals derived from the users involved in the participatory co-design sessions, which formed the basis for the definition of the functional requirements. The research relates to a comprehensive system that includes context-aware AI-based media analysis, a recommendation backend, and a user-friendly interface tailored to the needs of caregivers.

The thematic analysis of the co-design workshops with professional caregivers provided insights into the design considerations for an AI-based media recommender application for reminiscence therapy. Initially, four deductive themes were explored: Profiling, Media Search, Recommendation, and Stored Media Management.

In the area of Profiling, which is based on the collection of biographical data about an individual, participants emphasized the importance of features such as self-filling through learning, profile information based on life themes, dealing with the dynamics of disease, and ensuring low-effort profile maintenance.

In the area of Media Search, which includes searching for appropriate reminiscence content, preferences were expressed for simple media search functionality, assistance with the search process, and consideration of group interests to facilitate engaging reminiscence sessions.

In terms of Recommendation, participants emphasized the importance of using innovative approaches to improve the process of recommending media for people living with dementia. Key considerations included developing recommendation algorithms tailored to dementia-friendly media and incorporating user feedback mechanisms to refine suggestions.

Stored Media management focuses on features that allow the storage of reminiscence concepts or sessions and the provision of session templates for efficient reminiscence session planning.

Subsequently, three inductive themes emerged through reflexive thematic analysis: Media Rating, Personal Media use and Context Aware AI.

Under Media Rating, which refers to the documentation of reminiscence sessions as a basis for more appropriate recommendations in future sessions, participants highlighted the importance of mechanisms for low-effort media rating and automated note-taking to capture session outcomes and facilitate ongoing evaluation.

In terms of Personal Media use, considerations included the type of media used, the process of uploading content and the involvement of relatives in managing private media content. However, this aspect is not covered in this study, as it is the subject of separate research.

The Context-aware AI theme covers adaptation to changing contextual parameters relevant to people with dementia, such as gathering information based on personal profiles to adapt to the progression of dementia and associated disease dynamics.

These themes and categories provide guidance for the development of an effective tool to support reminiscence therapy in dementia care and are presented below in the form of requirements.

4.1. Overall system

Figure 2 depicts the structure of the whole application prototype concept, highlighting the respective areas that emerged from our workshops and previous work. The data flow between the components is illustrated by the arrows.

Figure 2. Application structure: overall system.

The functionality for conducting a reminiscence session is provided by the Media Search component. It offers a search interface for media retrieval or displays recommendations generated by the Recommendation System. The media obtained from a search iteration is planned to be downloaded from public image databases on the web and filtered by the Context-Aware Media Analysis component, ensuring that only dementia-friendly content is retrieved. The Recommendation System generates suggestions based on profile information obtained from the Profiling component. A profile is created for each person living with dementia in a facility. After completing a reminiscence session, a resident-specific evaluation (Media Rating) of the media used during the session is performed. The ratings serve both to populate the media profiles of the residents and as feedback for the Recommendation System, which processes the new information to generate even more suitable media recommendations. The Media Storage component allows successful media to be saved for later use and to provide sessions already prepared from research.

Aside from defining the application’s structure, the workshops resulted in the identification of both functional and non-functional requirements for the planned prototype. The non-functional requirements for the entire application can be viewed in Table 1, include addressing one of the caregivers’ most significant needs: using the application without any additional preparation effort. The application should be directly supportive for conducting reminiscence sessions and also cater to users with less technical skills.

Table 1. Overall system: non-functional requirements.

Requirement	User’s perspective	Technical description
Usability and simplicity	The application should be designed to be user-friendly, intuitive, and easy to understand, enabling users with limited tech literacy to utilize it without requiring extensive instructions or training.	The user interface (UI) design should be kept minimalistic, intuitive, and user-friendly to minimize the likelihood of user errors and ensure ease of use. Only the most important information should be displayed on the UI.
No preparation effort	The application should be seamlessly integrated into a reminiscence session without requiring any additional preparation effort or work.	The application should have the capability to generate all reminiscence session contents in real-time during a session, eliminating the need for time-consuming preparation efforts and the creation of biography forms.
Security	The system should ensure that unauthorized users are prevented from accessing both the application and the stored data.	The use of an encrypted user login should ensure that the application and stored data are protected from unauthorized access.
Adaptability	New requirements that may arise during the course of research should be accommodated in the application’s design.	The code of the application should be designed in a way that allows for easy adaptation to accommodate the evolving needs of people living with dementia.
Privacy	To ensure the highest level of protection for the personal data stored within the application, it is crucial to comply with the data protection regulations that are in effect within the country where it is being used.	The application should comply with the data protection guidelines according to the General Data Protection Regulation (GDPR). (Union & Union, 2016)

The remaining requirements for the main areas of the system are described in the following sections.

4.2. Profiling

Life story sheets have an important role in the care of people living with dementia (Kindell et al., 2014). Such a sheet contains information about the life of a person living with dementia, for example about important events, preferences or hobbies. During the workshops, the caregivers emphasized that integrating such a document into an application and maintaining it can be very time-consuming. They also expressed concerns about identifying dementia-friendly media quickly due to the large number of search results. They emphasized that the application would not be utilized if it imposes additional effort in conducting reminiscence sessions. This non-functional requirement is shown in Table 2.

Table 2. Profiling: non-functional requirements.

Requirement	User’s perspective	Technical description
Time saving profile creation	The procedure should be designed to be time-saving and capable of being carried out without the need for preparing a biographical sheet.	The profiling process should be kept simple, requiring as little information as possible. However, it should take into account important events in the biography of a person living with dementia to ensure that valuable information is not lost.

In our research, the question arises: How can the process of simplifying and improving the creation of life story sheets be addressed?

For this, a suitable profiling strategy was needed. The evaluation result from the caregivers suggests starting with an initially empty profile that gradually fills up over time through people living with dementia-related media ratings (such as like, dislike and not sure) during a reminiscence session. Each rated medium will be stored in the profile of a person living with dementia.

Starting with an almost empty profile means that important information known about a person living with dementia should not be lost at the time of profile creation. The information known about a person living with dementia will be integrated in the form of specific life themes that can be tailored to their personal life history and interests. We define life themes for reminiscence therapy for people living with dementia as overarching topics or subjects that encompass various aspects of a person’s life experiences. These themes aim to stimulate memories, engage individuals in meaningful conversations, and enhance their overall well-being. While specific themes may vary according to individual preferences and backgrounds, general life themes can be used that are universally representative of a generation’s experiences. The workshop participants felt it was important to specify general life themes for each user to provide the application with more individual information about a person living with dementia without having to create an entire biography but allowing for more personalized recommendations. For example, for the life theme of “sports,” possible specifications could include “skiing” or “football.”

The described process ensures that the profile contains valuable information at the time of creation, which can be utilized for the personalized media content recommendation system. The profiling process correlates strongly with the recommendation system, as it uses the profile information to make suitable suggestions. Based on the feedback, profile information can be updated in turn, to ensure that the profile remains up-to-date and grows with the person, even as their dementia progresses.

Figure 3 illustrates the described process with the help of an UI solution for the process of creating a profile, by using a Miro-based UI mockup. Figure 3 contains an example for including known preferences of a person living with dementia at the time of profile creation. The designed process involves the following steps:

Figure 3. Application mockup: profile creation process.

1. Entering the name

2. Selecting preferences based on life themes (e.g., sport)

3. Specifying the life theme (e.g., football, skiing)

If a profile is created without specific life themes, generic life themes that are generally relevant to a particular generation are assumed as default values. The entered values serve as profile-related meta-information and should be stored in a database so that the recommendation system can access and edit the information.

The study revealed a set of functional requirements for the Profiling component – as detailed in Table 3, that encompass the entire profile life cycle including layout design, creation, completion and deletion. After the profile creation process, it is necessary to have an appropriate representation and visual division of the profile area. From the workshops, it becomes evident that it is meaningful to categorize the media stored in a profile based on the criteria by which they were rated. As such, the profile areas include: like, dislike, not sure and personal media.

Table 3. Profiling: functional requirements.

Requirement	User’s perspective	Technical description
Profile creation	The only information needed to create a profile is the name of a person living with dementia. There should be an option to select some known biographical information about a resident.	A profile should be capable of being created simply by entering a resident’s name, with the optional ability to select life themes from a list of already known universal life themes. There should also be an optional facility to specify these using free text input.
Profile deletion	When a person living with dementia leaves a facility, their profile should be removable.	When a person living with dementia leaves a facility, the entire profile with all associated data should be removed from the application.
Profile content generation	Profile content and important biographical information should be automatically generated over time.	Following a reminiscence session, the content of a profile should be assigned a personalized Media Rating, which can be changed if necessary. The rated media should then be stored within the profile and categorized accordingly. The categorization of media allows for retrospective adjustments, ensuring flexibility. To keep track of the rating history, a caregiver-specific rating history, that documents which caregiver rated which media item, should be maintained.
Profile layout	For easier profile orientation, the profile content should be divided into the following areas: like, dislike, not sure, and personal media.	The media should be assigned to a profile area (like, dislike, and not sure) based on the media rating conducted.
Personal media	It should be possible to add personal media to a profile.	The media should be added to a profile by a simple file upload. For privacy reasons, personal profile content should only be accessible via a resident’s profile and should not be made available to the general public.

During the workshops, caregivers expressed their desire for an interface through which relatives could upload suitable media to a person living with dementia’s profile. However, uploaded media must be approved by a caregiver to prevent relatives – who are highly likely to not have nursing training – from causing “damage” to the already generated profile content. This could be implemented, for example, through a simple web application. Since this is not essential for the development of the application, we have considered this requirement optional and did not add it to the requirements table.

4.3. Media search and rating

The quick and easy execution of reminiscence sessions is the central element of the proposed application concept, for which a media search is essential. During the workshop, it was determined that media search should be conducted either through user-defined search terms or with the assistance of the application, which should provide relevant media suggestions from a Recommendation System.

At this point, the question arose: How many media examples should be generated within one search iteration, regardless of whether the media search is carried out via a search word or with the help of the Recommendation System?

The caregivers discussed and concluded that each search iteration should generate four media suggestions. If these are not suitable, there should be an option to generate four additional media suggestions.

Figure 4 shows the developed concept of the process of a reminiscence session using the described application prototype. The process of a reminiscence session supported by the proposed system is as follows: First, the persons who will participate in the session are selected. Next, the media search is performed using the Context-Aware and AI-based Media Analysis and Recommendation System. Then, the session can be completed by media evaluation. One requirement expressed by the caregivers is that media rating should be optional. However, it is important not to neglect this aspect despite its optional nature.

Figure 4. Reminiscence session approach.

The media search process entails inputting a keyword into the search interface, prompting the application to retrieve pertinent media content that corresponds with the given keyword. The Recommendation component plays a supportive role in the media search by suggesting suitable media content based on the participating person living with dementia, thereby assisting in finding appropriate media resources. The search is planned to be conducted with the help of an Application Programming Interface (API), which points to free media databases in the world wide web (e.g., Pixabay (2023)). The identified media are then passed to the Media Analysis framework, which can be influenced by the Recommendation component, depending on the selection of the participating people living with dementia.

Caregivers should be able to store the media resulting from a search iteration in a media tray, which is a key component of the proposed application. The media tray acts as a temporary cache for media and is used to facilitate reminiscence sessions. By selecting media from the media tray, users can view them in full-screen mode to conduct their session.

Table 4 shows the functional requirements of the media search, including the selection of the participating people living with dementia, the media search and selection, the media suggestions and the media tray.

Table 4. Media search: functional requirements.

Requirement	User’s perspective	Technical description
Selection of people living with dementia	Each person living with dementia should have their own profile. It should be possible to select people living with dementia for a reminiscence session by clicking on a profile and searching with text. Choosing a single profile should be possible if a one-to-one session is preferred. In the case of group therapy, several profiles should be selectable.	Profiles should be listed alphabetically and should be selectable by clicking on them. Selected profiles should be highlighted. Multiple selections should be possible for both group and individual therapy. A free text search should be available to find people, and search results should be displayed in a list format.
Media search and selection	The media search function should be accessible by entering a free text query or by selecting an example life theme provided by the application. Each search iteration should return four results. The option to search for more content should be available, allowing users to find additional relevant media based on their preferences and interests.	The entered text should be sent as a search request to public media databases. Selected life themes should be treated in the same way as free text. Search results should be analyzed for suitability for people living with dementia. Four media options should be displayed for selection, with the option to load more, enhancing the user’s ability to find appropriate media content for their reminiscence sessions.
Media Suggestions	To facilitate the media search, the application should be capable of automatically generating media suggestions based on the participating people living with dementia. Additionally, a list of sample life themes should be available to simplify the search process.	The media suggestions should be generated by a recommendation system, based on the profile content of the participating people living with dementia.
Media tray	The media tray should be used to conduct and support a reminiscence session with the application. The media resulting from the media search should be displayed here. Clicking on a media will display it in full-screen mode. If it is an audio or video file, a multimedia menu should be provided. Additional media should be added to the media tray during a session.	The media tray should store all identified media in a scrollable list. Clicking on a medium should allow selection and full-screen viewing. Audio or video files should have a multimedia menu. New media should be able to be added during a session while existing media remains in the tray.

The media rating should be completed at the end of each reminiscence session. It is important to rate the media content used in the session on a person-specific basis to provide feedback to the Recommendation System that can help improve the accuracy of future media recommendations. In addition, this process is necessary to populate the profiles with media content. However, the problem identified by caregivers was that individual rating of the media content used in the session can be time-consuming and carries the risk of double documentation. Nevertheless, there is an awareness of the concept of media rating. The caregivers emphasized that the evaluation process should be kept as short and simple as possible and should always relate to specific media to avoid double documentation. Through the expert workshops, it was determined that using a rating system with “like,” “dislike” and “not sure” buttons for media evaluation is an acceptable and efficient way for media rating. The caregivers thought that providing additional feedback options, such as indicating that (a) a picture is inappropriate due to the biography of the person living with dementia or (b) not dementia-friendly, would be helpful. However, the entire media evaluation process should be optional and designed to allow for the exclusive rating of specific media parts of a session. The corresponding functional requirements are shown in Table 5.

Table 5. Media rating: functional requirements.

Requirement	User’s perspective	Technical description
Media rating	Media ratings are important but should be optional. The rating should be provided for each participating person and each medium used in a session.	Media ratings should be used as feedback for the recommendation system and to fill the profiles of the people living with dementia with media content, but they should not be mandatory.
Media rating options	Media rating should be done on a per-resident basis using three different options: “Like,” ”Dislike,” and ”Not Sure.”	To keep the media rating as simple as possible, the rating should be done by clicking on a button (“Like,” “Dislike,” and “Not Sure”).
Media rating dislike [optional]	If a medium is disliked, an additional query should be answered. To keep it as simple as possible, the query should be button-based.	If a medium is rated as disliked, additional information should be provided as to why the medium did not fit, whether it was not dementia-friendly or did not meet the biographical needs or interests of the people living with dementia. This information should be used to identify which component needs improvement (media analysis or recommendation component), allowing for continuous enhancement of the application’s performance and content selection.

As described in Section 3, the design of the Media Rating system was developed through the creation of a paper-based mockup during the second part of the workshops. The design incorporates the partial results from all workshops. The Media Rating UI mockup is shown in Figure 5.

Figure 5. Application mockup: media rating.

The interface is divided into two sections, the first section contains a medium (it is scrollable), and the second section contains the participating people living with dementia and icons for each rating option (like, dislike and not sure). Caregivers can rate the media specifically for the person living with dementia by clicking on the icons. Only one choice is allowed per person living with dementia. If a dislike is selected, a pop-up window will appear allowing the user to specify whether the media is considered inappropriate because of the individual’s biography or because it is not dementia-friendly.

4.4. Stored media

The experts introduced the idea that it should be possible to run a session with stored media content. In any case, the selection of participating people living with dementia is crucial for a successful session, even if a session is started with previously stored media content. If a caregiver wishes to add more media during the session, the participating individuals are needed as input for the Recommendation System to make valuable media suggestions. In addition, media rating and feedback to the Recommendation System, as well as the filling of profiles with media, can only take place if the system knows which people have participated in a session.

We assume that all media stored in the application’s Media Storage has been subjected to Media Analysis at some point in the past. To prevent this information from being lost and to save resources, it is important to store it in a database in the form of media-related metadata. Meta-information can be any information that can be extracted from a medium (AI-based). For example, in the case of an image, (Context-Aware) AI-based Media Analysis could identify the content, objects, number of objects, colors or contrasts to describe it.

Caregivers expressed a desire for pre-designed media templates related to common life themes to be available in the application for easy selection. In addition, if valuable media content is discovered during a reminiscence session, it should be possible to store the media within the system.

A description of the arosen requirements for the Media Storage are presented in Table 6.

Table 6. Media storage: functional requirements.

Requirement	User’s perspective	Technical description
Participating people living with dementia selection	To conduct a reminiscence session with previously stored media sessions, it should be possible to select the participating people living with dementia.	Selecting the participating people living with dementia is a prerequisite for the recommendation and media rating components.
Meta information	Each media item stored in the application should have associated meta information that should be used internally by the system and should not directly affect the user.	The system should have meta-information about stored media to meet the needs of the recommendation system and media analysis. storing this information prevents duplicate analysis and helps to provide accurate media suggestions. Meta information can include any details about a media item.
Templates	The application should provide pre-composed media templates based on life themes.	Pre-defined media templates should be stored in the application together with their associated meta-information.

4.5. Media analysis framework

For the application to automatically detect whether a medium is suitable for people living with dementia, the Media Analysis Framework is required. This component is designed to act as a dementia filter, ensuring that only media that meets dementia-friendly criteria is made available in the application. Especially AI media analysis techniques have proven their worth in the past (Bekhet & Alghamdi, 2021; Lau & Ajoodha, 2022; Wang et al., 2021).

By employing a context-aware approach, it becomes feasible to personalize media content according to each individual’s distinct context and preferences (Xu et al., 2010; Hwang et al., 2007). The objective is to design an AI system capable of incorporating contextual information about a person living with dementia into the training process.

The requirements for the analysis of the media are set out in the following Table 7.

Table 7. Media analysis framework: functional requirements.

Requirement	User’s perspective	Technical description
AI	The media screening process to determine the suitability of media for people living with dementia should be implemented using AI techniques.	AI, machine learning (ML), and deep learning (DL) techniques should be employed for media analysis.
Adaptable	As additional relevant media features could be identified in research over time, it should be ensured that the AI system can adapt to changing circumstances.	The system should be adaptable to integrate new contextual information without requiring a time-consuming re-training of the entire network.
Explainable	The decisions about whether or not media is dementia friendly should be understandable to the user.	The system should be able to explain what it does and why, as it will be used directly in the context of people in need of care, ensuring transparency and understanding of the criteria used by the system.
Customizable	Users should be able to customize the system according to the specific needs and preferences of each person living with dementia, ensuring that the system addresses the unique needs and sensitivities of each person living with dementia.	A user should be able to define rules to change the media results, to meet the specific needs of each person living with dementia.
Context awareness	The Media Analysis process should utilize a context-aware AI approach to meet the other media analysis requirements: adaptable, explainable, and customizable. All information related to a person living with dementia should be considered as context information during the analysis.	Incorporating contextual information into a learning system significantly enhances the value of its results, seamlessly imbuing it with explainability, adaptability, and customizability. To fully harness the benefits of all its features, integrating context into the system necessitates the creation of a modular ensemble learning system.
Image filtering criteria	The AI system should be able to process all types of media: images, audio and videos.	The AI system should be able to analyze both syntactic and semantic media features, especially in the area of images, focusing on parameters such as contrast, colors, brightness, object identification, number of objects, object size, and semantic image content.

4.6. Recommendation system

The Recommendation System is necessary for the planned application to assist the media search process. This way, caregivers can focus on people living with dementia without having to put extra effort into finding suitable media. The goal is to automatically suggest media content (images, audio, videos) that is suitable for an individual person living with dementia or a group of people living with dementia. By using the collective tastes and interests of all participants in a reminiscence session, the Recommendation System can generate media suggestions that are tailored to the entire group and ensure that the tastes and preferences of all participants are considered. This approach can lead to a more enjoyable and engaging experience for all participants.

The Recommendation component is strongly correlated with the Profiling component, it facilitates media suggestions based on the profile contents of the people living with dementia. The utilisation of existing information regarding people living with dementia profile content ensures the preservation of important data and addresses the well-known item and user cold start problem in Recommendation Systems, described by Gope and Jain (2017). By leveraging pre-existing information, the system can overcome the challenge of lacking initial data and provide more accurate and personalized recommendations for people living with dementia. The problem refers to the state of a system that is unable to provide users with relevant recommendations (suitable media) as the system lacks information about the user. To tackle the item-related cold start problem, we address it by analyzing and storing meta-information related to the media. This meta-information provides valuable insights into the media content and assists in generating more accurate recommendations. The specific requirements for the Recommendation component are outlined in Table 8.

Table 8. Recommendation system: functional requirements.

Requirement	User’s perspective	Technical description
Personalized	The system should generate media suggestions tailored to the entire session group, ensuring that everyone’s biographical needs and preferences are taken into account.	The recommendation system should be highly flexible, allowing for tailored suggestions for both individuals and groups of people. It should also provide the option to adjust the number of remaining participants to account for varying group sizes.
Media category	The media suggestions made by the system should be related to images, audio, and videos.	The system’s ability to make recommendations should not be limited to a specific type of data, as it should be capable of making recommendations for a wide range of media, including images, audio, and video.
Explainable	A user should be able to understand why a media suggestion has been made.	A user should be able to understand the reasoning behind a media suggestion. To maintain transparency in the decision-making process, especially when working directly with individuals, the recommendation system should be explainable.

5. Discussion

This section provides a summarizing discussion of the work, including a reflection on the findings, the reflection on the participatory process as well as general limitations.

5.1. Reflection on the findings

This paper reports on the user requirements for a Context-Aware and AI-based Media Analysis and Recommendation System, which were identified through participatory co-design sessions with caregivers. The idea was carefully evaluated and discussed in close collaboration with workshop participants, ensuring their perspectives and needs were considered.

The original app’s concept could be further refined, resulting in six application components: (a) profiling, (b) media search, (c) media storage, (d) recommendation system, (e) media rating, and (f) context-aware media analysis.

5.1.1. Reflections on profiling

In particular, profiling needs to be efficient as well as time-saving and thus includes self-building profiles to avoid intensive profile maintenance (which typically relies on regular updates of biographical information by caregivers).

5.1.2. Reflections on media search

An effective media search relies on the selection of participating people living with dementia and their biographical data, as the Recommendation System can only generate suitable media suggestions based on this information. Moreover, the field partners have expressed the need for a free text search feature to quickly find appropriate media for people living with dementia. This includes the implementation of AI-based media filtering to ensure that only dementia-friendly content is available in the application.

5.1.3. Reflections on the recommendation system

Different types of recommendation systems could be used for people living with dementia. In particular, content-based filtering and demographics-based filtering systems are suitable. Content-based filtering systems rely on suggestions based on past user ratings and interactions with similar items, while demographic filtering recommendation systems use explicit knowledge about a person such as age, gender, location as expressed by Bobadilla et al. (2013) – or, in our case, biographical “life theme” information. Furthermore, collaborative filtering recommendation systems are particularly suitable in our case, as they can incorporate user ratings (Bobadilla et al., 2013). In our proposed system, we can provide these ratings in the form of media evaluations.

5.1.4. Reflections on media analysis and storage

Currently, there are no existing approaches that use AI technologies to effectively filter digital media for people living with dementia. Given the widespread adoption of digital systems and AI technologies in today’s era, it is important to address this gap, particularly in the context of people living with dementia in our ageing society. Our application supports the concept of dementia content screening through the Context-Aware AI Media Analysis component, which ensures that non-dementia-friendly content will be filtered out and not displayed to caregivers. Appropriate media content can be selected by the caregivers and stored in the current session’s easily accessible media tray, for further use.

Past studies have shown that using AI tools for the analysis of digital media has proven to be effective (Bekhet & Alghamdi, 2021; Lau & Ajoodha, 2022; Wang et al., 2021). Therefore, we propose to use AI techniques for the processing of images, audio, and video, as shown below.

During the co-design sessions, criteria for filtering images were gathered to create dementia-friendly images. Deep learning offers various methods to analyze image data based on the required information. For instance, simple classification networks can identify objects in images, object detection networks can highlight and measure object sizes, and visual attention networks can extract and describe semantic information from images. To achieve the goal of extracting both semantic and syntactic features from images, Convolutional Neural Networks (CNN) are considered a target-oriented solution and are widely regarded as the gold standard for AI image analysis.

Examples of CNN-based technologies used for classifying and detecting the mentioned criteria include well-known classification networks like VGG (Simonyan & Zisserman, 2015), Xception (Chollet, 2017), and Inception (Szegedy et al., 2016), as well as object detection networks like YOLO (Redmon et al., 2016) and R-CNNs (Girshick et al., 2014). Additionally, visual attention networks (Xu et al., 2015), which combine a “classic” classification network with a recurrent neural network for caption generation, are suitable for extracting semantic image criteria. These networks not only name objects present in an image but also capture relationships between them.

The workshops demonstrate that, from a technical perspective, logging and processing of syntactic as well as semantic object attributes is necessary, to provide dementia-friendly images. Both are possible by leveraging CNN-based methods, leading to the selection of dementia-friendly reminiscence image content and enhancing the effectiveness of context-sensitive systems in dementia care.

In the field of audio file analysis, image processing techniques have also proven effective (Pandey et al., 2019). Different types of diagrams can visually represent an audio file using a two-dimensional time-frequency representation. These image files can be processed and analyzed using CNNs that are typically used for classical image classification (Nam et al., 2019).

Videos can also be analyzed with the image neural network techniques described above since the video is a sequence of time-varying images. Applications in the field of people living with dementia include motion detection, object recognition and tracking, human action recognition, scene understanding, and video classification (Sharma et al., 2021).

Another outcome of the workshops is that media should be filtered taking into account the contextual features surrounding it. people living with dementia may have human-related information, such as age, gender or level of dementia, and a wide range of biographical preferences; a context-aware system is adaptable to individual biographical preferences. These biographical themes can be described by life themes that have played a significant role in the life of an individual person living with dementia, for example, animals, travel, nature or hobbies.

5.1.5. Reflections on context-aware system design

Furthermore, the system must be adaptable, customizable and explainable in terms of feature extraction for selecting dementia-friendly and non-dementia-friendly images. Adaptability allows for easy adaptation to the changing needs of people with dementia and goes beyond providing generic search results. Transparency is essential as the system has a direct impact on individuals and adaptability is necessary to address different situations that may arise in a research context. These essential attributes are encompassed by a context-aware machine learning system (Brdiczka, 2019, 2022; Vallath, 2022) highlighting the need to develop a context-based learning approach.

Fouopi et al. (2016) explained, that integrating context directly into a single neural network is challenging, so it is more feasible to incorporate context as a pre-or post-processing step. One potential solution is the use of Ensemble Neural Networks (Fouopi et al., 2016), where the larger problem is divided into multiple sub-problems, each of which is addressed by training a separate neural network. These networks form an ensemble that can dynamically adjust its members based on the context, eliminating the need for complete model retraining (Zhao et al., 2005). This flexibility is particularly important in the research domain.

5.1.6. Reflections on media rating

In terms of the media rating process, rating based on fixed criteria is considered useful to improve user-specific suggestions and to fill in the personalized media profiles. The challenge is to integrate the rating process into the reminiscence session in a way that does not interfere with the interaction with the person with dementia at any point during the person’s memory/emotion activation. On the other hand, if the system requires evaluation after the session, this represents an additional burden for the caregivers to add to their daily documentation. In general, a system that requires additional effort will not be used by caregivers. Furthermore, there is no need for systems that question the competence of caregivers or try to replace well-functioning processes. What is needed is a contextual understanding of the problems at hand that can be effectively supported by technology in reminiscence therapy, which in turn provides the basis for user-oriented requirements gathering.

After the media evaluation, the application offers the possibility to save a well-received media session. In addition to the media itself, the meta information generated by the Media Analysis is also stored in the application.

5.1.7. Reflections based on previous research

Previous work, such as CIRCA (Alm et al., 2004) and I-Care (Schultz et al., 2021), serves as the foundation for designing and conceptualizing the planned multimedia reminiscence session application. This builds upon efforts aimed at automating reminiscence sessions (Paay et al., 2022; Welsh et al., 2018). What is special about our work is that it does not only focus on the automated selection of suitable media via a recommendation system (Bejan et al., 2018; Steinert et al., 2022; Yang et al., 2013), but also on moving away from the idea of a standardized pool of multimedia content. Instead, we integrate the utilization of public image databases to identify relevant media content, which is then assessed for its suitability in dementia care through cutting-edge AI techniques. A particular focus is on the inclusion of contextual factors in a media-analytical AI and recommendation system, this allows for dynamic adaptation to potentially emerging domain-shifts. The importance of a contextualized system is based on existing work within the healthcare domain, but not necessarily from the domain of conducting reminiscence sessions (Abdollahi et al., 2021; Lam et al., 2019; Paay et al., 2022; Pei et al., 2020).

Moreover, our work encompasses media evaluation for recommendation system advancement. The majority of existing literature lacks exploration into how media impacts individuals with dementia. Our work is characterized by a consistent participatory approach, involving caregivers in the application design process, similar to that undertaken by Paay et al. (2022), not only to elicit system requirements from a caregiver’s perspective, but also to give caregivers the opportunity to make decisions about the design and thus actively contribute to the development of a system that is supposed to support them in their reminiscence work. This is in contrast to most of the other work (Berrett et al., 2022; Fikry, 2020; Hyry et al., 2011; Kerkhof et al., 2017; Meiland, de Boer, et al., 2012; Meiland, Reinersmann, et al., 2012; Nakamura et al., 2021), whose systems are primarily aimed at people with dementia and do not consider the people who accompany the use of such technology as potential users of the system. Another aspect is that we have developed a time-saving profiling method that addresses the concerns raised in previous studies (Bejan, 2020; Bejan et al., 2020) about the time-consuming and redundant nature of existing and biography-based profiling methods.

5.2. Reflection of the participatory process

In this section, we outline the core lessons learned and insights gained from the co-design sessions. Incorporating participatory design processes in the context of AI application development represents a great challenge.

While a fundamental comprehension of AI is indispensable, it cannot be presumed when collaborating with caregivers. In addition, the implementation of design ideas is often difficult because practical prototypes are hard to realize and AI decision-making often lacks transparency, making it difficult for users to comprehend its actions (Bratteteig & Verne, 2018).

Paper-based UI screen mock-ups turned out to be advantageous for making the idea of the media application understandable for laypersons, as the complexity of the topic could be reduced through the tangible visualization of the relevant information. In addition to the visualization, textual scenarios and joint discussion about the system contributed significantly to the understanding of how the media application can assist with reminiscence work. Bratteteig & Verne (Bratteteig & Verne, 2018) also point to the importance of storytelling methods in the context of AI development but emphasize that a basic understanding of “what AI can do and not do” is needed. To concretize the design idea, the mock-ups and scenarios developed for our project gave the participants a concrete idea of the planned system without exposing them to excessive technical information. Using these methods helped caregivers recognize the importance of feedback in improving the quality of content recommendations, but at the same time made clear that the rating process should be kept to a minimum. Concerning the cold start problem and to support the usage of already known biographical information, they considered it necessary to assign individual life themes when creating an individual user (i.e., person living with dementia) and to specify them if possible. As an AI system can never achieve 100% accuracy, it becomes crucial to develop a strategy for handling poor or incorrect suggestions. Caregivers have expressed the desire for transparency in understanding why a media recommendation was generated or why an image is considered dementia-friendly despite not being so. To improve the corresponding intelligent system component, it is essential to incorporate feedback on inadequate media ratings, indicating whether a media suggestion is unsuitable due to alignment with the individual’s biography or due to dementia-friendliness. The workshops have demonstrated that caregivers are open to making additional ratings by utilizing a straightforward button selection for providing such feedback. This leads us to the conclusion that the workshop participants understood the importance of training data sets for AI-based systems as well as the progressive improvement of suggestions through usage and rating and last but not least, their significant role as users in this process.

5.3. Limitations

This research has limitations related to the technical implementation of the AI-based media application. The biographical content of a person living with dementia, including their background, hobbies, childhood circumstances and various life themes, can be highly individualized and broadly positioned. While it is possible to categorize some areas into relevant life themes, it remains a challenge to accurately identify a person’s specific preferences, especially when trying to recreate them in a reminiscence session using digital media. This poses difficulties for the recommendation system in generating appropriate media suggestions and for the media analysis component in filtering out appropriate media.

The main challenge lies in the training of a neural network, as it would ideally require a huge amount of training data to cover all possible life themes, which may be impractical or even impossible without constraints. One possible approach could be to use already existing “everyday life” datasets, such as COCO (Lin et al., 2014), Flickr (Plummer et al., 2015) or PascalVOC (Everingham et al., 2010). Their content can roughly cover the life themes of people living with dementia, allowing for the generation of a satisfactory application. However, these datasets still fall short of capturing the full spectrum of biographical content.

There are also several limitations inherent to the process. Firstly, the extent to which the opinions, imagination, and decision-making of the participating caregivers were influenced by the provided low-fidelity mock-up – which was solely designed to convey the general concept -, remains uncertain. In addition, the requirements were assessed in three residential care facilities, which may restrict the generalizability of our findings due to the limited number of participants and the qualitative nature of the co-design methods employed. Moreover, the COVID-19 pandemic presented several “communicational” challenges during the workshop. Conducting an online-only workshop with one of the three institutions resulted in specific difficulties, such as the need to adjust to an entirely remote setting without tangible interactions. Utilizing digital low-fidelity prototypes and scenarios as a foundation for the discussion proved to be much more challenging than in the face-to-face workshops and assessing the participants’ level of comprehension of the concept from a remote location proved difficult. Furthermore, physical separation prevented researchers from actively participating in the team discussions that took place.

6. Future work

The next step involves the development of the application into a functional prototype. This includes creating the graphical user interface as well as implementing the recommendation system and context-aware AI media analysis framework. To implement the media analysis framework, various contextual parameters are required in order to enable the system to adapt to people living with dementia. The identification and refinement of these parameters is a planned research question in the further DIDEM project process. Furthermore, exploring how to address inaccurate or subpar media predictions in the future constitutes a research inquiry that warrants investigation. The holistic inclusion of the context and its users also benefits our research efforts, as the parameters of the media analysis component can be easily adjusted without requiring a complete retraining of the neural networks. Finally, the system concept will be tested, evaluated and refined in the field after implementation.

7. Conclusion

Our research aims to contribute to the body of knowledge of healthcare AI (recommendation) systems that support caregivers in conducting efficient and effective reminiscence therapy sessions. We provide a set of requirements for a comprehensive user-centered as well as context-appropriate implementation of the system. The collaborative experience and the resulting co-developed requirements can be seen as confirmation as well as refinement of the original assumptions about the feasibility of an easy-to-use, time- and resource-saving system tailored to the individual needs of caregivers and people living with dementia alike. Our findings have emerged from a participatory co-design approach and are directly linked to the practical needs of caregivers. These include a system design that incorporates a time-saving profiling and media rating method, requirements for a media-analytical AI, and a recommendation system. In addition, our research highlights the need for a contextual system design that is dynamically adaptable to a potential domain shift due to human factors. Beyond the technical requirements, our work provides some lessons learned from our participatory approach to eliciting AI requirements using paper-based system mock-ups and scenarios, which we hope will be valuable to future researchers, designers and developers working on similar applications and approaches to improving care work.

Acknowledgements

The authors would like to acknowledge that this work is part of the project “Digital Technologies for the Care of People Living with Dementia (DIDEM),” which was made possible by funding from the Carl Zeiss Foundation.

Disclosure statement

The authors declare that there are no conflicts of interest regarding the publication of this work. This research was conducted impartially and without any financial or personal relationships that could be perceived as potentially influencing the outcomes, interpretations, or conclusions presented in this document.

Additional information

Funding

This work was supported by the” Carl Zeiss Foundation” under Grant [P2019-03-001].

Notes on contributors

Liane-Marina Meßmer

Liane-Marina Meßmer is a computer scientist specializing in Artificial Intelligence, working at the Institute for Data Science, Cloud Computing and IT-Security at Furtwangen University. As part of her doctoral studies, she collaborates with the Université de Haute Alsace, focusing on context-aware AI concepts for image and signal processing.

Patrizia Held

Patrizia Held is a health scientist at Furtwangen University’s Care & Technology Lab (IMTT). Her research focuses on user-centered and participatory technology development and innovative care concepts for people with dementia and their formal informal caregivers.

Alexander Bejan

Alexander Bejan is a digital health research scientist at Furtwangen University’s Care & Technology Lab (IMTT). His current research focuses on co-operatively designed assistive technologies for people with dementia, especially in the field of interactive/multimodal computer-based reminiscence therapy.

Christophe Kunze

Christophe Kunze is professor for healthcare technologies and head of the Care & Technology Lab at Furtwangen University.

Christoph Reich

Christoph Reich is professor at the Faculty of Computer Science at Furtwangen University, where he teaches in IT-security, machine learning, and distributed systems. Additionally, he is head of the research institute for Data Science, Cloud Computing, and IT Security. His research focuses on machine learning and IT-security.

Notes

1 https://didem.hs-furtwangen.de.