Catalyst: Strategic – Investment in health-related A.I. research in partnership with Soul Machines

Imagine having a friend who knows how you are feeling, is always available to talk, and can come with you to meet a therapist when you need more help. This is the vision of Tōku Hoa: to create a digital person that provides continuous access to personalised mental health services wherever and whenever they are needed.

Mental health is a significant problem in NZ, particularly amongst adolescents, due to lack of access to mental health professionals, and limited availability of culturally meaningful treatment options. This project aims to develop a digital person platform which provides ready access to mental health support, customised to the person’s needs.  The platform provides an innovative combination of targeted mental health monitoring, support from a personalised digital person and an immersive therapeutic environment with access to real mental health professionals.  

The hypothesis is that digital people can provide mental health support that compliments working with real professionals, leading to increased engagement with therapists. This will be tested using a platform with three core attributes: (1) using wearable devices for sensor-based long-term monitoring of a person’s mood throughout their daily activities. (2) a digital person offering mental health support, that appears whenever the monitoring indicates depressive symptoms, or upon request. (3) an interface that enables connection with a remote therapist who leads a therapy session, with the digital person as support for client and therapist.

These three elements contribute to continuous tracking of a person’s mood, a digital person providing personalised mental health advice, and access to trained high quality professional help in the user’s own environment. The digital person will be customised by the user to match their cultural, age/gender preferences, will adapt its mental health feedback to the persons needs over time, and will exhibit behaviours designed to create a trusted relationship.

The recognition of emotions is an essential basis for social interaction. This ability is very different among people with autism spectrum disorders (ASD), which are characterised by "social blindness” or the inability to recognize, understand, and adequately respond to the emotional states of others. Current efforts to ensure social participation and the ability to contribute to society typically focus on one-to-one or one-to-few counselling using example scenarios and mimicking to teach autism patients how to recognise expressions and emotions. However, this approach is costly, overbooked by increasing demand, and underserved by limited availability of specialized therapists and social workers.  

This research proposes a digital solution combining the hyper-realism of Soul Machines avatars, computer vision based facial expression recognition technologies, and clinical therapeutic approaches to create a novel, two-way Digital Therapy Solution (DTS). The ultra-high resolution facial expressions of the avatars enable active, AI-driven therapeutic responses to interact with a subject. Computer vision emotion recognition, audio processing, and heartrate evaluates the patient’s individual emotional responses. Together, they create a novel two-way therapeutic system, where the avatar is an interactive element controlled by emotional/physiological feedback from the subject, and not just a display.

This solution augments one-to-one therapy, which is often in short supply and costly, and lets individuals practice further at their own time and pace, with training intensity controlled by the system based on feedback. All these outcomes offer therapeutic benefit and the potential to improve outcomes.

The main outcome is a foundation platform to build further interactive and software-based therapy solutions. It also creates a platform to “gamify” therapy, and eventually incorporate it within games requiring social interactions. Overall, it is a novel solution, enabled by Soul Machines’ novel avatars, to improve critical social skills for ASD patients improving their ability to succeed in school, work, and life.

Type 2 diabetes (T2D) and cardiovascular disease (CVD) are managed primarily at home by the patient and their whānau. Many patients struggle to understand their condition and how they should manage it, and very often they end up non-compliant with treatment strategies recommended by their doctor. This is especially true of lifestyle changes that can actually reverse early T2D. Intensive personal coaching or text-based motivation have been shown to be very effective, but these are too costly for widespread use. 

The most promising way to address this growing and economically unsustainable problem is with a digital technology that augments current care. We will create a platform of interconnected digital tools, starting with a culturally appropriate ‘digital health navigator’ fronted by Soul Machines’ Digital DNA Studio, to coach patients in understanding and self-managing their condition. We will build on a complementary physiological modelling project called ‘12 Labours’ (funded by an MBIE Catalyst grant) to make a ‘Digital Twin’ that is specific to the patient.

The Digital Twin will be informed by wearable sensor measurements of key health biomarkers and patient/navigator conversations about lifestyle. A new AI platform will analyse patient data and outputs from the Digital Twin, to help with rapidly adapting the patient’s management plan. Partnering with Māori to shape the implementation of our technology will create a framework for culturally appropriate AI, and a platform applicable to a wide range of groups that experience unique challenges when engaging with the health care system. Our collaboration with a network of partners in Europe – who are funded through the European Commission – and primary care providers in New Zealand, provides local and international pathways for uptake of our research into patient care.