- Catalyst: Strategic – New Zealand-China Strategic Research Alliance 2022
- Catalyst: Strategic New Zealand–German Aerospace Centre Joint Research Programme
- Catalyst: Strategic – Auckland Bioengineering Institute 12 Labours project
- Catalyst: Strategic – New Zealand-DLR Joint Research Programme December 2020
- Catalyst: Strategic – New Zealand-China joint research partnerships 2020/2021
- Catalyst: Strategic – New Zealand-Singapore Data Science Research Programme
- Catalyst: Strategic – New Zealand-Singapore Future Foods Research Programme
- Catalyst: Strategic - MethaneSAT atmospheric science project
- Catalyst: Strategic – New Zealand-China joint research partnerships 2019/2020
- Catalyst: Strategic – The Cyber Security Research Programme
- Catalyst: Strategic – Space 2019
- Catalyst: Strategic – NZ-Korea joint research partnerships
- Catalyst: Strategic – a collaborative biomedical science research programme with China
- Catalyst: Strategic – the New Zealand-China Research Collaboration Centres
- Catalyst: Strategic – New Zealand-Germany Green Hydrogen Research Programme
- Catalyst: Strategic – Investment in health-related A.I. research in partnership with Soul Machines
- Catalyst Fund
Catalyst: Strategic – Investment in health-related A.I. research in partnership with Soul Machines
MBIE has announced the 3 successful proposals under the $6 million investment in internationally connected research projects that will explore the potential applications of Soul Machines’ ‘Digital People’ TM Human OS Platform in an area of healthcare research.
About the programme
This Catalyst Strategic investment, conducted in partnership with Soul Machines Ltd, aims to advance the development of New Zealand’s dynamic digital health research ecosystem. The Call invited proposals internationally connected research projects that explore the application of Soul Machines’ ‘Digital Human’ A.I. technology in an area of healthcare research.
Soul Machines is a world leading A.I. company, whose Human OS platform aims to enable researchers and international organisations to leverage the full capabilities of human and machine collaboration. The New Zealand research teams were able to apply for up to $2 million over 3 years from MBIE, with Soul Machines providing the successful applicants with access to a ‘Digital People’ Product Environment for the projects’ duration.
Three priority areas were identified for this programme: Mental Health, Metabolic disease, and Medical Triage, with applications outside these areas also accepted for consideration. Proposals were also required to involve collaboration with world-class international research partners, with a focus on the United Kingdom, the United States, Canada, the Republic of Korea, Japan, and Germany.
The set of 3 projects was selected through independent assessment, with a total MBIE investment of $5.1 million (GST excluded) between them.
|Lead NZ organisation||Project title|
|University of Auckland||Tōku Hoa: A Personalized Agent for Mental Health|
|University of Canterbury||AI-driven Two-Way, Feedback Controlled Emotional Recognition Training for Individuals with Autism Spectrum Disorder|
|University of Auckland||Digital Twins for the management of chronic metabolic disease|
Public statements of funded projects
Tōku Hoa: A Personalized Agent for Mental Health
Imagine having a friend who knows how you are feeling, is always available to talk, and can come with you to meet a therapist when you need more help. This is the vision of Tōku Hoa: to create a digital person that provides continuous access to personalised mental health services wherever and whenever they are needed.
Mental health is a significant problem in NZ, particularly amongst adolescents, due to lack of access to mental health professionals, and limited availability of culturally meaningful treatment options. This project aims to develop a digital person platform which provides ready access to mental health support, customised to the person’s needs. The platform provides an innovative combination of targeted mental health monitoring, support from a personalised digital person and an immersive therapeutic environment with access to real mental health professionals.
The hypothesis is that digital people can provide mental health support that compliments working with real professionals, leading to increased engagement with therapists. This will be tested using a platform with three core attributes: (1) using wearable devices for sensor-based long-term monitoring of a person’s mood throughout their daily activities. (2) a digital person offering mental health support, that appears whenever the monitoring indicates depressive symptoms, or upon request. (3) an interface that enables connection with a remote therapist who leads a therapy session, with the digital person as support for client and therapist.
These three elements contribute to continuous tracking of a person’s mood, a digital person providing personalised mental health advice, and access to trained high quality professional help in the user’s own environment. The digital person will be customised by the user to match their cultural, age/gender preferences, will adapt its mental health feedback to the persons needs over time, and will exhibit behaviours designed to create a trusted relationship.
AI-driven Two-Way, Feedback Controlled Emotional Recognition Training for Individuals with Autism Spectrum Disorder
The recognition of emotions is an essential basis for social interaction. This ability is very different among people with autism spectrum disorders (ASD), which are characterised by "social blindness” or the inability to recognize, understand, and adequately respond to the emotional states of others. Current efforts to ensure social participation and the ability to contribute to society typically focus on one-to-one or one-to-few counselling using example scenarios and mimicking to teach autism patients how to recognise expressions and emotions. However, this approach is costly, overbooked by increasing demand, and underserved by limited availability of specialized therapists and social workers.
This research proposes a digital solution combining the hyper-realism of Soul Machines avatars, computer vision based facial expression recognition technologies, and clinical therapeutic approaches to create a novel, two-way Digital Therapy Solution (DTS). The ultra-high resolution facial expressions of the avatars enable active, AI-driven therapeutic responses to interact with a subject. Computer vision emotion recognition, audio processing, and heartrate evaluates the patient’s individual emotional responses. Together, they create a novel two-way therapeutic system, where the avatar is an interactive element controlled by emotional/physiological feedback from the subject, and not just a display.
This solution augments one-to-one therapy, which is often in short supply and costly, and lets individuals practice further at their own time and pace, with training intensity controlled by the system based on feedback. All these outcomes offer therapeutic benefit and the potential to improve outcomes.
The main outcome is a foundation platform to build further interactive and software-based therapy solutions. It also creates a platform to “gamify” therapy, and eventually incorporate it within games requiring social interactions. Overall, it is a novel solution, enabled by Soul Machines’ novel avatars, to improve critical social skills for ASD patients improving their ability to succeed in school, work, and life.
Digital Twins for the management of chronic metabolic disease
Type 2 diabetes (T2D) and cardiovascular disease (CVD) are managed primarily at home by the patient and their whānau. Many patients struggle to understand their condition and how they should manage it, and very often they end up non-compliant with treatment strategies recommended by their doctor. This is especially true of lifestyle changes that can actually reverse early T2D. Intensive personal coaching or text-based motivation have been shown to be very effective, but these are too costly for widespread use.
The most promising way to address this growing and economically unsustainable problem is with a digital technology that augments current care. We will create a platform of interconnected digital tools, starting with a culturally appropriate ‘digital health navigator’ fronted by Soul Machines’ Digital DNA Studio, to coach patients in understanding and self-managing their condition. We will build on a complementary physiological modelling project called ‘12 Labours’ (funded by an MBIE Catalyst grant) to make a ‘Digital Twin’ that is specific to the patient.
The Digital Twin will be informed by wearable sensor measurements of key health biomarkers and patient/navigator conversations about lifestyle. A new AI platform will analyse patient data and outputs from the Digital Twin, to help with rapidly adapting the patient’s management plan. Partnering with Māori to shape the implementation of our technology will create a framework for culturally appropriate AI, and a platform applicable to a wide range of groups that experience unique challenges when engaging with the health care system. Our collaboration with a network of partners in Europe – who are funded through the European Commission – and primary care providers in New Zealand, provides local and international pathways for uptake of our research into patient care.
For more information, email email@example.com