Finding a Place for Machine Learning in Mental Health Assessment

With over 10,000 mental health apps available, it’s difficult to know which ones will actually have a therapeutic impact. Fortunately, enough high-quality evidence is available to help clinicians and patients make an informed choice.

By Paul Cerrato, senior research analyst and communications specialist, and Mayo Clinic Platform John Halamka, M.D., President, Mayo Clinic Platform.

“Castles made of sand fall into the sea, eventually.” Jimi Hendrix may not have been thinking about machine learning when he sang those words, but they certainly apply to many of the emerging AI-fueled algorithms. As we have pointed out in previous blog posts and medical journal papers, the foundation upon which these digital tools rest is rather “sandy.” Some algorithms that have reached the market are only supported by retrospective studies, rely on data sets that don’t represent the patient population they were designed to serve, or have proven unreliable because of data shift and bias. These problems beg the question: How does one separate the wheat from the chaff, especially in the field of mental health?

If you have ever witnessed a person’s psychotic break from reality, you know firsthand what a frightening crisis this can become. Both of us have seen it happen. All the more reason to ensure any AI algorithm that addresses mental health crises is based on strong evidence. Roger Garriga, with Universitat Pompeu Fabra, Department of Information and Communication Technologies, Barcelona, Spain, and his co-authors have published the results of retrospective and prospective studies that evaluated over 5 million patient records and more than 17,000 patients diagnosed with mood, psychotic, organic, neurotic, and personality disorders to determine if machine learning models might help predict which patients are most likely to experience a crisis requiring professional care. Among the best predictors:

• Weeks since the last crisis
• Never hospitalized
• Weeks since the last referral
• Number of crisis episodes
• Weeks since the previous referral from acute services
• Never needed mental health intervention
• Number of years since the first visit

Their retrospective analysis of data gleaned from EHR systems achieved an AUC of 0.797 and predicted a crisis with 58% sensitivity and 85% specificity. In the prospective follow-up study, the investigators asked clinicians to use the algorithm. They “observed predictions to be clinically valuable in terms of either managing caseloads or mitigating the risk of crisis in 64% of cases.” Given the mental health toll taken by the COVID pandemic and the shortage of mental health professionals available to manage a psychiatric crisis, a tool like this may improve the ability to prevent these episodes or treat them at a much earlier stage.

Other researchers have explored the value of AI for patients with less urgent psychiatric problems. Sarah Lagan and John Torous, MD, two of the world’s top authorities on digital psychiatry, have done a detailed analysis of the available mental health apps on the market and concluded: “Research and our initiatives at the Division of Digital Psychiatry at Harvard Medical School suggest that there are issues and limitations that app users need to be aware of. However, certain apps have the potential to be a successful supplement to mental health treatment if users find the right program to fit their individual needs.”

Among the problems they outline are limited accessibility, with many apps blocking the most important features behind a paywall, and a relative lack of support for certain psychiatric disorders, including schizophrenia and bipolar disorder. Other apps exaggerate their claims about being evidence-based or don’t provide the necessary features to protect users’ privacy, an especially important issue for anyone coping with a disorder that carries a social stigma.

But on a more positive note, Lagan and Torous point out that a carefully chosen app can complement professional treatment. Unfortunately, with over 10,000 mental health apps to choose from, the decision can be overwhelming, especially for a patient who is already dealing with the cognitive burden associated with many psychiatric conditions. To help address this issue, the authors have developed a search database of mental health apps that lets them use criteria that are most important to them as individuals. The Mental health Index and Navigation Database is available at MIND.

No one should have to cope with a serious mental health condition on their own. Fortunately, compassionate and well-informed investigators and clinicians continue to provide the best resources to get them through the journey.