A systematic review and meta-analysis of 57 studies scrutinizing the practice of standardized diagnostic interviews (SDIs) used in mental health assessments is raising serious questions about these questionnaires’ reliability, thanks to efforts led by psychological researchers at McMaster University in Canada.
Critics of current mental health practices say that mental health practitioners are working in the shadow of the 1980 edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM III). This revision—as one medical journal essay put it back in 2012—sought to paper over “sectarian discord among proponents of psychodynamic, behavioral, and neurobiologic explanations of mental illness” by reducing all psychological disorders down to digestible checklists of signs and symptoms.
The new deep-dive meta-analysis, which covered SDIs conducted with over 8,000 adults from 26 countries, found that these interviews turned out to be not particularly consistent across cases where the same patient was assessed via two separate SDIs even just days apart. More concerning still, the reliability between tests and retests appeared to vary significantly depending on which mental health issue was being investigated.
“If we give the same interview to the same person twice, we would like to think the interview would produce the same result, but that’s not always the case,” the study’s senior author Laura Duncan, an assistant professor at McMaster’s Department of Psychiatry and Behavioural Neurosciences, said in a statement.
Duncan, whose prior work as a doctoral candidate critically examined health research methodology, added: “Our findings show that these interviews are not as reliable or consistent as many people believe.”
Eye of the beholder
Duncan and her team, which included researchers from the University of Copenhagen in Denmark and the University of Massachusetts’ Chan Medical School in Boston, noted that these interviews were often more reliable for substance use disorders than for mental health issues.
“These differences suggest that structured interviews work better for conditions with clearer behaviours or timelines than for disorders that rely heavily on personal experiences and interpretation,” Duncan said.
Of the 57 studies pooled together for this meta-analysis, 46 were ultimately selected for detailed cross comparison, using a statistical method called Cohen’s kappa to compensate for unknown sloppiness in the interview data.
Pairs of SDI tests and retests on various individual patients coping with substance use disorders were found to be consistent in roughly 72% of cases, while SDI retests for mental disorders only matched their original test about 65% of the time. Reliability was basically 50/50 with SDIs testing for non-affective psychoses (a category which includes schizophrenia spectrum disorders); those retests came to the same conclusion in only 55% of cases. Retests for bipolar disorders, in contrast, matched the first SDI result in about 74% of the cases reviewed.
One of the most consistent diagnoses studied, they found, were tests for opioid addiction which was consistent 81% of the time.
A world of difference
Duncan and her colleagues acknowledged “several limitations” to their systematic review, including an “incomplete reporting of data,” which they wrote included missing information on certain variables that might have impacted each of the 46 studies’ data. Nevertheless, their conclusions were stark.
The researchers warned in their study, published last week in JAMA Network Open, a journal of the American Medical Association, that “readministration of a typical SDI (ie, getting a second clinical opinion) would result in reversing initial diagnostic findings for a substantial proportion of patients.”
Duncan’s team theorized that this discrepancy likely arises because mental health conditions—like anxiety, depression, or psychotic disorders—are more subjective experiences and can manifest in ways that are not only richly personalized, but also vary in time across a single patient’s life.
“Reliable diagnosis likely requires combining standardized tools with knowledge about the course and complexity of disorders that could impact how reliably they can be assessed,” Duncan said.
In other words, SDIs should be performed with both a grain of salt and some diagnostic backup, recognizing their limitations and integrating the results with more contextual information. At the very least, Duncan said, “We should reconsider treating them as a ‘gold standard’ of assessment.”
Read the full article here
