Columbia and the DSM: The Evolution of Psychiatric Diagnosis

By Christine Yu | Photos by Jörg Meyer

Michael First

Michael First

No resource has had a bigger influence on the field of mental health than the “Diagnostic and Statistical Manual of Mental Disorders” (DSM). Since it was first published over 70 years ago, it has evolved from what Michael First, MD, professor of clinical psychiatry at VP&S, described as a book that was used almost exclusively for administrative purposes for its diagnostic codes to become the comprehensive resource on the diagnosis of mental health disorders and the go-to guidebook for clinicians and researchers.

For Dr. First, it makes sense that his work on the DSM has become a defining part of his career. As a medical student with a computer science background, he wanted to develop algorithms that could help with medical diagnosis. “As I looked into it, I saw that psychiatry actually has these rules that are embodied in the DSM,” he says. Naturally, when he looked into residency programs, he wanted to work with the people who put together the DSM, then in its third iteration. And that led him to Columbia and Robert Spitzer, MD, a longtime psychiatry faculty member who died in 2015.

But when Dr. First came to Columbia as a resident, he didn’t necessarily know that he was joining a community of psychiatry experts who would ultimately play a major role in the development and use of the DSM.

Columbia’s Role in the Development of the DSM

Efforts to classify psychiatric conditions have been going on for centuries as a way of trying to make sense of psychiatric illness. After World War II, four different classifications of mental disorders were in use in the United States. The American Psychiatric Association created the DSM to unify the different approaches and establish a single standard for psychiatric classification, “simply to help us know who was in our institutions and what kind of diagnoses they had,” says Paul Appelbaum, MD, the Elizabeth K. Dollard Professor of Psychiatry, Medicine and Law. When the first edition of the DSM was published in 1952, it included 106 diagnoses, compared with nearly 300 diagnostic categories 28 years later.

While the early editions of the DSM gave mental health professionals a tool to categorize and track patients clinically, diagnosis was still hit or miss. Psychiatrists lacked clear or consistent criteria, with diagnosis based more on prevailing theories than empirical evidence. That meant that two psychiatrists could come up with different diagnoses for the same patient, which called into question the validity of diagnostic methods. “It was so unreliable, like flipping a coin,” says Dr. First.

During the second half of the 20th century, Columbia had become one of the meccas of psychiatric diagnosis, and the APA appointed Dr. Spitzer to lead the development of the third edition of the DSM. It was Dr. Spitzer who recognized the need for a more scientific system for diagnosis. When the DSM-III was published in 1980, it had nearly 500 pages and 265 diagnostic categories. Diagnostic labels based on psychoanalytic theories were replaced with a system based on descriptive symptoms, operationalized criteria, and agreed-upon decision rules.

The DSM-III led to a paradigm shift. It was embraced by clinicians who needed a sensible system to assess patients and make treatment decisions. It became a pivotal tool for researchers too. “All the studies that have been done in the past 40 years use the DSM system as a way to organize our knowledge about mental disorders,” says Dr. First. By improving the reliability of psychiatric diagnosis, the DSM also helped legitimize psychiatry in the eyes of the larger medical field and allowed for reimbursement from government entities and insurance companies.

The Columbia Influence

The DSM-III marked the beginning of Columbia’s long involvement with the publication. Dr. Spitzer was chosen to lead the effort to develop DSM-III as well as the subsequent edition DSM-III and to serve as an adviser to the next edition, the DSM-IV. In his role, Dr. Spitzer invited Columbia professors to join the task force and work groups that were integral to shaping the content and direction of the book. He mentored the next generation of editors too, including Dr. First, who did a fellowship with Dr. Spitzer, was involved in the development of DSM-III-R and served as the editor of the DSM-IV, the DSM-IV text revision (DSM-IV-TR),and the current edition, DSM-5-TR.

“If you wanted to have a breeding ground for people with an interest in psychiatric nosology, it would be difficult to imagine a better place than Columbia. In the halls on any given day, you can bump into leading experts in so many of the areas of psychiatry that are reflected in the DSM,” says Dr. Appelbaum.

Dr. Spitzer also created the first edition of the structured clinical interview for DSM disorders, known as SCID, which has become the most widely used structured clinical interview in the world. “We don’t have lab tests like other medical fields. We just talk to people,” says Dr. First, who helped develop and co-author successive editions of the SCID. “We needed a method to improve the reliability and validity of the diagnostic process. “SCID gave clinicians a standardized interview format and is considered the gold standard for making psychiatric diagnoses in research settings. It further cemented Columbia’s association with the DSM elevated the manual’s reliability.

But there were downsides to Columbia’s extensive influence over such a critical publication. When it was time to assemble the task force and work groups for the DSM-5, the APA didn’t invite Dr. Spitzer, Dr.First, or others affiliated with Columbia who had supervised prior editions to be part of the effort. “They wanted to cut Columbia out of the picture because they felt Columbia had dominated the publication,” says Dr. First. They also limited the number of work group members who could come from the same institution.

Dr. First says the impetus to exclude those who led the development of prior editions of the DSM was a desire to break from the past and move beyond the descriptive diagnostic system. Other areas of medicine have an underlying understanding of the pathophysiology, genetics, and other factors that give rise to a disease or disorder, but that’s not the case in psychiatry. Those leading the DSM-5 efforts hoped that incorporating etiological research into the new edition would lead to another paradigm shift.

Paul Appelbaum

Paul Appelbaum

But, Dr. Appelbaum says, “The reality of psychiatric diagnosis is that it is syndromic, which is to say we look at the cluster of signs and symptoms in order to categorize disorders. We’re not yet at the point where we can identify conditions based on their etiology.” Even if the etiology of psychiatric disorders could be identified, he says, it’s not clear that it would lead to more effective therapeutic approaches.

By excluding experts affiliated with Columbia, the DSM-5 task force was left with a leadership vacuum and no institutional knowledge about how to put together the publication, which led to confusion and delays. That led the APA to ultimately ask Dr. First to serve as an editorial consultant to help finish the project, which was released in 2013, 13 years after the previous edition.

Following the publication of the DSM-5, Dr.Appelbaum led the work group to develop recommendations for an iterative revision model. As chair of the DSM steering committee, he oversees the implementation of the new process. The APA appointed Dr. First as editor and co-chaired the text revision of the DSM-5, which was released in March 2022. “It’s back in the hands of Columbia,” says Dr. First.

Continued Evolution

While diagnosis is a cornerstone in the practice of medicine, publications such as the DSM have inherent limitations. By its nature, the DSM aims to categorize mental disorders and their accompanying diagnostic criteria. It requires establishing thresholds that set boundaries for mental disorders. What’s considered within bounds and what’s considered out-of-bounds can be contentious. Some worry that these distinctions could lead to misuse and overdiagnosis of normal behavior. It’s a concern that Dr. Appelbaum says the steering committee considers seriously.

Take, for example, the addition of prolonged grief disorder as a new disorder in the DSM-5-TR. Some worry that its inclusion pathologizes normal grief. Yet for some people, grief doesn’t always resolve: It can go on for an extended period of time and be debilitating. The challenge for the work group, which included M. Katherine Shear, MD, the Marion E. Kenworthy Professor of Psychiatry at Columbia’s School of Social Work and an expert in complicated grief, was to determine where to set the cutoff point, based on research data. The work group sought to be conservative as a way to minimize the number of false positives and false negatives. The group settled on the requirement that the disorder be defined by substantial impairment in everyday life or substantial distress that persisted for at least one year after the loss.

“We have to be very careful both about the conditions included in DSM and the thresholds that define those conditions so that, as best as possible, we’re identifying people with the disorder and not labeling people who are just a little different as having the disorder, “says Dr. First. At the same time, it’s important to recognize these disorders. “There are all these people who are suffering. By identifying these things as conditions, we’re actually legitimizing their suffering and helping them to get the proper treatment.

“What started as a book of codes used to facilitate payment has become foundational to the field of mental health, influencing perceptions, and understanding of mental health—by clinicians, researchers, and patients. It also has defined the field’s relationship to government agencies, insurance and pharmaceutical companies, and other medical specialties. The nature of a publication like the DSM is that it needs to be updated regularly to reflect advances in clinical practice and research as well as the changing nature of society.

“Columbia’s leadership in the DSM has been important to the growth and importance of the field of psychiatry,” says Dr. First, “and we hope to continue to be a part of the evolution of both the DSM and the profession to find better ways to help patients."