Outcome Data

I support the "open data" movement towards making clinical outcome data available to consumers, payers, policy-makers, and researchers.

(Identifying information has been removed from the data.)

Atlantic.jpg

Big Data has transformed everything from sports to politics to education.

It could transform mental-health treatment, too—if only psychologists would stop ignoring it.

The outcome data process is highlighted in the Atlantic Monthly article “What Your Therapist Doesn’t Know” written by Tony Rousmaniere.

Read it here >

Drivers-Seat.jpg

Keeping Psychologists in the Drivers Seat:

An academic exploration of this topic is available in the paper "Keeping Psychologists in the Driver's Seat" in the peer-reviewed journal Psychotherapy: Theory, Research, and Practice.

 
Tony Rousmaniere Therapy Outcome Data.png

Psychotherapy Practice Outcome Data

A poster that describes a representative sample of my psychotherapy outcome data from two years of practice at the University of Alaska Fairbanks Student Health and Counseling Center.  This poster was created by Data Visualization expert Andy Kirk. 

Scroll down for more Data Samples & Resources…

Clinical Outcome Data Samples

The following are a sample of my clinical outcome data since 2011.

  • • Total # of clients in dataset: 42
    • Avg # of sessions: 8.58
    • Total # of clients with more than one session: 38 (90%)
    • Single session clients: 4 (10%)
    • Start in clinical range (of clients with 2+ sessions): 25 (66%)

    Clinical Outcomes

    • Ended Year in Clinical Change: 22 (88% of clients starting in clinical range)
    • Ended Year in Clinical Recovery: 20 (80% of clients starting in clinical range)
    • Ended Year in Clinical Deterioration: 2 (8% of clients starting in clinical range)
    • Average change in ORS score: 7.52  (clients starting in clinical range)
    • Cohen's D effect size for clients starting in clinical range: 1.22.  (Pre-tx std dev=6.14, mean intake ORS=22.27, mean last session ORS=29.79, n=38)

    Duration of Treatment

    • Avg # of sessions for low distress clients: 5.27
    • Avg # of sessions for high distress clients: 10.74
    • Avg # of sessions for clinical change to occur: 3.27 

  • Outcome Questionnaire-45 (OQ-45) data

    • Total # of clients in dataset: 102
    • Avg # of sessions:  4.60
    • Total # of clients with more than one session: 73 (72%)
    • Single session clients: 29 (28%)
    • Start in clinical range (of clients with 2+ sessions): 53 (73%)

    Clinical Outcomes

    • Ended Year in Clinical Change: 24 (45% of clients starting in clinical range)
    • Ended Year in Clinical Recovery:  13 (25% of clients starting in clinical range)
    • Ended Year in Clinical Deterioration: 3 (6% of clients starting in clinical range) 
    • Average change in OQ score: 15.06  (clients starting in clinical range)
    • Cohen's D effect size for clients starting in clinical range: .82.  (pre-tx std dev=18.25, mean intake OQ=86.30, mean last session OQ=71.24, n=53)       

    Duration of Treatment

    • Avg # of sessions for low distress clients: 4.24
    • Avg # of sessions for high distress clients: 4.78

    Outcome Rating Scale (ORS) data

    • Total # of clients in dataset:  89
    • Avg # of sessions: 5.37
    • Total # of clients with more than one session: 65 (73%)
    • Single session clients: 24 (27%)
    • Start in clinical range (of clients with 2+ sessions): 42 (65%)

    Clinical Outcomes

    • Ended Year in Clinical Change: 20 (48% of clients starting in clinical range)
    • Ended Year in Clinical Recovery:  11 (26% of clients starting in clinical range)
    • Ended Year in Clinical Deterioration: 4 (6% of clients starting in clinical range) 
    • Average change in ORS score: 4.84  (clients starting in clinical range)
    • Cohen's D effect size for clients starting in clinical range: .97.  (pre-tx std dev=4.98, mean intake ORS=15.63, mean last session ORS=20.47, n=42)

    Duration of Treatment

    • Avg # of sessions for low distress clients: 5.61
    • Avg # of sessions for high distress clients: 5.24
    • Avg # of sessions for clinical change to occur: 2.60
    • Maximum # of sessions for clinical change: 5.00

  • I do not have data for these years because my counseling center at the University of Alaska Fairbanks switched to a different outcome measurement program (the Counseling Center Assessment of Psychological Symptoms - CCAPS) which does not facilitate analysis of outcome data at the clinician level.  We started using the CCAPS because (a) it integrated better with our electronic records software, (b) was easier for counselors and medical staff to use for consultations, and (c) was designed specifically for university counseling centers.  

  • I moved to Seattle and started a private practice in the Fall of 2015. Following are the data from my practice from Fall 2015 to Winter 2018. The dataset is small because I work part-time as a therapist, some clients have opted-out of having their data included in this analysis, and I do not include data from clients who start treatment outside the "clinical range" (see below for an explanation), nor data from clients who attend "block therapy" sessions.  These results should be considered preliminary because there may not be sufficient data for a statistically reliable gauge of my work at my Seattle private practice.

    In 2015 my primary professional focus gradually shifted towards more writing, research, leading workshops and webinars, and providing consultation for agencies and training programs. This means I worked with fewer new clients and had less time for my own professional development, including clinical training, expert consultation, and deliberate practice. Research from the science of expertise suggests it is reasonable to expect that my clinical effectiveness plateaued and could decline over time.

    In 2020 I started the process of closing my private practice to fully focus my career on editing a books series for the American Psychological Association and starting a new psychotherapy graduate program based on the principles of Deliberate Practice.

    The outcome measure I used from 2015-2016 was the Behavioral Health Measure-20 (BHM-20). Since January 2016 I have used the Outcome Questionnaire-45.

    • Total # of clients with data: 38
    • Clients who started in clinical range (below "clinical cutoff"): 29 (76%)
    • Clients for whom outcome data is not reliable due to starting outside clinical range: 9 (24%)
    • Range of sessions: 1 to over 100
    • Clients with more than one session: 37 (88%)
    • Clients who did not return after first session: 4 (11%)
    • Average # of sessions: 10 (estimated)

    Clinical Outcomes

    • Clients who met the BHM-20 or OQ-45 "recovery" or “improved” benchmark: 12 (75% of the 16 clients who started in clinical range and had two or more sessions.  Two clients were "improved" and 10 were "recovered".)
    • Clients who showed no clinically significant change on the BHM-20 or OQ-45: 3 (19% of the 16 clients who started in clinical range and had two or more sessions.)
    • Clients who deteriorated: 1 (5% of the 16 clients who started in clinical range and had two or more sessions.)

Open Data Approach

Outcome Measurement and Client Response

I have used multiple outcome measures:

  • The Outcome Rating System (ORS), an ultra-brief measure with four items that is free to use.  Information about the ORS is here and here.   Clients completed the ORS at the beginning of each session, on a laptop computer or with paper/pencil.  It takes about 30 to 60 seconds to complete. 

  • The Outcome Questionnaire-45 (OQ-45).  Information about the OQ-45 is here.  Clients completed the OQ at the beginning of each session on a computer in the waiting room.  It takes about 3-5 minutes to complete.

  • The Behavioral Health Measure (BHM 20).  Information about the BHM-20 is available here.  Clients completed the BHM-20 at the beginning of each session, on a laptop computer or with paper/pencil.  It takes about 2 minutes to complete.

Consent for tracking outcomes is obtained at the beginning of treatment and is voluntary for clients.  Consent includes agreement for the outcomes to be used for research and publication, after being de-identified.

Based on my subjective judgment, I consider the ORS, BHM-20, and OQ-45 results to be valid for about 75% of my cases.  For some clients I think the measures under-represent the amount of positive change, especially for clients who started treatment with less clinical distress.  For some clients the measures over-represent the amount of positive change, in my judgement.  

Client Impressions of Outcome Measures

When I asked clients their thoughts about the outcome measurement, most clients were curious about their results but said they gave more weight to my subjective clinical impression of their case.  A few clients were skeptical of the validity of standardized outcome measurements, and a few clients refused to complete the measures because of this concern.  Some clinicians have reported that they do not collect outcome data because of concerns about harming the therapeutic alliance.  I did not find this to be the case, but rather found that discussing outcomes with some clients strengthened the therapeutic alliance.  Research suggests that clinician attitudes towards the outcome measures strongly affect clients’ reactions to the measures.

Outcome Measurement and Client Response

There are no outcome measures that are good at assessing the amount of change in psychotherapy for clients who start therapy with low levels of distress, termed the "clinical range" in the psychotherapy research literature.  For this reason, I do not have reliable quantitative data on the amount of change for low-distress clients.  

Limitations of the Data and Outcome Measurement

It is difficult to assess how good these results are, because there are no available data from therapists working in the same locations where I worked.  Likewise, it is not possible to compare outcome data from my different work locations because the client populations are different.  Another validity concern is that I do not have follow-up outcome data, so I do not know if gains from treatment are lasting. 

Therapy outcome measures are not perfect.  Multiple clients report negative change on outcome measures when in fact their symptoms improved.  Likewise, paper forms can never describe the rich complexities of growth and healing in therapy.  Given these limitations, I still think it is important for therapists to track their clients' outcomes with unbiased measures.  While the measures should never be used alone for clinical decisions, they are one important source of information regarding client progress or deterioration, and are can be helpful for assessing overall therapist effectiveness, when combined with a range of other assessment tools.

Critical Resources