Measurement-based Care in Psychotherapy Workshop - Future of Psychotherapy Conference 2019

[00:00:00]so I realized actually quite a number of the points I was going to make in this workshop I've already made the questions that people have asked and and also my presentation so I'll flip through some things quite quickly but then spend a little bit more time on the things that we haven't covered if that's all right and so I'm really going to be talking about the potential benefits of regular outcome and process measurement so not just the outcomes but also measuring on a session by session basis the processes that you're trying to change in psychotherapy and so my plan of the talk is to say what we need to do to get complete data why there are real benefits for getting complete data from patients by complete I mean data at the beginning and end of treatments at least so that we know whether people have improved or how much they've improved and what you need to do to get there because it's not easy and then talk a little bit about the advantages of getting some measures that really very precisely target the particular clinical problems that you're trying to treat to not say just to generalize the GAD as a measure of anxiety of a more precise measures and then I'm going to talk a little bit about the case for why it's useful to also measure the psychological processes that we're trying to change in whatever model of psychotherapy that we're using and why that really does change outcomes a lot so firstly why it's important to get data well I'll be quick on this because I mentioned this in the talk I think the problem is that if we have missing data it tends to be the people who've done less well that are missing and so we don't really get an accurate idea of how we're doing as an individual clinician or as a service if we have missing data and knowing what things are going well but also what things aren't going well is something that we all want because you know this is what we got into the profession to do to try and help people but also understand the people that were not helping and be able to think about what we can do with them so that's why I think having the aim of getting data from everyone is really a critical one because you're just in a much better place as a professional because you know really what's happening and of course it's also the case that psychological treatments are much more difficult to deliver the medication and of course you know we have ways of trying to get around that we try and ensure that we all get as good a training as we can and hopefully many of us have the opportunity to get regular supervision because there are lots of studies that's showing however senior you are if you don't get regular supervision your outcomes you know tend to deviate a bit but the experience of I apt is that that is simply not enough here we have a vast national initiative which has standardized training and requires every therapist to have an hour and a half of supervision every week it's part of what happens if you're an ayap service so you know we sort of got that right but that didn't give us the outcomes we wanted at the start you see we were still a long way below where we wanted to be in it took us quite some time to get there but if we hadn't collected outcome data on everyone we would never have got to our 50% target and the only good thing I could say about it is we would have been ignorant of the fact that we hadn't but given the scale of the aya program I think I mentioned this morning in terms of you know the public that would mean that a hundred thousand people every year who we could get to recover wouldn't recover and you know that's not what we want as professionals is that so that's why getting the data is really critical because we can learn how to improve things from it and how do you get it and I think this is summarized some points again some of them have come up in discussion first I think it's really important not to spend too much your time worrying about whether you've got the perfect measure decide on some basic things that you would want to measure in everyone and collect that you can all mint it with other things but having our sort of minimum data set is good and what will be the key characteristics of measures that you're likely to find you can get data from most people well the first thing is they have to be brief if you're getting someone to do it every session and we find we don't get complete data unless you do it every session so that's the answer to getting you know consistent data but then it has to be something short it has to be valid so it has to discriminate between people with the problem you're treating and other people so if it's a measure of PTSD you know you have to score higher if you've got PTSD then you if you've got OCD you know it's got to be tracking the problem that you've got and it needs to be sensitive to change there are lots of valid measures which don't change very much for the repeated measurement they're very insensitive to that they've got the wrong time frame is over the last month say whereas you're seeing people every week so it has to be you know the right time frame so think about all of those and then give it every session because if people finish therapy earlier than you anticipate and that often happens then if you have aren't giving it every session you won't have a final score but you don't want to waste your valuable therapy time on measures so what we would do if we're using pen and pencil measures we would give them the pack of measures to the patient at the end of a session and ask them to fill them you know over breakfast or in the waiting room or on the bus to the session but not spending any clinical time on it but once of course they have filled them in always make sure you pick it up from the patient straight away and always look at it because if you're not showing great interest in it of course they'll stop using them and then do think about can you have an IT system which will help you for all the reasons we've talked about because it's very difficult otherwise you don't have the time to do all this graphing and look at these things in detail unless it's done fairly automatically for you and we're now using systems where people input them from the mobile phone so if you can get your hospital and your clinic to invest in a something like that it really helps your life a lot and patients really like it to do it that way and I think you want to know your measures very well and you want to choose the measures that you find most informative and that means if you look at individual items it will tell you something different if someone's scoring high on this pattern of items versus that pattern items I mentioned the case of PTSD where intrusive memories might be going down but that might be because of a bad thing avoidant more or avoiding less and you get that in in most of the other measures as well in the depression I should like the PhD you know there's one which is totally critical item 9 isn't it which is on risk and you'd always be looking at that but then there are other measures which are telling you very much more is there a sort of avoidance sort of thing going on there which you could target with behavioral activation or is there something else going on there so always look at the measures on an item-by-item basis and track changes in the individual items and I think it makes an awful lot of difference for the public buying if we're transparent or the public and so we our experience has been it is really good if a clinic puts on their website their outcome data and the public really want to come and see you then they can see actually people get better because many people in the public think well you know mental health problems are are bad it'll be nice to get some support but it's it's just going to take the edge off it a bit I'm not really going to get better and seeing that clinics are actually publishing data really changes public perceptions a lot and I explained that at least in our national thing we try and construct these measures so that there isn't an incentive not to collect the data but I think you all understand that bit of the map don't you so I'd like to say little bit about why it can be useful for some of your measures to be really very targeted on their problems that you're treating so here's an example social anxiety disorder one of the conditions that we treat this is from one of our randomized control trials and so we have on the far corner here a measure of recovery which is the standard I act on its have you dropped below the clinical threshold on the depression measure which is a pH Q but also on the general anxiety measure which is the GAD the trouble with just using that I mean social anxiety is that none of the items measure anything to do with social interactions and they don't measure social avoidance whereas there is a measure which we recommend in our service it's called the spin which does measure those things and if you don't give that you can see you can get very misled so in this study what I've plotted here is the recovery rates using the the general measure anxiety and the general measure depression and you can see in this trial where you are offered up to 14 sessions of therapy by session 7 you're doing fabulously well on the general measure 71% of people have already recovered but you can see actually on this spin and the pH Q it's only 28% and the reason for that is that what happens in therapy is early on people get met much less anxious in the situation's they're already going into but their life is still very restricted because there are lots of situations are completely avoiding but the GAD won't pick that up and you can see that that's what's happening because on the first column you have something called the wah sass work and social adjustment scale which is a measure of how much you're disabled in your life family life and work by it and you can see 71 percent recovery on the garden of phq your life hasn't changed the way so asus has only come down a little bit but once you come down on the spin as well you'll see a big change you've changed someone's life so it's really very important to measure the thing that the particular thing that you want to change precisely as well as having these general measures I know there's a big debate in psychotherapy around this but it if you want to change someone's life then you need the specific measure as well and you need a measure of disability how much is her life interfered with it's not just symptoms then you want to measure so then I'd like to finish off by talking about process measures so all psychotherapy is whatever school they're from have particular psychological processes that you're trying to target in therapy and ideally you want to be able to measure those on a regular basis as well we in our experience we come to therapists so all of our work is on Cobra therapy for anxiety disorders we find it immensely helpful to have measures of the constructs that we're trying to change every session so we have a measure of common negative thoughts that people have in different conditions whether it's PTSD social anxiety panic and how much they believe them and we look at that every week and we adjust what we're focusing on in therapy on the basis of which are the thoughts that have the highest belief rate that week and we may start with a particular set of thoughts because they have the highest belief ratings as we deal with them some other thoughts now are the ones that are most pressing and we shift the focus of therapy on to that similarly there are certain types of safety behaviors protective strategies that people use to manage their anxiety will give regular measures of those and we often find that if you look at the midway through therapy people have improved a lot but they're still using some of these safety strategies but if you can measure them systematically you can focus a second half of therapy on really just removing all of those safety strategies but if you haven't got regular measures you won't know that and I think this is one of the main reasons why cognitive therapy social anxiety does so well in comparison to other treatments because in most of the trials where it has done better than other treatments if we've had session by session process measures and the therapists have been using those to guide the content of their sessions there are a couple of international trials well that wasn't done so there's a German by flashing ring of cognitive therapy versus like dynamic therapy and Phalke prevented the therapist from using any process measures in cognitive therapy so they weren't allowed to measure the negative thoughts they weren't allowed to use safety behavior measures and although Kankuro p did better than the psychodynamic therapy still it in this particular case the difference neither treatment did anywhere near as well as in those studies where your measuring process all the time so that's just a plea to add that into your armoury and that's really all I wanted to say other than I mentioned there is a free therapist resources website that we have from our clinic and if you're interested in process measures particularly for come to therapy and social anxiety panic PTSD you'll find them all on that website along with lots of videos that you can watch if you'd like because we're sort of thinking now that therapist manuals they're okay but they're not great because if you're presented with a patient coming into your next session and you want to know what to do with them you're not going to read a whole chapter in a therapist manual but we have 10-minute clips of some of the key interventions and we find a lot of therapists will look at that before the post patient comes in so it's it augments the sort of manuals and thank you very much thank you very much David I'm going to kind of build on David's talk and take us in a in a bit of a different direction if I can get my slides okay right so what I hope to do in in a few minutes with you is to talk about evidence-based practice and practice based evidence and distinguish between the two to look at outcome monitoring in group therapy and talk about challenges and opportunities in implementation of outcome monitoring so first a general word about psychotherapy effectiveness and you'll see that David and I have have similar and different interpretations of the literature what we see is that a lot of reviews demonstrate the psychotherapy is robust effectiveness highly sought after by patients and a series of meta analyses signal to us that there is generally no single gold standard a psychotherapy if we're looking in particular for complex patients depression in particular we can also recommend group therapy has effective and equivalent to one-to-one therapy with the moderate effect size and it's more efficient with regard to therapist resources and recently group therapy has been recognized by the American Psychological Association has a distinct specialty which we see has a big advance because it's going to I think really elevate rigor in training we also know that access to mental health care is challenging and people in North America are getting less and less access to psychotherapy and are being prescribed antidepressants at a much greater rate one in five women in America is on a is on an antidepressants which is a stunning figure so having made a point about fundamental equivalence of therapies and models that's I think fairly good news the not-so-good news is that there is no such equivalence with regard to therapists is enormous variability with regard to how therapists do with their patients and this work emerges from a series of studies of looking at practice research networks so it's helpful I think to think about being an evidence-based practitioner in addition to using evidence-based psychotherapies and if we think about things in this context then we move in addition to empirically supported therapies such as cognitive behavior therapy for social anxiety we look at the role of practice guidelines and we look at the role of practice based evidence which is my segue to outcome monitoring and I see this as essential to remedy the quality gap because I think one of the worst things we can do is provide poor quality psychotherapy it demoralizes patients and demoralizes therapists and because our work is relationally based what we do matters and how we do it matters sometimes even more that's the implementation gap that we have to remedy and you heard an earlier question about common factors and what the literature tells us is that therapist effects and common factors like the therapeutic alliance contribute to two and a half times as much variance to outcomes as does a particular model and we need to be alert as to how we can use those elements that reside within us and reside within the psycho therapies across the board we have to be mindful of culture and we have to deliver what we do well the American Psychological Association recently completed its third task force looking at what is correlated with effective psychotherapy and the findings echo of a second task force with some some new emphases therapeutic alliance or group cohesion if we're looking about group therapy has a medium effect size good correlation with outcome empathy similarly good strong correlation with outcome empathy is tailored it's specific it embraces a cultural dimensions and it involves not only the receptive capacity to understand but the expressive capacity to understand and communicate that it adapts to improve alleges patience with regard to their attachment style and as I mentioned the role of culture reactants other patient variables the third important domain that is associated with better clinical outcomes is client centered outcome tracking that's what I'm gonna focus on for the rest of my few minutes why is this important I think David has made already a strong case about how much outcomes improved when data was tracked why is this important because we underestimate deterioration effects and there's a study in individual therapy and studying group therapy that when therapists are told you have patients in your care who are failing therapists fail to recognize this clinically it's quite surprising but the findings are robust some of it I think is our own lack of self-awareness 90% of therapists rate themselves as being the top 25% performers and that their patients get better at a much higher proportion than they actually do so routine outcome monitoring is a way for us to offset our therapists positive bias and an overestimation of our effectiveness and catch deterioration or patients who are off-track earlier so we have an opportunity to remediate the difficulties and a series of studies have shown that patients at risk of deteriorating who are being monitored are much less likely to deteriorate and if you add not only outcome monitoring and I'll talk in a minute about some of the some of the prevalent models but you add clinical support tips to therapists you again dramatically increase your effectiveness and we see positive outcome on clinical outcomes reduce dropouts and reduce treatment failures and SAP Samsa the American substance and addiction and mental health agency looks at evidence-based practice has including routine outcome monitoring so if you want to be an evidence-based psychotherapist do what you do but do it in the best way possible and the best way for you to know that you're doing it in the best way possible is don't assume you're doing a good job get data that shows that so in essence everyone in this room can be a local clinical scientist using the equivalent of GPS in your car to augment your clinical judgment a couple of the more prominent systems that are used in North America are the OQ 45 you've come questionnaire or the partners for change outcome monitoring system and generally what these systems have in common is that they measure outcome based upon individual distress well-being interpersonal relationships and social function and they also capture a process measure as david referred to that evaluates what is the quality of the relationship what is a quality of the alliance and gives you data that tells you you're on track you get a green message yellow that says this patient is not progressing the way he should or a red message that says this patient is off track and headed for trouble unless you do something differently if you're working in the area group therapy that process measure could be effectively used has the group questionnaire which samples the patient's experience of the quality of the bond the presence of negative factors in their relationship and the quality of the work group as we know is much more complicated than individual therapy because you have so many different layers of interaction but the data that's coming out from recent studies shows that using a group cohesion group outcome I'm sorry group question they're measure improves your ability to assess what's happening in your in your clinical care feedback can come to you as a therapist it can come to you and your patient or it can be aggregated at a system level the more you use each platform is easier and quicker it is because we want to be able to get this data in a quick and timely fashion and you can even use the reports as part of your clinical record-keeping what the routine outcome monitoring also shows us is that for treatments that are progressing generally well adding routine acum monitoring probably doesn't give you a big return on your investment but for the 20 or 30 percent of unrecognized treatments that are going poorly it has a dramatic impact improving outcomes from one in five to better than one in - and reducing deterioration rates from 20 percent to 5 percent patients can be readily engaged in using these kinds of measurements it's often the providers who are reluctant to utilize this rather than the patients and evidence shows we can improve quality of care and reach more collaborative decisions about duration of treatment what the literature also shows us and what clinical experience shows us is that providers are often resistant this was a question that came up at the panel I think people don't like the scrutiny they don't like the intrusion and has bruce Walpole has noted if you have to pay for your patients measurement it's the equivalent of asking a general practitioner to pay for the blood work that patients in his or her clinic see so we need a kind of macro level response to make this more widely adopted and some studies have shown that when this is introduced into an organization and I think David made reference to this earlier this morning some people are going to be very unhappy and if the organizational culture is critical and shaming people will use the data has a reason to punish therapists rather than to elevate their performance and effectiveness and of course the most important thing is that you get the data you use the data and there are studies that show for some people even when the data is easily accessible electronically it's not embraced and utilized psychotherapeutic Lee this is what the outcome monitoring looks like if you are working in a group you'll see obviously a made-up name you'll see feedback about the positive bond positive work negative relationship you can get it on each individual patient you can get a group questionnaire outcome on each individual patient or on the group as a whole and you can see a series of outcome question there alerts that shows you what the initial score was and what the most recent score was and you get a red alert for somebody who's deteriorating a yellow alert green alert as I mentioned before I want to share quickly about how this kind of routine outcome monitoring can be transformative in a system I was a consultant to Mercy behavioral health care in the states and they demonstrated really a transformational approach to providing care in 23 hospitals providing care to 25,000 patients in the u.s.

[00:28:28]Midwest by using group therapy and monitoring outcome and this was a Mercy Hospital in Chicago was like the Mercy Hospital you see on TV it was a place where security guards would only walk in pairs it was that level of disturbance but across her system they found that they were able to move beyond meds and beds to meaningful effective and memorializing care by training their social workers how to deliver group therapy monitoring that and utilizing outcome data to improve quality of care across a system they described a kind of friendly competition that developed across different settings transformative with regard to patient satisfaction transformative in regard to staff morale they trained 60 people in their network to be certified has group psychotherapists much in the same way that I act trains people and for those who work in these settings a significant reduction in aggression and critical incidents from 50 down to 54 from 67 and the continuous quality improvement opportunities contributed to early problem spotting and promoted reflection and reduced the risk of therapist drift which happens in any kind of ongoing therapy so that's the last thing that I want to say and welcome David back up on the stage and we'll have an opportunity for questions and comments and thank you for your attention so far [Applause] don't hesitate we want to encourage as much participation as possible we'll share this up here my question is about dr. Tiller's Clarke's reference to both having outcome measure and process measure actually my understanding of outcome measure is checking how the client is doing based on some criteria and also it not only shows if the client is improving or deteriorating it also reflects me how the practitioners are doing but the presense our process measure like a like a routine process or routine check off during each session when we do this aren't we running the risk of kind of compromising the contingent whiteness and spontaneous the treatment because it looks to me like a list of check off or I'm talking about process measure going through a list of checkups to see if I as a psychotherapist or psychologist have done things regarding my treatment plan or even the treatment mode I am actually employing and then it looks like a report to my supervisor then just having a feedback of what's going on in during them actually therapy because already the outcome measure gives us that feedback or maybe my understanding of actually process measure is different than what you meant but that's my understanding yes difference between outcome measure and process lesion yeah so I think I didn't explain what I meant by process measure clearly enough because the process measures we have are nothing nothing really to do with the therapists behavior they are measures to do with psychological constructs that you're trying to change in therapy so in if you're treating someone with social anxiety there are lots of different fears that people have they might fear that people think they're stupid they might fear that people notice they're blushing and think they're weird because of that they might fear that people think they're inadequate in some way and so we track those beliefs as we go through therapy and what we find is that you you may focus your early therapy on a small number of beliefs and change them quite a lot but if you are giving regular measures covering a whole range of beliefs you can spot that you've got some other beliefs that you haven't really touched in therapy there haven't been mentioned much by the patient but they are still strongly believed and so we focus on on those and the reason we do this is because of a painful experience in our first-ever randomized controlled trial so it was a trial of treatment of panic disorder overall the results were good but when we finish a clinical trial we always do what we call a post-mortem and we take the whole team we lock ourselves in a way in a room for three or four days and we just look at those patients who didn't improve as much as we wanted them to and we try and learn from them we look at the tapes we look at our notes and things and what we found is that we were giving one of these measures a wide range of beliefs at the beginning and end of treatment for research purposes but the therapist hadn't looked at it and we had a couple of patients who really we got down it took us right down to the wire to get them panic free and two of them didn't get panic free and there weren't we looked at this measure we felt it at the beginning of therapy they had a couple of beliefs one of which is I'm dying which never came up in the therapy sessions and so it was never challenged but if we've been looking at the measure we'd never have got into that problem so that's why we we measure visits cognitive therapy a whole range of beliefs and make sure we cover the whole cognitive territory the other thing in social anxiety that you're very interested in is people's focus of attention we know that when your people are very anxious they focus their attention very much on themselves we want to make them more externally focused so they can pick up how other people are responding to them and if we have a measure each week of the focus of your attention we couldn't see whether that's changing or not if it's changing nicely with what we're doing in therapy then we don't bother about it we just let it carry on where we find some patients in whom they're still staying very self-focused even if their beliefs are changing a bit and we know for those we have to work on shifting their focus of attention so it's those sort of processes we're involving but I think Mullen also gave a very good case for tracking the therapeutic alliance which is another process which is central to therapies and you could see quite clearly the tracking that was helpful because you could see from that chart with Peter Pan the therapeutic alliance wasn't all that good and the therapist may not have been entirely aware of that because they had a positive bias but the tracking would help them but I'll infinity I appreciate your question and we're going to make room also for other people's questions that because your question speaks to one of the common resistances therapists have that getting that kind of processed data but the quality of the relationship may feel like it's going to distort the treatment or it becomes self-serving or what else is the patient going to say except yeah you're great but the experience in the field is that people are often are honest on these kinds of measurements than they are face-to-face although I would strongly encourage always trying to do this face-to-face as well and as you heard earlier it's the patient's experience of the Alliance that is most highly correlated with outcome so we want that we want that experience Thank You Peggy hi I'm not sure if this is the right venue to ask the question so if you'd prefer that we maybe take it up after that would be okay too but I'm involved right now in the rural out of health quality standards for anxiety disorders and OCD in Ontario and it's a really great opportunity because with the launch of these standards we have the opportunity to develop aspirational standards but we don't have a mandate to enforce adherence or change and so I'm really curious about any advice you can give we are planning to embed in the standards statements about the use of routine outcome monitoring because I firmly believe that the evidence is there as you've both shown us so convincingly but this is the way we need to go and yet we know it's really difficult for clinicians to embrace this kind of a shift in their practice I think in I hope you had the big advantage of a leadership that was invested in it and could mandate it and we won't have that so I'm wondering if either of you have any thoughts about one or two best ways to kind of if we have the opportunity to make a couple of requests for support around rolling out these standards and making sure this happens if you have any thoughts about one or two best ways we could support this happening well I think you know always the first step must be to to win the agreement of your workforce wasn't it and so you know you won't be surprised for me to say as a cognitive therapist you want to get into the head of the people that are not using the measures try and find out what the block is for them and then you know work with them on that in ayat we have a little role players who sometimes do so you might have some therapists are not that keen to measure the depression every week session or something we might say to the well you know let's take someone who goes to their doctor they've been under a lot of strain a lot of stress and the doctor takes her blood pressure and finds its really pretty high you know 160 over 110 or something Ben says well you know you've been it under a lot of stress it's probably just that they'll come back in a week and we'll see you know what it's like and then it's still way up there and then the doctor says well maybe that's not a transient thing maybe we better to do some 24-hour monitoring and it comes back and it's still high so the doctor then says well actually you know there's a very good chance you'll be dead within ten years at this level of blood pressure unless we deal with something do deal with it so we've got to do something we could work on your diet but I think we need some medication as well the good news is there are lots of medications at work but you know we have to try and find the right one but I don't want to over stress you so I won't take a blood pressure each time just trust me and you know you're both patients look at you is it you're totally mad okay well let's take it into depression so you've got someone who comes along they come up high on the phq and you know it's just there's one episode and and the clinician is seeing them says well you know it sounds like you had a big disappointment at work and things like that but you know hopefully things will be a lot better next week but let's just meet up in any case and it's still up there and you're also coming out quite high on item nine you know you're feeling maybe life isn't worth living at things like that you say well you know I think we need to get into some treatment here this is quite important because actually you know there's a chance you will be dead in the next ten years from your own hand but um I don't want to stress you by measuring things to my child just try out different things and hopefully what gets the right place it's no different is that the reason that I made a comment about the Mercy behavioral health is anticipation of that kind of resistance and what you need is organizational buy-in an organizational culture that is interested in the development of the people to work in in in in the program no one likes to do work that isn't effective it's demoralizing and I think you can begin to make the case for this is going to improve the quality of your patient care and the quality of your professional life and we're going to use the data and a way to make you more effective not in a way to beat you up but in the wrong hands it can be misused and but in the right hands I think it elevates quick care and it will face the quality of life for the providers so I mean that's another really helpful answer isn't it and we mentioned IT systems as well on Monday we're having a conference call all of those are involved in planning NHS provision in mental health and one of the things were going to debate is whether we are going to adjust the funding that we give to different services dependent not on the outcomes they get but on whether they collect outcome data and I think it is very lightly that because we're expanding funding in mental health new commitments that that we will adjust the amount of funding that we give to different areas depending on whether they are able to consistently collect the data without going the extra step and saying you and you need this particular outcome we just say you have to get the data so that's the thing of worth thinking about thank you for all the different information you presented today and I'm interested in outcome measures too but I'm I realised this conference is focused on CBT I'm going to talk about a psychodynamic dimension which is transference because I'm particularly interested in certain people that I've seen in my own patient population over the years which is a complicated collection of people with anxiety disorders but also characterological issues one of which is the good woman pleasing patient and how your outcome measures can measure for those kinds of things because I've seen many people over the years that I could probably write a paper on quote failed CBT because what has happened is the person has played along with the expectations and the treatment and has said thank you very much and has gone on for six months and then tripped into quite a profound depression because some of the interpersonal dynamics aren't being addressed in the CBT model and I'm just wondering what if any things you're looking at that I in terms of the measurement and and and how you control for that when you're looking at outcomes yep so the first thing to say is there are failed cases of every therapy so that's just a fact and I think without kumano Turing you need to look as broadly as possible so it's very good to have some minimum data set you do you agree on and everyone tries to collect it but you mustn't get obsessed with just looking at that you need to look at the patient in the round and our view is that you also need to make sure you measure things that the patient says they want measuring so we often as clinicians measure clinical symptoms most often patients talk about a disability or how it's interfering with their life what they can't do and so it's important to have both of those but the other thing which is very relevant to your point of course is to track people over a long period of time because the goal of psychotherapy isn't simply to get someone better at a particular time it's to keep them well in the long term isn't it that's what we're all in the business for so it's not just the measures you select and making them broad enough but it's also tracking people over the long term and of course if you know someone's had some therapy and they and things go badly wrong in the future you want to know that and you want to know as quickly as possible but I guess at what I was maybe I wasn't as clear about was how you're looking at outcome measures where the patient because they want to make sure you feel good as a good therapist is saying that things are going a lot better than they really are I mean I personally have been tripped up by that myself yes and how you can control for that yeah so in clinical trials of course we have a whole methodology for controlling for that we have independent Assessors and you don't forget a trial published unless you have independent Assessors in anxiety disorders we also have behavior tests that we use a lot so the video I showed of the person with height phobia is an example of that isn't it you can see can you really do things are not much less too dependent on on softball in PTSD you know we try and get people back to the site of the trauma and see how they react emotionally and so certainly an anxiety I think it's pretty important to get people into the real situations that provoke the anxiety and see if they're emotionally change not just a self-report to you as a therapist in the clinic depressions a bit more tricky isn't it because we don't have such a broad-based way of looking at that and so I think it can be an issue so you you sort of address my question about kind of like the social desirability bias but I have another question regarding yeah you talked about you know collecting outcome measurements which I again as everyone has heard is very very important but you talked about collecting them every session and my question to you is more so especially because it's based on self-report what is the susceptibility to testing effects that repeated testing over time especially clients who are coming in you know once a week multiple times a week can you elaborate on that well so I mean most of the measures that we use have a timeframe of a week so there's not suit appropriate to give them multiple times in a week so if someone's coming in more than once you still only give it once a week is the way we do it I think for a lot of these measures there is likely to be some sort of repeated measurement trend isn't there but there are quite a lot of trials of course that are repeatedly giving these things in wait lists as well and you don't see the same sort of big drops so I wouldn't rule them out but for most of the measures it's notes are quite modest effect I'm I'm just curious if you have any advice with respect to bringing outcome measurements to administrators to have that implemented in systems because this is something that I'm doing in my own practice and we're involving some of the outcome measurements that we already take but the individual sessions and the session by session rating isn't happening so how to get buy-in from I guess administrators well I suppose with I out you know we sold the model and we've got the government investment for it and you know that is a key bit of the model and the administrators didn't have a difficulty with that that they they realized quite early on that it's better to have for them to see the data they're not so that worked out well I think if you're working as an individual therapist it's a different issue I guess is that you or your situation yes yeah so that makes it much more difficult if you don't have an IT system to support you doesn't it but I think also you know if you're working in a situation where you know you've got a supportive group of therapists there then we've had some wonderful experiences with with people you know in their mutual sort of supervision sharing their outcome data so the thing that really got me hooked on this was the last of the very big bombs from the IRA so in August 1998 the IRA planted a car bomb in a particular city in northern Domar a message came and it actually identified the location of the bomb wrongly so the police moved everyone to stand around the bomb and then it blew up 31 people were killed 531 died Oh No injured and there was no real expertise in treating PTSD so we were asked to come and train the local clinicians in treating PTSD and we had clinicians from lots of different orientations really wanting to help but you know most of them hadn't got much familiarity with trauma-focused CBT but we did group supervision and every time we did the supervision we put up this sort of outcome measures of each case at the start of treatment and the last time they were seen on the board and we it was just the most wonderfully informative thing because we had an enormous amount of group learning we saw there are some things that really worked well for these particular people some of the beliefs they had about other people in the community or may or may not have been involved in the bombing for example and addressing those really pull down the PTSD well other things didn't but this was a sort of group learning that we got because we all saw everyone's cases and and we heard what each person was doing and it actually produced in a very short period of time a real interest in learning this particular treatment so you know that's what we were training so people had come from very different therapeutic backgrounds but they they moved in a way towards you know acquiring these skills very rapidly because they saw the outcome data on all the cases and some of that I mean I'm very close friends with many of them still and many of them say about it they never really would have learned the power of this treatment approach if they hadn't gone through that exercise because they all knew they were doing things that were useful for patients but what they didn't wouldn't have known is how you know the other therapists treated someone who seemed as tricky but they done so slightly different and they'd really improved a lot so it really is a a journey of wonderful discovery we're an inner group but you can't get it unless you've got the data yeah I mean we use such I mean I think there is one thing probably so the reason we have a session by session outcome monitoring and I opted is because of that bomb so what happened is that there was a concern that one of the nurses in the local psychiatric hospital may have been involved as a bomber and so absolutely no one was going to go to the psychiatric hospital for treatment of their PTSD so there had to be a drop-in center created in the town with actually rather minimal records on people as well and so they would just come in and we'd give them therapy when they came in and we don't know whether they come back next time but we wanted to see what they were doing anything useful so that's why we decided on session by session monitoring and we got data on every one from that and we presented it to a government minister and he gave money to found a trauma treatment center for the next ten years that treated other victims of terrorism so that's what taught me that getting data generates money and getting it every session is the way to get it complete but no I haven't talked about that other bit of the story I'd recommend our article by Whittingham which I've cited in Psych in the journal psychotherapy published in 2018 that talks about the transformation at Mercy behavioral health and they did many of the things that David has referenced they provided feedback they videotaped group sessions they provided supervision on the group sessions then they had subsequent data they saw the improvement in outcome in and they just built on itself in that kind of way bringing people together to learn together and to support one another is integral to this you can't do this very well just in isolation I think we have time for one or two more questions you