Is it Rigorous?
Best Practice 8, Provision D, Use of Rigorous Evaluation Methods
What is Rigor?
In teaching this means setting high expectations, while maintaining accessibility and cognitive engagement with content.
In research, this means that every step of a research program is thorough, peer reviews, based on the highest quality evidence and practices. We can think of rigor as a way of determining trustworthiness of a particular piece of research. This eight minute video from Yale University describes rigor in qualitative research, if you are interested in learning more.
In the best practice standards section on this topic (pages 175-178) they give information about how to develop social service research programs, some general methodology, and guidance some of the major barriers. In this Just Data January report, I’m going to focus on rigorous evaluation at three different levels, full program evaluation, using a change management team, and pre- and post-testing (one way of rapid cycle testing).
Full Program Evaluation and Full Program Changes
If you are going to do a full program evaluation well, you really need to bring in the experts: organizations that focus on the courts and Substance use disorders. Or at the very least, use their tools. This list is in no way exclusive, but will get you started:
Some SAMHSA grants have data collection and equity as an allowable or focal activity, consider building evaluation in your grant package
Opioid Treatment Programs can be accredited via a peer review process then certified by SAHHSA. Your treatment providers be able to provide cross training about this process if applicable. Program evaluation may include moving to or using accredited programs
The Tribal Law and Policy Institute (TLPI) has developed a website for Healing to Wellness Courts with numerous evaluation guides, examples and reports. Depending on current funding they may also be able to guide Family Healing to Wellness Courts in program evaluation via technical assistance.
The National Drug Court Resource Center develops tools and resources
The Child Welfare Capacity Building Center developed a video and discussion guide called “Co-creating Race Equity Tools: Practical Strategies for Evaluation and Continuous Quality Improvement Efforts that gives excellent advice
Children and Family Futures provides technical assistance via our grant and many others, the website linked here is just a brief example of some of their work.
Companies that provide evaluation services: Mathematica, NPC Research, and more
The Best Practice Standards suggest an independent outside evaluation should be done about every 3-5 years. If your court is interested in moving in this direction, we’d be happy to guide you in making the leap.
Note from Meghan I can offer one other option to consider if grant funding seems far off or outside of your courts capacity. I once wrote a very long dissertation about spiders, and I can say without question, that finding an appropriate topic is one of the most difficult tasks to take on early in a young researchers career. There are many criminal justice, social psychology, and criminology students who are not current aware of the work family treatment courts are doing, and are equally unaware of how many excellent research questions you might be able to bring them. This is an excellent way to have an expert (professor-supervised) opinion, new energy, and very low cost assistance in designing a way to collect good information and compare it to existing research. University programs in psychology, addiction medicine and criminal justice, are all places to reach out and see about partnerships that might be available with pre-tenure professors, graduate students, and even undergraduate researchers. The process might take some time, but it will be very rewarding for you and the researcher in the long run. I would be happy to chat with you about ideas for this work: meghan.fitzgerald@courts.wa.gov.
Finding a Middle Ground: Local Change Management Teams
For some of our smaller courts or our newer courts this full program evaluation might be out of reach. Instead you can think about making more focused strategic changes a way to evaluate what areas you can work on. All the existing FTCs in Washington completed an NEFTC assessment that helps guide you towards best practice areas to work on. You might consider doing this again as a baseline, and putting together a change management team to decide what will have the greatest impact. Our data resources page from Just Data January 2022 has a NEFTC self-assessment guide that points you to the sections of the standards that are most appropriate. We also have a full example of a change management and are happy to start those meetings off with you as a team.
Something more immediate
Did someone say rapid cycle testing? If you’ve been paying attention to Just Data January 2023, you know this is a highly recommended tactic for something you can start relatively quickly, see results fast, and make high quality changes while you strategize for the long term. In this fast and easy method, you can pick some area of best practice that you know your court is not quite meeting yet. Design a survey to test your baseline of what’s happening, make the change and six months later you give that same survey again. In education we call these pre- and post- tests. These types of surveys in teaching can be knowledge based so you’ll teach a skill after the pre-test, then test again to see if learning happened. And example of this might be: You court decides to have all phase 1 participants do a class about healthy self-coping skills or emotional regulation skills. After participants have had time to implement those strategies, you check and see if they were able to implement them, and which ones works best. Then you can change the curriculum or task as needed. Another option is to do a survey based on thoughts, opinions, or feedback about a particular area. The one thing that is critical in rapid cycle testing is that its specific. Your survey should focus in on a single topic area, and your change should be directly related to that.
