RSI has more than 20 years of experience conducting program evaluations for courts that want an independent, external assessment of their ADR programs. When we develop an evaluation plan, we not only call on that experience, but also use what we have learned through our long history of designing and administering ADR programs to help us better understand what to measure for each program, how to measure it, and what to do with that information. This has provided courts we serve with a more complete and nuanced picture of how their programs are functioning and what they can do to improve.

Our evaluations have ranged from large, multi-year assessments of all aspects of a program to targeted analyses of particular program processes or outcomes. In addition, we provide analysis and reporting on previously collected data.

Our Approach

Quality evaluation is intrinsic to the way RSI approaches court ADR. While we are passionate about the value of ADR to courts, parties and society more broadly, we are just as passionate about courts doing ADR well. We understand that every court ADR setting is different, and resources for evaluation are limited. That's why our approach to evaluation services is collaborative and tailored. We work with courts and other stakeholders to identify program goals and determine the key indicators to measure. We customize the evaluation plan to fit with the program’s needs and its financial and staff capacity.

Our Areas of Work

We have designed evaluation systems and conducted evaluations in a large variety of settings: courts, state executive branch, community mediation centers and more. Our work has included programs for the following:

• Online dispute resolution
• Child abuse and neglect
• Debt and small claims
• Eviction
• Family
• Foreclosure
• Juvenile justice
• Community mediation
• Public policy dispute facilitation
• Large civil cases

RSI's Evaluation Process

Engagement and Understanding

RSI’s evaluation system design often begins with a question: What do you want the ADR program to accomplish? While seemingly simple, articulating an answer to that question forms the foundation for the evaluation. That’s why we work closely with courts and stakeholders to articulate program goals and determine information needs. During this part of the process, we determine the goals of the evaluation, find out how the program works and what the potential issues are, and learn what data may be available.

Evaluation Plan

We use what we learn from our engagement with courts and stakeholders to develop the evaluation plan. At this time, we work with the court to finalize the evaluation questions and to design the process for collecting the data needed to answer them. Our evaluation designs have focused on minimizing cost and staff time through strategic use of technology. We have found that the more we can simplify data collection processes, the more likely it is that data will be collected and used.

This is also the point at which we design the data collection tools. Evaluations are not all the same, and different methods and instruments may be needed to capture the required information. After consulting with the court, we develop the tools needed, such as surveys, interview protocols, spreadsheets or observation protocols. Once the tools are developed, we ask that the court review them to be sure that we have used appropriate language and have not included data that it isn’t possible to capture.

Data Collection

Data collection starts with a pilot period during which the data collection processes are tested. Once any kinks are worked out, the data collection period begins. During this time, RSI regularly checks the data to be sure that there are no problems, and makes adjustments if need be. The court’s responsibility at this time is to provide the data.

Data Analysis and Interpretation

After data collection is complete, we start the analysis. In this analysis, we examine the data through the lens of how interconnected issues may affect our conclusions, such as the manner in which cases reach mediation, the presence of lawyers or the time allotted to mediate.

Reporting and Recommendations

RSI tailors the evaluation report to the needs of the court. All our reports include actionable recommendations that courts can use to build on their programs’ successes and to address their programs’ weaknesses.

We know that, by and large, people are too busy to read long reports. Therefore, although we do provide full-length reports for our large projects, we also determine the best method to communicate evaluation findings to judges, program staff and stakeholders. These have included short summaries of the evaluation, infographics and oral presentations.

Implementation

This part of the process is up to the courts. Courts that have implemented the recommendations that result from our evaluations to improve their programs have seen improvements in program use and participant assessments of their experience. Others have been able to expand their programs by using our evidence of their programs’ success. 

Program Evaluation Examples

Evaluation of Online Dispute Resolution Programs

RSI conducted evaluations of online dispute resolution (ODR) programs in a Texas small claims court and a family court in Michigan in partnership with the University of California, Davis, and The Pew Charitable Trusts. These evaluations examine the impact of ODR on access to justice, participant experience and efficiency.

Multi-Site Evaluation of Foreclosure Mediation Programs

RSI conducted an evaluation of eight foreclosure mediation programs in Illinois. We assessed each program individually on a number of performance indicators, and all were compared with one another to determine whether program design had an effect on program performance. We then made recommendations for improvement for each program and developed overall program design recommendations for any court developing a foreclosure mediation program. The evaluation of the eight programs built upon an initial evaluation of the first six programs to launch. That evaluation pointed courts to areas for improvement, including lowering barriers for homeowners to enter the program and providing pre-mediation counseling services to help homeowners understand their options and complete their loan modification packets. Courts that changed their processes in response to recommendation significantly improved their effectiveness.