IRVINE Case Assistant

Overview

The IRVINE Case Assistant is an AI-driven tool designed to enhance probation officers' understanding of their cases. It operates within the context of the Risk Needs Responsivity (RNR) framework, a method widely employed in probation that evaluates individuals based on risk levels, personal needs, and the potential for effective interventions. Equipped with a sophisticated, user-friendly dashboard, the IRVINE Case Assistant enables probation officers to swiftly grasp the essence of a case as reflected in the often daunting collection of case narratives. This breakthrough empowers officers to make well-informed decisions, fostering stronger relationships and advancing rehabilitation efforts with the goal of reducing recidivism.

Problem and Motivation

The probation system faces significant challenges. A study by The Pew Charitable Trusts has shown that extended probation terms may not necessarily enhance public safety, with many low-risk individuals remaining under supervision for prolonged periods. This over-supervision can lead to worse outcomes, representing an ineffective use of resources.

We partnered with a State Probation Agency to understand the challenges they face in serving their clients. The Agency is a critical component of community-based supervision and has been facing significant challenges, namely the large number of people currently on probation (“clients”) with limited resources and probation officers to manage them.

Within the agency, there are approximately 70,000 active cases with about 1,000 probation officers to attend to them. With high average caseloads, officers can easily be overwhelmed as their ability to provide the right intervention at the right time for the right client is impeded by their limited time and resources. This high average caseload holds true across the United States: according to the US Department of Justice, there are an estimated 3,890,400 adults under community supervision with around 46,000 probation officers currently employed in the United States.

Each case generates a great deal of documentation: dozens of free-form case notes known as "narratives", intake forms, regular assessments, and more. Officers currently spend hours manually sifting through this documentation to find meaningful information and understand the case. This time could be better spent with their clients, providing human connection, valuable guidance, and emotional support.

Privacy and Security

Privacy and security have been our highest concern throughout, and our data and models are all isolated behind multiple layers of security. Our data comes from our partner Agency and consists of termination data, regular assessments, behavioral change data, and, most crucially, free-form narratives. Our Agency partners exported this data from their systems, removed personally identifiable information, and sent it through a secure file transfer process. This is highly sensitive data that still contains personally identifiable information due to the free-form nature of narratives. Because of this, we have not made any part of our demo publicly available.

Product Overview

The Case Assistant provides probation officers with a single dashboard that serves various insights concisely and with links to source data. We developed each feature of our product at the request of our Agency partners.

For each case, the Case Assistant first provides a Case Overview that consists of two concise sections: the first paragraph captures the overarching tone of the case, and the second emphasizes key outcomes and, where possible, offers actionable insights. Because each of these is grounded in the case data, the Case Assistant is able to provide citation links to the originating narratives where the supporting evidence was sourced.

The Case Assistant also provides an interactive timeline of the case's tone. Our partner Agency asked for a helpful visual to quickly gauge the trajectory of the case.

Under the Insights section, we further provide five elements that each consist of multiple dimensions. These five elements enrich and inform probation officers' rehabilitation strategies. Some examples of these include:

• responsivity factors such as employment status, housing security, and access to transportation;
• details regarding the client's mental health, substance abuse, and past criminal history;
• treatment tracking, which summarizes the client's willingness to engage in the program and their successes or lack thereof;
• and more.

For each dimension, the Case Assistant also provides proper citations. When a probation officer clicks a citation, it brings them to the corresponding original narrative along with a short generated commentary that summarizes that particular narrative.

Methodology

The Case Assistant is grounded in narratives, which we work with using state of the art large language models.

Each feature has a detailed narrative-focused prompt grounded in Agency terminology and domain knowledge. We worked with our Agency experts to get the substance rights and built the prompts using current and rapidly evolving prompt engineering techniques.

The Case Overview and interactive case tone chart are each derived from a set of condensed narrative assessments. The Insight summary components are built from source narratives because the condensed assessments are not granular enough.

Evaluation of generative models is challenging. We lack labeled data and much of what we are analyzing is subjective. Our approach to evaluation consists of 1) generating citations that support the assertions made, and 2) when the model generates an assertion without a supporting citation, disregarding the assertion or noting that we lack sufficient evidence to draw an inference.

Value Delivered

We presented our product to both a group of probation officers and data analysts from our partner Agency, and a group of key decision makers there. They noted that the Case Assistant provides remarkable precision on important case details and synthesizes insights from different sources. Furthermore, we provide systematic oversight that was not previously possible, by helping to identify "loudly silent" areas within a case. This will also serve data analysts and data stewards within the Agency, aiding in long-term data quality enhancement.

Next Steps

We would like to further refine the complex prompts used in the Case Assistant. In addition, we would like to add another LLM-driven evaluation step to identify and eliminate suspect results. From a feature standpoint, a conversational front end is a logical next step.

In addition to our Case Assistant code base, we are also delivering a report to our Agency partners with recommendations designed to prepare the Agency to make use of cutting edge AI and ML techniques.

Acknowledgments

Our work here would not have been possible without the partnership of the Agency, who provided both our data and probation domain expertise. Their feedback throughout the project was invaluable.

We are also grateful to our capstone professors Joyce Shen and Kira Wetzel for their guidance and advice.