Development of a Pretrial Risk Assessment Tool

Transcription

1 San Francisco Pretrial Diversion Project Development of a Pretrial Risk Assessment Tool Submitted by: Brian Lovins brian.lovins@uc.edu Lori Lovins lori.lovins@uc.edu Correctional Consultants Inc. November 5,

2 Project Introduction Correctional Consultants Inc. was contracted by the Crime and Justice Institute to assist the San Francisco Pretrial Diversion Project (SFPDP) in the development of a pretrial risk assessment tool. The SFPDP was established in 1976 to serve individuals charged with misdemeanor or non-violent offenses who were awaiting trial. SFPDP uses rehabilitation, education, and community service strategies in lieu of jail detention. Furthermore, the program provides offenders with the opportunity to have their case dismissed if they successfully complete a PDP program. The SFPDP is interested in the development and implementation of a pretrial risk prediction tool to assist in the selection of appropriate candidates for the programs. Their desire is to adopt a pretrial tool that will assist in selecting cases where further detainment is necessary versus cases where release to the community may be a safe alternative. Data from three of the SFPDP programs were made available for development of the tool. These include: Own Recognizance (OR), Supervised Pretrial Release (SPR), and Court Accountable Homeless Services (CAHS). The following report provides a brief background on the use of pretrial assessments as well as a description of the data used to develop the SFPDP pretrial risk tool, how the instrument was developed, and predictability of the tool. Finally, recommendations for implementing the tool are provided. Pretrial Risk Assessment Tools The purpose of a pretrial risk assessment tool is to assist courts in predicting the likelihood that a defendant will fail if released to the community before disposition of his or her case (Summers and Willis, 2010). Failure is typically determined by either failure to appear (FTA) for a scheduled court date or by arrest for criminal behavior prior to case disposition. Hence, generally these tools examine both flight risk and risk for further criminal behavior. One 2

3 goal of a pretrial risk assessment is to standardize recommendations about pretrial release so that these decisions are less subjective and more consistent (Cooprider, 2009). A second goal is to maximize the success rates of pretrial releases. This requires that a maximum number of defendants are released without compromising FTA rates or community safety (Summers and Willis, 2010). Notwithstanding issues of crowding and decreasing budgets, the presumption that defendants are assumed innocent until proven guilty underscores the importance of maximizing successful releases (Lowenkamp, 2008). In a study of more than 500,000 cases processed through the Federal Pretrial Services System, several risk factors were deemed predictive of risk of pretrial failure: the nature of the pending charges, criminal history, community supervision at the time of arrest, history of FTA, history of violence, employment stability, residential stability, and substance abuse (VanNostrand and Rose, 2009; Winterfield, Coggeshall, and Harrell, 2003). Levin (2007) found that jurisdictions that used quantitative pretrial risk assessment tools had lower FTA and re-arrest rates as well as fewer problems with jail overcrowding. Thus, more jurisdictions appear to be considering the use of pretrial risk assessments. Study Methods Sample Data for the development of the SFPDP pretrial risk assessment were provided by SFPDP. Data were cleaned and recoded by staff from the Crime and Justice Institute (CJI) at Community Resources for Justice before being sent to Correctional Consultants Inc. (CCI) for analysis 1. Data were provided for three of the SFPDP programs: Own Recognizance (OR), Supervised Pretrial Release (SPR), and Court Accountable Homeless Services (CAHS). There 1 All identifying information was removed from the datasets and all datasets were password protected to protect SFPDP participant identity. 3

4 were two datasets provided on the SPR program a more recent version (2008 to 2011) containing fewer variables and a 2007 version with more comprehensive information compiled by a former SFPDP staff member. Each dataset varied in the items included and the timeframe in which the data were collected. In a review of these data, it was determined that the OR and CAHS datasets would not be used in the development of the pretrial tool and that the development of the pretrial tool would be concentrated primarily from the SPR 2007 dataset. There were several reasons driving the decision to use the SPR 2007 dataset. It contained the most comprehensive information since in contrast to the OR program that collects limited data points, it includes data needed to make decisions about more intensive supervision. Since it serves SFPDP to have one pretrial tool that can be administered at the bail hearing for OR decisions, but can also be applied to defendants if they are not eligible for the OR program but might be better suited for more intensive supervision through SPR or CAHS, it was determined that the most robust data would be the most useful for building the comprehensive pretrial tool. As mentioned, the range of data points collected for the OR program are limited. These defendants are typically seen by the bench and a determination made at the time of the initial hearing. Very limited information is available at that hearing and the data collected did not provide enough items to build a comprehensive tool. Finally, the CAHS dataset had a wider range of variables available for the development of a pretrial tool, but not the same variables as the SPR 2007 data. Moreover, the CAHS dataset did not differentiate between FTA and new arrests. Given these reasons, it was determined that the best option was to use the SPR 2007 data. As mentioned previously, two SPR datasets were made available; one from 2007 and a second more current database including offenders being served between 2008 and While 4

5 the recent dataset more closely reflects current program participants, the 2007 database was selected as it contained the most comprehensive measures for the development of a pretrial tool including both FTA and new arrests. Hence, limitations of the data used to develop this tool include the age of the current database the data is five years old and the population that is being served may have shifted. Secondly, the analysis is based on the data provided, with no means to test the accuracy of the data. Nonetheless, the data selected for use in the development of the tool is the most robust of the data provided, and while base rates of reoffending/fta may change, predictors typically do not change over time. Hence, while there are limitations to the data used to construct this instrument, the robustness of the measures of risk outweighs the limitations noted. Variables The three outcome variables that were used in the development of the SFPDP pretrial tool were failure to appear, new arrest, and unsuccessful termination (failure for any reason). A range of potential predictors were included in the analysis to determine what was predictive of failure. The following provides a review of each of those variables: Table 1 Data points provided in the SPR dataset Current charge Under current supervision Pending case(s) at time of arrest Violent arrests past year Any misdemeanor Active community supervision Outstanding warrants Violent arrests Violent arrests past 5 years Any felony 5

6 Violent convictions Domestic Violence/Court Order Robbery Bench warrants (past year, past 5 years) Current substance abuse treatment Assault/Battery Weapons Charge Felony convictions (past year, 5 years) Previous substance abuse treatment Housing Employment Analysis The development of the pretrial tool was completed in multiple stages. Bivariate analyses were conducted to determine the variables correlated with any failure, failure to appear, and arrested for a new crime. Next, the variables that were identified through the bivariate relationship were combined in a stepwise logistic regression model to determine if any of the variables could be eliminated. In addition to examining the additive benefit of each variable, logistic regression analyses were conducted to ensure that the variables selected were not highly correlated with gender or race. Upon identifying the predictors of recidivism, each item was converted to a 0/1 scoring system similar to the Burgess Scale (Burgess, 1928). While recent risk assessments use a more robust weighting system, it was decided that the simplicity of a simple additive scoring process provided more face validity to the assessment while still providing a valid assessment of risk (Nuttall, Barnard, Fowles, Frost, Hammond, Mayhew, Pease, Tarling, & Weatheritt, 1977). Once the pretrial tool was developed, Receiver Operating Characteristics (ROC) analyses were conducted to determine the specificity of the model. The ROC analysis measured by the Area Under the Curve (AUC) provides a measure of accuracy by balancing false positives with false negatives to determine how well the tool predicts over 6

7 chance. An AUC of.500 would suggest that a tool was not able to classify a person any better than chance. Schwalbe (2007) found that AUCs ranged from.532 to.780 with an average AUC of.640 across 18 risk assessments examined. Findings Table 2 shows the demographic breakdown of the sample used for the analysis. There were a total of 335 participants in the sample. Nearly three-quarters of the sample was male. Approximately two-thirds of the participants were persons of color versus white. Most participants were between the ages of 33 and 45, followed by the age categories 18 to 21 and 46 or older. Table 2 Demographics of the 2007 SPR sample Gender N % Male Female Race White Persons of Color Age and older To determine the predictors of unsuccessful termination, the bivariate relationship between defendant characteristic and outcome was examined. Recall that outcome variable unsuccessful termination was measured based on either FTA or arrest for a new crime. Chisquare analysis determined whether differences in outcome were significant. Table 3 provides a review of the bivariate relationships. Differences in failure rates ranged from 14 percentage points to 32 percentage points. All items reached significance, suggesting that offenders that 7

8 held each of the criminal history characteristics were significantly more likely to fail pretrial release. Offenders that had substance abuse issues, were employed at arrest, and lived in stable housing were significantly less likely to fail pretrial release. Table 3 Predictors of new arrest/failure to appear FTA/New Arrest Chi square Sig level Pending case at arrest No 41.9% p.05 Yes 55.6% Felony probation 2 or more violent arrests w/in 5 years 2 or more misdemeanor convictions w/in 2 years No 43.0% p.01 Yes 75.0% No 43.5% p.05 Yes 66.7% No 42.7% p.05 Yes 63.9% Substance abuse problems Yes 40.0% p.001 No 67.2% 2 or more bench warrants w/in 2 years No 41.2% p.01 Yes 63.2% Employed at arrest Yes 39.2% p.05 No 52.8% Stable residence Yes 40.9% p.01 No 61.2% While there are several methods to create a risk assessment based on the identified predictors, a Burgess Scale style is often favored over more technically complicated models that weight items. This is due to the simplistic, straightforward scoring. Table 4 provides the scoring for the SFPDP pretrial tool. 8

9 Table 4 Scoring for the Pretrial Tool Pending case at time of arrest Currently on felony probation 0 = No case pending 1 = Pending case (any level) 0 = Not on probation for a felony 1 = On probation for a felony 2 or more violent arrests past 5 years 0 = 1 or fewer violent arrests 1 = 2 or more violent arrests 2 or more misdemeanor convictions past 12 months Substance abuse problem 0 = 1 or fewer misdemeanor convictions 1 = 2 or more misdemeanor convictions past 12 months 0 = No substance abuse issue 1 = Has a substance abuse issue 2 or more bench warrants in past 2 years 0 = 1 or fewer bench warrants in past 2 years 1 = 2 or more bench warrants in past 2 years Employed at time of arrest Stable housing 0 = Employed 1 = Not Employed 0 = Has stable housing 1 = No stable housing As noted in Table 3, there are 8 items with a 0 or 1 scoring for a total of 8 points. Each item is defined through specific scoring rules. Chart 1 provides a histogram for the number of defendants by score on the tool. The majority of offenders scored a 2 on the instrument, followed by a score of either 1 or 3. A small proportion of offenders scored more than 4, with none reaching the top of the scale. Chart 1 9

10 Histogram of defendants by pre-trial score Table 5 provides the number of defendants by risk category. As noted, 58.7 percent of the sample is identified as low risk by the pretrial tool. In contrast, 6.5 percent of the sample is identified as high risk. Table 5 Results from the pretrial tool analysis Sample % of sample Low Risk (0-1) Moderate Risk (2-3) High Risk (4-8) Chart 2 provides failure rates separated by low, moderate, and high risk. Low risk was determined to include a 0 or 1 score, moderate risk a 2 or 3 score, and high risk a 4 thru 8 score. 10

11 Based on these results, 32.1 percent of those that were low risk were terminated for any reason (new crime/fta), 58.6 percent of the moderate risk defendants were terminated, and 90.9 percent of the high risk defendants were terminated unsuccessfully. While the rate of failure for any reason was relatively high, the rate of new arrest for defendants was significantly lower. Low risk defendants were rearrested during pretrial 8.7 percent of the time, moderate risk were rearrested 11.2 percent of the time, and high risk were rearrested 18.2 percent of the time. In contrast, the rate of bench warrants was relatively high. Low risk defendants were issued a bench warrant 23.5 percent of the time, moderate risk were issued a bench warrant 48.3 percent of the time, and high risk defendants were issued a bench warrant 72.7 percent of the time. Additionally, Receiver Operating Characteristics (ROC) analysis was conducted to determine the Area under the Curve (AUC) for the ROC analysis. The AUC is.670 for any failure. Chart 2 Rates of failure by risk level 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% New Arrest FTA Any Reason Low Mod High One of the necessary steps in developing a pretrial tool is to examine the effectiveness of such a tool to predict for subpopulations of the larger group. In this case, we are most interested in the pretrial tool s ability to predict failure by gender and race to ensure that women and people of color are not over classified. An example of over classification of a subgroup would be if the 11

12 tool tended to classify larger proportions of a subgroup as a certain risk level (e.g., a significantly higher percentage in the high risk group). Table 6 provides the distribution of risk by gender and race. As noted, males were slightly more likely to be lower risk than females. 2 As for the differences in race, the pretrial tool separates whites and persons of color almost exactly into equally proportioned groups. Table 6 Distribution of risk by race and gender Gender Low Moderate High Male 151 (61%) 80 (32.4%) 16 (6.5%) Female 45 (52.3%) 35 (40.7%) 6 (7.0%) Race White 60 (60.6%) 32 (32.3%) 7 (7.1%) Persons of color 128 (60.1%) 71 (33.3%) 14 (6.6%) The next two charts demonstrate how the pretrial tool predicts failure by gender and race. Chart 3 shows a breakdown in failure rates by gender. As indicated in the chart, the scale is predictive for both males and females, with fairly similar failure rates between the genders. Low risk females have a slightly lower failure rate (31% versus 33%); yet moderate and high risk females fail at slightly higher rates (58% versus 63% and 88% versus 100% respectively). The AUC for males is.658 and for females is.707. Chart 3 2 While the differences between males and females do not reflect general findings that women tend to be lower risk, it was found that a larger proportion of females were released in the OR setting and could explain why more higher risk females were included in the SPR program. 12

13 Failure rates by risk level for males and females 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Male (AUC =.658) Female (AUC =.707) Low 33% 31% Mod 58% 63% High 88% 100% Chart 4 examines failure rates by race. Again, the chart suggests that the items predict well for whites as well as persons of color, as indicated by the differentiation in failure rates between those scoring low, moderate and high within each group. However, failure rates are higher in each of the three risk categories for persons of color. This suggests that while the base rates of pretrial failure are higher for persons of color, the instrument demonstrated predictive validity for both whites and persons of color. The AUCs for whites and persons of color are almost exactly the same, suggesting that the tools identify defendants between low, moderate, and high risk equally well across race. Chart 4 Failure Rates by Risk Level for Whites and Persons of Color 13

14 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% White (AUC =.667) Persons of Color (AUC =.671) Low 28% 34% Moderate 53% 62% High 86% 93% Comparison with National Averages Questions often asked regarding pretrial release include: what is the industry standard regarding the type of defendants released and what are typical failure rates? These questions are often difficult to answer because of the diverse environmental, cultural, and political contexts. The systems that offer pretrial services often do not publish their failure rates or they fail to publish the types of offenders that are held in custody and not released. Some systems have broader release policies while other systems have presumptive hold policies that result in significantly fewer (and presumably) lower risk defendants being released. Moreover, socioeconomic forces have some impact on failure rates, especially when examining failures to appear, due to larger portion of defendants unemployed, homeless, and without services. Ultimately, one of the most telling factors associated with higher rates of pretrial failure is the length of processing time between court dates. Failure rates climb significantly when the time between release and court date extends beyond 1 month, increasing from 32 percent failure 14

15 within 1 month of release to 67 percent for those defendants that had up to 3 months pre-release time (Cohen and Reaves, 2007). With that said, it is helpful to put failure rates in some context. Chart 5 provides a review of the average of pretrial failures from 75 of the largest cities across the United States compared to the average pretrial rates across all pretrial samples provided for this study. 3 As noted, the average failure rates across the country are lower for any type of failure and failure to appear, but are significantly higher than the 2007 SPR sample s re-arrest rates. This trend appears to be true across the samples provided for analysis; the higher than overall failure rates experienced in the pretrial release programs (OR, SPR, and CAHS) are generally driven more by bench warrants than by pretrial arrests. Chart 5 Average national failure rates for pretrial services 70% 60% 50% 40% 30% 20% US Avg Failure Rates SPR 2007 SF AVG Failure Rate 10% 0% Any Type FTA Rearrest Overrides 3 The average pretrial failure rate for SF was compiled across the SPR, OR, and CAHS samples. 15

16 As is true with any tool, there are times when the assessor will not agree with the final risk level produced by the tool based on his or her professional judgment or presence of aggravating circumstances not captured by the tool. It is expected that the final risk level be overridden in cases where staff have determined that the risk of release is too high compared to risk to community safety. While this is not an exact science, it is often suggested that overrides range between 5 and 15 percent of the total number of cases. Reasons for override often range across settings but generally include: specialized mental health needs (defendant presents as actively psychotic, suicidal, or homicidal), actively involved in gang activities, the offense was very serious or heinous per community standards, presents a violent threat to potential victims, has a significant (and recent) history of failure to appear, or has a significant amount of incarceration time he or she is facing. Overrides can be used to move a person up a level of intervention (own recognizance to supervised release) or to hold a person despite that the tool suggests release. Ultimately, override is to be used as a means to protect the community over the likelihood of the defendant to fail. Implementation of the Pretrial Tool While it is important to adopt a validated pretrial tool, it is just as important to ensure that the tool that is selected be implemented appropriately. To implement the pretrial tool effectively, it is recommended that SFPDP complete the following steps: 1. Train all staff that will be conducting the assessments on the purpose of the pretrial tool and the scoring elements. 2. Examine inter-rater reliability for a period of time, as follows: 16

17 a. Have one person gather all the information while another person is observing. Have both parties score out the tool independently and compare the scores. If there are conflicting results, have the two parties discuss the scoring and come to a consensus. Repeat this process for all staff conducting assessments to ensure that the scoring is similar across all staff. 3. Develop strategies for release associated with each level of risk. The following are offered as suggestions and should be modified where appropriate for the resources available to the SFPDP. a. Low risk should be eligible for OR program and/or minimal services. These defendants should be given a court date and reminders sent prior to his or her court date. b. Moderate risk should receive pretrial release services. These services should include referrals to programming, expectations of attending services, and a contract that provides the defendant with the rules and expectations of release. c. High risk should be reviewed on a case-by-case basis as appropriate for prerelease services. These individuals pose a significant risk to fail and if released, should be provided very intensive services to ensure that they attend their court hearings. This should include some aspect of intensive monitoring (GPS, house arrest, etc) and should be mandated to any treatment services identified as appropriate, and all efforts afforded to ensure that the barriers to attending future court dates be removed (transportation, notifications, etc.). 17

18 4. Establish a protocol for overrides of the assessment and monitor overrides to ensure that they do not occur too often (generally more than 15 percent of the time) or too infrequent (less than 5 percent). 5. Collect data for a period of no shorter than 1 month and then get feedback from staff in regards to barriers to accurately collecting information, scoring, or interpretation. 6. Collect data from all participants and track the defendant for any FTAs and/or arrests. a. Ensure that additional data points are collected on violence so when the revalidation is conducted, the scoring can be refined if needed. Revalidation of the Pretrial Tool It is recommended that the pretrial tool be revalidated on the current population once implemented and reliability testing occurs. To complete the revalidation process the following steps should be conducted: 1. Collect a minimum of 300 cases evenly distributed across each program. 2. Track each defendant for Failure to Appear and new arrests during the pre-trial process. 3. Examine each individual item as well as the total score with emphasis on determining if the violence and substance abuse measures need refined. 4. If the items need to be modified or refined based on the revalidation, these changes should be made and staff should be trained on the changes. 5. It is recommended that the tool be revalidated every 5 years to ensure that it continues to demonstrate predictive validity. 18

19 Conclusion The purpose of this study was to develop a pretrial risk assessment for the SFPDP. Data on 335 participants of the SPR pretrial release program from 2007 was used to develop the tool. Eight variables were used to measure risk of unsuccessful discharge, as defined by failure to appear (FTA) or re-arrest while on pretrial release. The eight variables selected for use in the tool are consistent with many of the other pretrial risk assessments developed elsewhere. The use of this tool by SFPDP will help to inform decisions about appropriate placement into the various programs (OR, SPR, and CAHS). Limitations to the use of this tool should be noted. This tool was developed based on a sample from just one of the SFPDP programs (Supervised Pretrial Release). Hence, the tool should be piloted with other program populations. Also, this tool was developed based on data from Hence, piloting should also include use of the tool with the current SPR program participants as an additional test of validity. Re-validating the tool across different program populations and with current participants will help ensure that the predictors hold up and the cutoffs continue to appropriately differentiate between low, moderate and high risk defendants. It should also be noted that the intention of this tool is to predict risk of pretrial release failure. The tool contains few dynamic items; therefore SFPDP programs should be using additional tools to identify defendant needs as appropriate targets for the various interventions. In addition to piloting, next steps are to develop policies and procedures on use of the tool, and include these procedures in staff training. In developing policy and procedures, SFPDP should consider policies around overrides as well as other quality assurance practices that help maintain validity and reliability of the assessment results. This should include on-going periodic checks of inter-rater reliability by supervisors or trainers of the tool. Beyond the pilot 19

20 period, it is recommended that the tool is revalidated periodically so that population changes can be captured and the tool remains a valid measure of risk for pretrial release. 20

21 References Cohen, T. and Reaves, B. (2007). Pretrial release of felony defendants in state courts. Washington, DC: Bureau of Justice Statistics, US Department of Justice. Cooprider, K. (2009). Pretrial risk assessment and case classification: A case study. Federal Probation, 73(1). Levin, D. (2007). Examining the Efficacy of Pretrial Release Conditions, Sanctions and Screening with the State Court Processing Statistics Data Series. Washington, DC: Pretrial Justice Institute. Lowenkamp, C. (2008). The development and validation of a pretrial screening tool. Federal Probation, 72(2). Nuttall, C.P; Barnard, E.F; Fowles, A.J; Frost, A; Hammond, W.H; Mayhew, P;Pease, K; Tarling, R and Weatheritt, M.J. (1977) Parole in England and Wales, Home Office Research Study No. 38. London, Home Office. Schwalbe, C. (2007). Risk assessment for juvenile justice: A meta-analysis. Law and Human Behavior; 31: Summers, C. and Willis, T. (2010). Pretrial Risk Assessments: Research Summary. Washington, DC: Bureau of Justice Assistance, U.S. Department of Justice. VanNostrand, M. and Rose, K. (2009). Pretrial Risk Assessment in Virginia. Report for the Virginia Department of Criminal Justice Services, Richmond. Winterfield, L., Coggeshall, M. and Harrell, A. (2003). Development of an Empirically-Based Risk Assessment Instrument: Final Report. Washington, DC: Urban Institute Justice Policy Center. 21