Risk assessment tools plagued by 'authorship bias,' study finds

9-5-2013 National:

Reported predictive validity higher in studies by an instrument's designers than by independent researchers

The use of actuarial risk assessment instruments to predict violence is becoming more and more central to forensic psychology practice. And clinicians and courts rely on published data to establish that the tools live up to their claims of accurately separating high-risk from low-risk offenders.

But as it turns out, the predictive validity of risk assessment instruments such as the Static-99 and the VRAG depends in part on the researcher's connection to the instrument in question.

Published studies authored by tool designers reported predictive validity findings around two times higher than investigations by independent researchers, according to a systematic meta-analysis that included 30,165 participants in 104 samples from 83 independent studies.

Conflicts of interest shrouded

Compounding the problem, in not a single case did instrument designers openly report this potential conflict of interest, even when a journal's policies mandated such disclosure.

As the study authors point out, an instrument’s designers have a vested interest in their procedure working well. Financial profits from manuals, coding sheets and training sessions depend in part on the perceived accuracy of a risk assessment tool. Indirectly, developers of successful instruments can be hired as expert witnesses, attract research funding, and achieve professional recognition and career advancement.

These potential rewards may make tool designers more reluctant to publish studies in which their instrument performs poorly. This "file drawer problem," well established in other scientific fields, has led to a call for researchers to publicly register intended studies in advance, before their outcomes are known.

The researchers found no evidence that the authorship effect was due to higher methodological rigor in studies carried out by instrument designers, such as better inter-rater reliability or more standardized training of instrument raters.

"The credibility of future research findings may be questioned in the absence of measures to tackle these issues," the authors warn. "To promote transparency in future research, tool authors and translators should routinely report their potential conflict of interest when publishing research investigating the predictive validity of their tool." ..continued.. by Karen Franklin, In the News

Read More of Article...

Article Summary only...

A Multisite Comparison of Actuarial Risk Instruments for Sex Offenders

2003

Introduction:
Four actuarial instruments for the prediction of violent and sexual reoffending (the Violence Risk Appraisal Guide [VRAG], Sex Offender Risk Appraisal Guide [SORAG], Rapid Risk Assessment for Sex Offender Recidivism [RRASOR], and Static-99) were evaluated in 4 samples of sex offenders (N = 396). Although all 4 instruments predicted violent (including sexual) recidivism and recidivism known to be sexually motivated, areas under the receiver operating characteristic (ROC) were consistently higher for the VRAG and the SORAG. The instruments performed better when there were fewer missing items and follow-up time was fixed, with an ROC area up to .84 for the VRAG, for example, under such favorable conditions. Predictive accuracy was higher for child molesters than for rapists, especially for the Static-99 and the RRASOR. Consistent with past research, survival analyses revealed that those offenders high in both psychopathy and sexual deviance were an especially high-risk group. ..Rest of Study.. by Grant T. Harris -and- Marnie E. Rice -and- Vernon L. Quinsey -and- Martin L. Lalumiere -and- Douglas Boer -and- Carol Lang

Read More of Article...

Article Summary only...

Sex Offender Risk Assessment

June 30, 2006

Prepared for: Missouri Sentencing Advisory Commission

Executive Summary
Compared to clinical methods, actuarial risk instruments are a preferred method to discern sex offenders risk for sexual as well as violent recidivism because, unlike clinical practices, they are considered inexpensive, objective and modestly accurate. Scientists argue that risk instruments that employ only static, or historic measures of offender characteristics, rather than dynamic, are certainly sufficient for the purposes of gauging individuals’ likelihood of recidivism. In fact, Harris and Rice (2003:207) contend that dynamic constructs are “unnecessary for anticipating who will recidivate in a given time period”; furthermore they state that “very accurate statements about the likelihood of another…offense can be based upon knowledge of an individual’s lifetime conduct.” In their view, offender risk scales that incorporate only static information are essentially capturing factors that refl ect a person’s underlying antisocial propensity.

Although there are a considerable number of risk instruments available for corrections officials to utilize, far fewer have been rigorously evaluated. Of those that have, Harris and Rice (2003) recommend that the MnSOST-R and the Static-99 are two of the most “promising” scales for predicting sexual recidivism. An emerging body of work also suggests that the SORAG is quite effectual in terms of its predictive accuracy. Additional empirical research is likely to surface which will provide further evidence of the statistical accuracy of sex offender risk instruments.

For the remainder of this paper: by Dr. Beth Huebner (Prepared by: Institute of Public Policy, Truman School of Public Affairs, University of Missouri – Columbia)

Read More of Article...

Article Summary only...