Study In Process: Inter-Rater Reliability of the J-RAT
Informed by the larger literature and research in our field, Stetson School uses instruments that we have developed to assess risk for continued problematic sexual and non-sexual behavioral difficulties, including risk for continued sexually abusive behavior for those students who have previously engaged in sexually abusive behavior. Although we have developed a series of risk instruments for different age groups and levels of cognitive development, the “parent” instrument is the Juvenile Risk Assessment Tool (J-RAT), an instrument designed to assess the risk for continued sexually abusive behavior.
Although it is difficult and complex to conduct research on any risk assessment instrument, since 2009 Stetson School has been engaged in a research project designed to measure the “inter-rater reliability” of the J-RAT. Inter-rater reliability refers to the ability of an instrument to produce the same result when used by different evaluators (raters) in the assessment of the same client. High inter-rater reliability is important in the development and use of any evaluation tool as it suggests that the tool is sufficiently well developed and standardized in its language and method of measurement so that it reliably produces the same result, regardless of the evaluator using the measure. In addition, the study was designed to help pinpoint weaknesses in the design of the J-RAT, and help produce a stronger instrument.
The study remains on-going, and the J-RAT has now been re-designed based on the first stage of the study and data analysis, and is now in use as Version 4 (V4). The intention of the re-design was to continue building the instrument to ensure strong inter-rater reliability. The next and current stage of the research project is to now examine the re-designed instrument for its inter-rater reliability. The extended study will use the V4 version of the J-RAT to assess 30 student cases using four evaluators, including one independent external evaluator, who will each independently examine the same cases in order to establish the level of reliability (that is produce the same results). We expect to complete the study by early 2013.
Although it is difficult and complex to conduct research on any risk assessment instrument, since 2009 Stetson School has been engaged in a research project designed to measure the “inter-rater reliability” of the J-RAT. Inter-rater reliability refers to the ability of an instrument to produce the same result when used by different evaluators (raters) in the assessment of the same client. High inter-rater reliability is important in the development and use of any evaluation tool as it suggests that the tool is sufficiently well developed and standardized in its language and method of measurement so that it reliably produces the same result, regardless of the evaluator using the measure. In addition, the study was designed to help pinpoint weaknesses in the design of the J-RAT, and help produce a stronger instrument.
The study remains on-going, and the J-RAT has now been re-designed based on the first stage of the study and data analysis, and is now in use as Version 4 (V4). The intention of the re-design was to continue building the instrument to ensure strong inter-rater reliability. The next and current stage of the research project is to now examine the re-designed instrument for its inter-rater reliability. The extended study will use the V4 version of the J-RAT to assess 30 student cases using four evaluators, including one independent external evaluator, who will each independently examine the same cases in order to establish the level of reliability (that is produce the same results). We expect to complete the study by early 2013.
