Electronic Thesis and Dissertation Repository

Thesis Format

Monograph

Degree

Master of Arts

Program

Education

Supervisor

Crooks, Claire V.

Abstract

For community-based organizations that work with vulnerable youth, evaluation measures and activities are important strategies for assessing a program’s impact on youth on different outcomes. However, rigorous program evaluation involving pretest-posttest measures and control trials are impractical to implement in community settings. It is critical for organizations to continuously measure programming efficacy, as it is an issue of accountability, ethical responsibility, and program improvement. Additionally, funders, policymakers, and stakeholders typically require organizations to monitor the effects of programming in their setting to continue receiving support. However, organizations conduct program evaluation under many constraints. There is an emergent need for a feasible tool for community organizations to collect data from their programs in an efficient yet effective manner that captures impactful information about program efficacy. The present study follows the development of a retrospective survey for community organizations to evaluate the Healthy Relationships Plus – Enhanced (HRP-E) program. The purpose of such a survey is to provide a measure for organizations to use so they can engage in ongoing program evaluation when more rigorous approaches are not feasible. An initial pool of items was generated based on HRP-E content and previous Fourth R surveys. Upon the development of the survey, nine experts were interviewed to gain their feedback on the draft. Interview transcripts from experts were coded and used for an inductive thematic analysis to organize, find patterns, and extract meaning from the interviews. Results discuss the major themes from the interviews and provide insight on important considerations for survey development in the context of research with community organizations and youth.

Summary for Lay Audience

For community-based organizations that work with vulnerable youth (youth who might be at risk of experiencing hardships or harm due to environmental or social reasons), having evaluation surveys and activities are important for examining how a program or intervention affects youth in different ways. However, more demanding program evaluation activities that involve participants completing surveys before and after a program (i.e., pre and post surveys) and control trials (i.e., having a group that receives no programming) are impractical to implement in community settings. In work with vulnerable populations, there is a need for program evaluation to be more practical, efficient, and respectful of youths’ time and effort. It is critical for organizations to continuously measure programming efficacy, as it is an issue of accountability, ethical responsibility not to harm clients through programming, and program improvement. Funders, policymakers, and stakeholders often want organizations to monitor the effects of programs in their setting to continue receiving financial support. However, organizations may encounter several obstacles when attempting to evaluate programs. Therefore, there is a need for a feasible measure for community organizations to collect data from their programs in an efficient yet effective way that captures important information about the effectiveness of programs. The present study follows the development of a retrospective survey (i.e., only one-time administration) for community organizations to evaluate the Healthy Relationships Plus – Enhanced (HRP-E) program. The purpose of such a survey is to provide a measure for organizations to use so they can engage in ongoing program evaluation when more demanding research approaches are not practical. Initial survey items were created based on HRP-E content and previous surveys related to the HRP-E. Once the initial survey was developed, nine experts were interviewed to gain their feedback on the survey draft. Interview transcripts were coded and used for an analysis process, called inductive thematic analysis, to organize ideas, find patterns in experts’ discussions, and uncover important themes from the interviews. Results show the major themes from the interviews and provide insight on key considerations for survey development in the context of research with community organizations and youth.

Download

Share

COinS