An agency contracted with CCNY to provide evaluation services for their implementation of a training program. The training program consisted of a weeklong training aimed at preparing Family Peer Specialists (FPS) for their role. The training program consisted of multiple modules focused on the various systems FPS’s will be expected to interact with, and were led by 2 faculty members (a professional expert and family advocate). The agency had recently completed the training of Cohort 2 and was slotted to train Cohorts 3 and 4 in the coming year. The Agency looked to identify the strengths and weaknesses of the training, including collaboration in the workgroup between faculty and other stakeholders to understand how the training had prepared FPS’s for their role, and where additional training may be helpful. In order to identify the effectiveness of the training and understand areas of strength and weakness, CCNY, together with their evaluator partners conducted a series of focus groups – Faculty, Workgroup and FPS. All of the groups were audio recorded and later transcribed. Following the training of Cohort 1, a similar process was conducted, which yielded controversial results that appeared to target a specific stakeholder group for negative feedback. In addition to the implementation aspects of the focus groups, another goal was to support the agency in strengthening relationships with this stakeholder group and to develop confidence in the evaluation process.
The goals of the focus groups were discussed at the outset of each group, with the purpose, methods and data collection discussed. Questions were provided to all participants to review. To ensure that all participants were able to provide feedback, brief questionnaires were provided to all participants and CCNY staff sent email to all participants asking for additional feedback.
The challenges that arose during the focus groups were a result of interpersonal dynamics of the group rather than of the facilitation or evaluation process itself. In 2 of the groups, the conversations were dominated by 2-3 individuals who seemed to have a limited understanding of the purpose, structure or method of a focus group. Additionally, at least one of these individuals did not appear to be well informed that they were attending a focus group versus a regularly scheduled implementation workgroup. These individuals tended to drive the focus of the conversation away from the purpose and the conversation on hand, making it difficult for the facilitators and other group members to have a productive discussion.
In order to address the interpersonal dynamics, CCNY and partner staff developed a new protocol for focus groups. Although they were unable to implement it, they plan on using it in the future. In future focus groups, CCNY staff will provide a clear explanation of what an evaluator is, what an evaluation can accomplish, and what the purpose, function and format of a focus group is. Ground rules will be established and clarified at the beginning, discussing rules for length of time speaking. The use of the results will also be discussed, particularly in relationship to the process for dissemination.