Understanding The Quality of A Program’s Implementation


A local organization contracted with CCNY to conduct an evaluation of a three year grant-funded program at various sites in Buffalo, NY. The program was designed to (1) help bridge the gaps in community services and provide resources needed for academic success, emotional and social development, physical development, and creative self-expression for youth served and (2) aid in the accessibility, availability, and dissemination of curriculum-based target  programs and high yield activities designed to help youth succeed in school and life. The organization asked CCNY to design and execute an evaluation plan that qualitatively and quantitatively analyzed the program’s implementation and outcomes, which also aligned with the New York State (NYS) Department of Education’s Evaluation Guidelines.


As the focus of the evaluation was program implementation, we completed bi-annual observations of each after school program at all locations.  Observations allowed our Research and Evaluation team to give feedback to program directors about program implementation successes, challenges, and quality improvement opportunities.  Surveys were also utilized as an instrument to determine members’ and parents’ views on the program, both at the start of the school year and again at the conclusion. Our team worked with all program staff to determine recommended times of survey distribution.  Data collection was completed by program staff before being delivered to the evaluators.


CCNY’s Research and Evaluation team worked with program staff to create strategies for addressing challenges, and providing quality improvement ideas to ensure complete data collection for future years.  Additionally, maintaining an open and continual line of communication ensured expectations and deadlines were met.  The observations and survey data were analyzed quantitatively on an aggregate level to demonstrate successes and barriers with implementation, and impact on students.  A yearly review discussing the results for each program was provided to the program director, and was submitted to NYS Department of Education.  The review will aid the customer in future discussions of program implementation for future years and continued quality improvement.