ISSN: 2007-7033 | No. 59 | e1422 | Thematic section: research articles

Positive youth development in community sport:
A program evaluation using the RE-AIM framework

Desarrollo positivo de los jóvenes en el deporte comunitario: evaluación del programa con base en el modelo RE-AIM

Kelsey Saizew*

Jennifer Turnnidge**

Alicia Luciani***

Jean Côté****

Luc J. Martin*****

The need for developmentally appropriate youth sport programs has instigated evidence-informed research interventions, yet intersectoral exchanges with pre-existing community-based sport programs remain rare. As such, this study involved a collaborative evaluation of a community sport program designed for underserved youth using the RE-AIM framework (Glasglow et al., 1999). Data were available to evaluate the programs reach and maintenance; however, the adoption and implementation dimensions required further evidence-informed tools for reliable evaluation. Similarly, whereas the organization deemed their effectiveness to be a critical component of the program, an inadequate amount of data was available to enable its measurement. We provide recommendations for establishing partnerships between researchers and pre-existing youth sport programs and discuss the implications of developing a user-friendly and evidence-informed evaluation toolkit.

Keywords:

community-based program, community partnership, non-profit sport, positive youth
development

La necesidad de desarrollar programas deportivos apropiados para los jóvenes ha motivado a la realización de trabajos de investigación-intervención basados en evidencia; no obstante, las colaboraciones e intercambios intersectoriales con programas deportivos comunitarios ya existentes siguen siendo escasos. De esta manera, el estudio presentado comprendió una evaluación colaborativa de un programa deportivo comunitario diseñado para jóvenes marginados que utiliza como marco el RE-AIM (Glasglow et al., 1999). Había datos disponibles para evaluar el alcance del programa y su mantenimiento, pero las dimensiones de apropiación e implementación requerían herramientas de medición con mayor apego a evidencias a fin de tener una evaluación más confiable. De igual forma, mientras que, para la organización, la eficacia del programa era un componente crítico, no se tenía la cantidad de datos suficientes que permitiera su evaluación. Se formularon recomendaciones para establecer colaboraciones entre los investigadores y los programas deportivos juveniles preexistentes y se discutió sobre las implicaciones de desarrollar un conjunto de herramientas de evaluación que fueran fáciles de usar y basadas en evidencias.

Palabras clave:

programas comunitarios, colaboración comunitaria, deporte sin fines de lucro, desarrollo positivo juvenil

Submitted: March 22, 2022 | Accepted for publication: August 19, 2022 |

Published: September 8, 2022

Citation: Saizew, K., Turnnidge, J., Luciani, A., Côté, J. & Martin, L. J. (2022). Positive youth development in community sport: A program evaluation using the RE-AIM framework. Sinéctica, Revista Electrónica de Educación, (59), e1422. https://doi.org/10.31391/S2007-7033(2022)0059-010

* Master of Science from Queen’s University. PhD candidate at Queen’s University. Research areas involve positive youth development and group dynamics in sport. Email: Kelsey.saizew@queensu.ca

** PhD in Kinesiology and Health Studies from Queen’s University, Canada. Health Education Research Associate in Queen’s Health Sciences at Queen’s University. Research interests involve coaching and leadership in sport and healthcare. Email: jm123@queensu.ca/https://orcid.org/0000-0001-5055-9361

*** Master of Science in Exercise Sciences from University of Toronto. Client Success Specialist at Indeed. Email: alicia.m.luciani@gmail.com

**** PhD. Professor at Queen’s University, School of Kinesiology and Health Studies. Research interests include the examination of the interaction between athletes and their social dynamics (e.g. coaches, parents, peers) for the development of personal assets and excellence in sport. Email: jc46@queensu.ca/https://orcid.org/0000-0002-3242-599X

***** PhD in Kinesiology from Western University, Canada. Associate Professor and Associate Director in the School of Kinesiology and Health Studies at Queen’s University. Research interests involve positive youth development and group dynamics in sport. Email: luc.martin@queensu.ca/https://orcid.org/0000-0001-5336-6119

Challenges facing community youth sport programs

Sport can provide participating youth with a range of positive outcomes, including improved self-esteem, quality peer interactions, and character development (Côté & Fraser-Thomas, 2016). However, although sport can provide opportunities for such benefits, their emergence is not automatic (Coakley, 2011). Indeed, sport programs must be structured appropriately, meaning that the activities undertaken are suitable for a particular age category and skill level and that the social agents within the environment engage in healthy interactions and provide adequate support to developing athletes (Côté et al., 2014). As an example, the Personal Assets Framework (PAF; Côté et al., 2020), which was developed specifically for sport, posits that quality social dynamics (e.g., relationships, cohesion), personal engagement in activities (e.g., practice, play), and appropriate settings (e.g., organizational structures, community size), all contribute to immediate, short- and long-term youth development.

In addition to exploring the mechanisms through which sport can facilitate youth development (Holt, 2016), researchers have also emphasized the need to improve the evaluation of established youth development programs (Toivonen et al., 2021; Weiss, 2016). For instance, research-based programs such as the First Tee (Weiss et al., 2013) and the Sports United to Promote Education and Recreation (SUPER; Danish et al., 2004) were designed to foster the transfer of life skills in sport. Although results from preliminary evaluative research-based programs are promising (Brunelle et al., 2007; Weiss et al., 2013), these programs only evaluated program effectiveness which is insufficient to evaluate the programs impact fully. Recently, an evaluation of the Girls on the Run program also measured the delivery of the program and its curriculum (Weiss et al., 2020). However, thorough investigations into different components of a program are still warranted (e.g., the long-term sustainability). Further, program evaluation in the context of sport remains relatively new and requires further understanding of the difficulties faced by community-based programs -outside of the research paradigm- for youth development programs to be most effective.

Despite the vast number of community-based programs, there remains minimal intersection between researchers and practitioners (Graham et al., 2006). This is important as intervention programs may only be sustainable while they have the necessary resources available to them (e.g., training, personnel; Turnnidge et al., 2014). Specifically, Whitley and colleagues (2014, 2015) outlined the challenges faced when evaluating and sustaining programs for underserved youth, with a main issue being the discrepancy between researcher and community program objectives. These authors noted a lack of communication between researchers and community stakeholders, resulting in poor trust and knowledge translation, especially in marginalized populations. Creating effective partnerships with established community programs requires consistent and transparent communication, which will ensure the pragmatic nature of the research and the subsequent relevance of empirical findings to community stakeholders. Further, researchers have noted the lack of intervention studies that test positive youth development (PYD) program impact beyond internal validity (i.e., effectiveness) and advocate for better integration of theory and application through intervention and evaluation studies (i.e., knowledge translation, Turnnidge et al., 2014; Weiss, 2016).

Effective knowledge translation involves interaction between decision-makers and researchers and results in mutual learning through the process of planning, producing, disseminating, and applying existing or new research in decision-making (Graham et al., 2006). Although a gap remains between evidence-informed interventions and community implementation, knowledge translation efforts focus on creating community partnerships in the hopes that a reciprocal approach will better engage both researchers and practitioners (Benoit et al., 2011; Graham et al., 2006). One approach advocated to facilitate such reciprocal efforts has been to emphasize the community program’s needs as the research focal point (e.g., Verhagen et al., 2013). To best understand these needs, it is important to first evaluate programs to understand their current impact and identify barriers faced by practitioners (e.g., lack of resources, limited time, poor communication, Hoekstra et al., 2018). However, although evaluation frameworks provide insight into a program’s impact on the target community (e.g., underserved youth), traditional efficacy-based evaluation can be insufficient for understanding the many dimensions of program effectiveness and sustainability (Glasgow et al., 1999). Indeed, a recent systematic review by Williams and colleagues (2022) suggests there remains a lack of evaluations which focus on the quality of the program design and their evaluation. The RE-AIM framework was created with this issue in mind and was designed as a multidimensional tool used to evaluate both the internal (effectiveness) and external (reach, adoption, implementation, maintenance) validity of a particular program (Glasgow et al., 1999).

The RE-AIM framework

The RE-AIM model enables the evaluation of five dimensions: (a) Reach, (b) Effectiveness, (c) Adoption, (d) Implementation, and (e) Maintenance. Reach is defined as the absolute participation rates of the program and the representativeness of the individuals the program was aimed for. Effectiveness encompasses the influence of the program for all individuals involved (e.g., athletes, coaches, parents, etc.), pertaining to both positive and negative outcomes. Adoption is the assessment of the delivery of the program, including the proportion of settings, participation, and program resources. This can include adherence to the program principles or the accessibility of the program itself. Implementation is defined as the extent to which the program was delivered as intended. Maintenance is the degree to which the program is sustained, and participant outcomes are maintained over time.

By using these broad and flexible dimensions, researchers and practitioners can collaboratively evaluate the impact of sport programs. Such efforts have been undertaken with multisport service organizations (MSO; Lawrason et al., 2021); however, we were not aware of attempts to collaborate with grass-roots level municipal sport programs. Accordingly, the purpose of the current project was to demonstrate the feasibility and benefit of an intersectoral exchange with a not-for-profit community sport program. In partnership with the NUTMEG program, we collaboratively conducted an evaluation using the RE-AIM framework (Glasgow et al., 1999). The name NUTMEG refers to a skill in football/soccer whereby one player dribbles the ball through the legs of their opponent. We established evidence-informed strategies for evaluating their program’s future impact.

NUTMEG Soccer Program

NUTMEG was established in 2016 and is a not-for-profit soccer organization targeting physical and life skills (i.e., skills that can be applied in contexts beyond sport; Pierce et al., 2016) for underserved youth in Toronto, Canada (NUTMEG, 2018). Importantly, underserved populations, such as low-income families, typically represent the lowest participation rates in youth sport, with financial and time restrictions posing as the most significant barriers (Holt & Neely, 2011). It is a free ‘drop-in’ program that enables families from diverse demographic and socio-economic backgrounds to participate. By providing a more casual ‘drop-in’ setting, youth can attend sessions frequently, or infrequently, as they are without fear of financial consequence or ‘falling behind’ in skills. Informal settings such as these are important as they promote sustained participation in youth sport and thus beneficial developmental outcomes to those who may not otherwise have access (Holt & Neely, 2011).

At the time of this study, NUTMEG provided two separate programs (i.e., co-ed and female only) that ran during the summer (i.e., outdoor season) and fall/winter (i.e., indoor season). Programs are held once a week for two hours, with each session including 90 minutes of activity and 30 minutes of personal development education. The education sessions change weekly and are designed to develop confidence, emphasize equality, and discuss topics such as prejudice and peer pressure using in-house (i.e., coach-athlete and athlete-athlete interactions) and guest speaker programming (e.g., Toronto Police Service, NUTMEG, 2018). Such an approach aligns with life skills research that advocates for the intentional and systematic teaching physical, psychological, and social skills (Camiré et al., 2011).

NUTMEG has six core values that inform all weekly programs, coach training, and education sessions: friendship, fun, respect, equity, education, and teamwork. Programs such as NUTMEG, which focus on value-based programming, have been shown to be socially rewarding while specifically providing an opportunity for at-risk youth to stay on track personally (Bean et al., 2014). Beyond the benefits to youth, NUTMEG also provides volunteer opportunities for adults and older adolescents to engage with their communities. At the time of the evaluation, 19 staff and volunteers worked within the program. No certifications are required to volunteer with NUTMEG, but optional workshops and training are provided. For example, the ‘High Five’ certification is offered to help interested coaches provide positive youth experiences and promote life-long participation (High Five, 2020).

NUTMEG provides a positive and caring environment to play sports for those who otherwise would not have the means to do so. Programs such as NUTMEG that focus on motivating athletes to learn sport and life skills in accessible (i.e., free of charge) and developmentally appropriate settings have been shown to influence the social and personal development of youth (Bean et al., 2018; Gould et al., 2011). Interestingly, whereas the potential benefits of such programs are clear, Martinek and colleagues (2001) caution that simply providing developmental opportunities in sport does not guarantee their transfer to other contexts. Consequently, it is important to evaluate such programs to ensure their success in engaging and maintaining PYD for participating children/youth.

The NUTMEG Evaluation

This evaluation involved a pragmatic approach to ensure the needs of the current program were met (Gibson, 2017). Such an approach allowed data collection to remain feasible for NUTMEG while prioritizing their needs and guaranteeing their input was heard. The NUTMEG staff approached the research group and expressed their desire to have a complete program evaluation conducted. Once the two sides agreed to work together, an initial list of indicators was created and sent to NUTMEG representatives. A meeting was scheduled to ensure the organization’s objectives were met and to answer any questions and concerns from either party.

From the initial meeting, it was determined that the project would involve two phases: (1) an evaluation based on the data that were currently available (e.g., registration, attendance lists, season-end evaluation forms, exit interviews) and (2) the provision of recommendations for data to be collected in the future (e.g., coach training, PYD transfer, physical skill development). Data available included information such as athlete demographics (e.g., age, gender, postal code), athlete perceived engagement and satisfaction of the program (e.g., “How much do you enjoy the [activities] of the NUTMEG program?”), and athlete/coach reasoning for leaving the program (e.g., they moved). The NUTMEG organization received a Royal Bank of Canada (RBC) Learn to Play grant, which required a summary report to be submitted to the agency. The documents provided to RBC assisted the researchers in understanding what data NUTMEG had previously collected and were also used in culmination with RE-AIM requirements to create an overarching list of potential indicators (see Table 1). The following sections summarize the results from the initial evaluation (Table 1) and advance recommendations for researchers and practitioners involved in community-based program evaluation efforts.

Table 1. RE-AIM indicators including the measurements used

Indicator	Measurement	Results
Reach: Participation rates and representation of individuals
Absolute number of participants**	Total number of athletes registered to date/ Total number of youth (5-14 years) in Toronto	155/ 675 545 0.0002%
Absolute number of volunteers**	Total number of volunteers involved in the program	19
Online Impact (website/social media)	Website hits Social Media followers	N/A Facebook: 187 Twitter: 205 Instagram: 99
Representativeness	Demographics (% youth in program)	Single Parent: 14 (9%) Low Income: 27 (17.4%) Aboriginal: 3 (1.9%) Immigrant: 29 (18.7%)
Effectiveness: Impact of program on positive and negative outcomes
Absolute activity hours**	Total number of hours that you offer physical activity per program (# hours)	2
Number and percent increase of hours of physical activity as a direct result of this program**	Reported average daily physical activity >60minutes (# hours increased)	N/A
Youth impact**	Qualitative analysis of how youth are impacted by this program (physical, emotional, life skills, etc.)	N/A
Physical skills transfer (27 responses)	Percent improvement in physical skills before and after program (i.e. dribbling, shooting, passing, ball control, positioning, creativity etc.)	N/A
Athlete perceived importance and enjoyment on physical skills	Mean score (5-point Likert scale)	Importance: important (1.81) Enjoyment: enjoy (1.68)
PYD transfer	Percent improvement in PYD before and after program (i.e. teamwork, sportsmanship, confidence, peer pressure)	N/A
Athlete perceived importance and enjoyment on PYD outcomes	Mean score (5-point Likert scale)	Importance: important (1.56) Enjoyment: enjoy (1.92)
Leadership behaviours	Coach observation– Number transformational leadership behaviours/ Number of other behaviours (% transformational)	N/A
Negative effects	Perceived negative outcomes as a result, direct or indirect, of participating in program (volunteers, athletes, parents)	N/A
Adoption: Assessment of the participation and delivery of settings
Coaches adoption**	Percent increase in coaches ability to deliver quality programming as a direct result of the program Total number of volunteers with formal certification as a direct result of the program	N/A
Coaches perceived adoption**	Percent of people who believe that the training they attended provided skills/knowledge that make it easier for them to increase the quality of the program	N/A
Philosophy adherence	Percent of volunteers/ administrators that are aware of and adhere to NUTMEG philosophy	N/A
Accessibility	Number of locations, website accessibility and ease of use, number of programs offered, travel (km)	Number of locations: 1 Number of programs: 2 Travel avg. (km): 5.74 Travel avg. (min): 10-20 Website Access: N/A
Implementation: Cost and extent to which the program was delivered as intended
Physical Literacy Awareness**	Number and percent of parents with increased physical awareness of physical literacy as a direct result of NUTMEG	N/A
Weekly training plans	Percent volunteers follow weekly training plans Perception that NUTMEG principles are met through weekly training plans (5-point Likert scale)	N/A Friendship: always (1.31) Fun: always (1.08) Respect: always (1.33) Equity: most times (1.84) Education: always (1.38) Teamwork: always (1.27)
Ratio of athletes to volunteers	Average number of athletes per session/ average number of volunteers per session	2.6:1
Program Investments	Resources needed to implement program (total cost of infrastructure, fees, etc. subtracted from total income and funding)	N/A
Facilitators and barriers to participation	Perceptions from volunteers, athletes and parents	N/A
Maintenance: Degree to which the program and participant outcomes are sustained
Participant retention	Number of participants who attend >50 percent of sessions Absolute number of athletes who left program	11 (8.9%) 14 (8.2%)
Volunteer turnover	Percent volunteer turnover per year	3 (13.6%)
Long-term program plan	Perceptions of program longevity and support and both the individual and organization levels by administrators, parents, volunteers	N/A

** indicates RBC learn to play requirement.

Reach

At the time of the evaluation, 155 athletes had participated in the NUTMEG program and 19 coaches had volunteered. Of the athletes, 67.5% were male and averaged 8.6 years of age. Sessions typically included 20.5 (SD = 4.9) athletes, and athletes averaged attending a total of 7 (SD = 8.5) sessions. NUTMEG had been collecting athlete demographic data through their registration forms which enabled the assessment of the representativeness of the target population (i.e., low-income, and underserved populations in Toronto). Parents were asked to provide details such as their child(ren)’s home and school postal codes to evaluate where most families were located and how far they had to travel to attend the NUTMEG program. Parents were also asked if they identified as being any of the following underserved populations: single-parent, low-income, aboriginal, or immigrant. Representativeness was lowest in the Aboriginal population (1.9%) and highest among low-income (17.4%) and immigrant (18.7%) families. Overall, the data suggested that 47.7% of participants who responded to the questions identified as underserved. Many parents preferred not to answer socioeconomic status (SES) questions, which may have resulted in an underrepresentation of the underserved population.

Effectiveness

NUTMEG did not systematically collect data that would enable the evaluation of Effectiveness. As a result, only three of the potential nine RE-AIM indicators for this dimension were collected: absolute activity hours and athlete perceived importance and enjoyment for physical skills and PYD outcomes. The staff sought informal feedback through casual conversation and end-of-season surveys for perceived enjoyment from the athletes and relevance/importance for physical skills and PYD outcomes of the program from parents. Given the omission of collected data, recommendations from the research team to NUTMEG included resources to help guide the definition and measurement of effectiveness. In this regard, we noted the importance of distinguishing between program, coach, and athlete effectiveness and provided NUTMEG with valid and reliable measures for each of these areas. These recommendations will be discussed in greater detail in a subsequent section.

Adoption

Within the Adoption dimension, only accessibility (i.e., 1 of 4 indicators) could be evaluated as coordinators deemed this, in addition to the free admission, of utmost importance to participation rates. The average distance and travel time were measured; however, modes of transportation were not available, which skewed the results. These findings could not be compared to any valid denominators, such as absolute distance or travel time to other sport programs. Although no denominators were used for calculations of accessibility, these findings are practical because they allow NUTMEG representatives to identify potential facilitators and barriers to participation.

Implementation

Two indicators of Implementation were measured (i.e., 2 of 5 indicators): athlete-coach ratios and athlete perceptions of coach and organization implementation. NUTMEG surpassed its goal of maintaining a ratio of 3 to 4 athletes for every volunteer, with an average of 2.6 athletes for every volunteer. Additionally, NUTMEG conducted end-of-season focus groups with 27 athletes from the co-ed program at the final session of the season. Here, athletes were asked to verbally rate different aspects of the program using a Likert-scale as a guide (e.g., “How well were each of the core principles implemented?”). All six NUTMEG values scored highly. Athletes perceived that five of the six were always present in the weekly sessions while the last value, equity, was considered present most times. However, the questions and language used in these surveys were not validated. For instance, principles such as equity and education could be confusing or ambiguous for the target age group.

Maintenance

Maintenance indicators were well documented (i.e., 3 of 4 indicators). Findings showed that since NUTMEG began, athlete dropout and volunteer turnover were rated relatively low (8.2% and 13.6%, respectively). Although these results look promising for the program’s sustainability, there are no current requirements to define when an athlete is no longer active. As such, athlete dropout was only documented when participants indicated that they would not be returning. Similarly, volunteer turnover was represented by responses suggesting that they did not agree to coach in subsequent sessions. Participant attendance/retention was low, with only 11 athletes attending over half of the sessions. The indicator chosen for participant retention may not be best suited for a ‘drop-in’ program like NUTMEG.

Discussion

The long-term sustainability of any sport program is dependent on continual evaluation, ongoing planning, and improvement of partnerships, resources, effective recruitment, enhanced internal capacity, and clear objectives (Whitley et al., 2015). As there is a limited body of research on pre-existing community-based sport programs and even less on programs aimed at underserved youth, it is important to critically discuss the evaluation process in such contexts (Cullen et al., 2006; Jung et al., 2018). Researchers rarely discuss the challenges they face in research and tend to present an ‘ironed-out’ version of their work that has been subject to peer review (Whitley et al., 2014). However, real-world sport programs do not have the luxury of funded experimental trials or intervention efforts. They must navigate the balance between delivering quality programming with limited resources and volunteer support. Such reality positions evaluation research as central to understanding the strengths and opportunities for growth within programs and aiding the research community to understand more accurately what is happening in youth sport programs. In this regard, it is important to highlight challenges and limitations identified throughout the current evaluation to inform future work. Specifically, three main limitations faced when using the RE-AIM framework were: (1) the lack of indicators being currently measured, (2) the lack of clarity between and within the indicators measured, and (3) the need to navigate current program constraints, such as funding requirements.

First, few indicators from the RE-AIM framework were collected. Although steps had been put in place by NUTMEG to evaluate their program, very few of those measures could be used within the RE-AIM indicators. Such a finding speaks to the narrative that a gap remains between research and practice. Researchers should be providing practical evidence-informed resources that practitioners can use outside of the research paradigm. For instance, Whitley and colleagues (2015) suggested that programs are most effective when they address specific PYD outcomes, have appropriate resources, trained staff, and include meaningful collaborations. The NUTMEG program’s overarching objective was to foster PYD in participating youth. Unfortunately, the omission of measures that enabled the effectiveness assessment made it difficult to determine the extent to which this objective was being met. In sport, researchers have highlighted the need to clearly articulate a program’s objectives and how the attainment of those objectives will be evaluated (Lawrason et al., 2021). Although it is an important endeavor for youth sport programs, it is worth noting the difficulty based on the current research landscape for practitioners to obtain specific tools to facilitate the evaluation of their programs.

A second closely related limitation was that data collected by NUTMEG were not easily translated into RE-AIM measures. Much of the available data did not align with the represented indicators deemed important by researchers. Traditional evaluations put unequal importance on evidence-informed effectiveness measures in comparison to other RE-AIM indicators. The emphasis on evidence-informed over pragmatic indicators can serve as an additional barrier for pre-existing programs as rigorous methods are not likely to be adopted by grassroots programs (Lawrason et al., 2021; Shaw et al., 2019). Additionally, certain aspects of the NUTMEG philosophy were unclear, making it difficult to measure. For instance, education is one of the six principles within NUTMEG, but a consistent description of their education sessions was not provided. This made it difficult to provide suggestions on potential measurement options in the future. This could also be because it was difficult for NUTMEG to define what effectiveness meant to them, which is in line with other non-profit programs (Herman & Renz, 1997). As such, this exemplifies the need for bottom-up research, whereby the needs of the practitioners should be given the highest priority, and resources should be tailored to accommodate the context of interest (e.g., Verhagen et al., 2013).

Finally, challenges were encountered because NUTMEG was a pre-existing program with policies, funding, and programming already in place. As previously mentioned, other PYD sport programs were more easily evaluated as they were implemented with their foundations rooted in theory (e.g., Weiss et al., 2013). In this instance, the evaluation had to align with predetermined requirements put in place by the RBC Learn to Play grant. An extensive summary report is requisite for receiving this grant, so to ease the burden placed on NUTMEG directors, parents, athletes, and volunteers when completing the evaluation, extensive efforts were made to use the Learn to Play measures when possible. For instance, the absolute number of participants was a required measurement for the grant; however, due to the drop-in nature of the program, this was of limited value for NUTMEG to understand their impact in the community.

Recommendations

This section provides recommendations for data collection for youth sport practitioners. These recommendations offer insight pertaining to the facilitators and barriers to completing an evaluation on pre-existing community programs. It is also our intent that this section provides guidance to interested researchers and practitioners seeking to engage in a similar partnership in the future. In addition to the availability of specific indicators, as discussed previously, the feasibility of data collection by community-based programs must be considered. Underserved communities and programs have reduced access to many social and economic resources, which could negatively impact the potential for PYD of its participants. In this regard, we make practical tools more readily available to ensure the sustainability of programs that cultivate PYD (Whitley et al., 2015). This evaluation process informed the following four recommendations for both researchers and practitioners: (a) develop meaningful definitions and measures for individual programs, (b) assess indicators at individual and organizational levels, and (c) create clear and measurable definitions of program success (including effectiveness, adoption, and implementation), and (d) emphasize and be flexible about the needs of the program.

Developing and defining meaningful measures

Since the applicability of the indicators will largely depend on the context of the program, care needs to be taken with how the RE-AIM dimensions are defined (Finch & Donaldson, 2010). Whereas using a framework can provide rigor to the evaluation process, it can also result in strict guidelines that are not applicable in the real world (Cullen et al., 2006). A benefit to the RE-AIM framework is the broad definitions of the five dimensions and the flexibility of indicators chosen within each dimension. This is important as pre-existing programs are typically developed based on the needs of a specific population, and their measurement of success cannot be automatically translated to other programs. For example, a national level organization will have a different definition of adoption than a community-based program and should not use the same measurements. It is therefore important to define the RE-AIM dimensions and create applicable measures for holistic program assessment.

Assessing indicators at individual and organizational levels

In addition to developing relevant evaluation dimensions, Finch and Donaldson (2010) proposed a matrix for navigating the various levels within a sport organization and how to operationally define the different dimensions of the RE-AIM framework depending on the level (e.g., defining effectiveness for stakeholders vs. athletes). Within a community program like NUTMEG, influencers could be described as stakeholders (e.g., partners, directors), volunteers, parents, and athletes. Upon reflection, ensuring all influential levels of the NUTMEG were measured within the dimensions could have been more adequately targeted. As such, it is recommended that a table be developed (as seen in Table 2) concurrently with the selection of indicators when working with a program. By creating a hierarchical matrix prior to the evaluation, both researchers and practitioners can contribute to the assessment of evaluation tools.

Table 2. Hierarchical matrix of suggested RE-AIM indicators

RE-AIM Indicators	Stakeholders/Partners	Program/Directors	Volunteers	Parents	Athletes
Reach	Active recruitment for funding/ sponsorship opportunities	Online impact (website/social media) Other marketing avenues used	Absolute number of volunteers How volunteers recruited	How parents recruited	Absolute number of participants How athletes recruited Representativeness
Effectiveness	Percentage of funding goals met	Goals for year accomplished	Leadership behaviours Perceived satisfaction	Parent perceived influence	Athlete perceived influence Improvement in physical skills Improvement in PYD outcomes
Adoption	Number of guest speakers	Accessibility (locations, website accessibility/ease of use, number of programs offered, travel)	Total number of volunteers with formal certification as a direct result of program Number of volunteers per session		Number of athletes per session
Implementation	Resources needed to implement program (cost of program)	Perception that NUTMEG principles are met through weekly training plans Ratio of athletes to volunteers	Percent volunteers follow weekly training plans Facilitators and Barriers	Physical literacy awareness Facilitators and Barriers	Facilitators and Barriers
Maintenance	Changes over time	Changes over time	Volunteer turnover Perception of program longevity and support Changes over time	Perceptions of program longevity and support Changes over time	Participant retention/continual participation

Defining program success

Community programs should be able to provide a detailed definition of program success. As previously mentioned, there was little data collected to evaluate effectiveness of the current evaluation. NUTMEG could not define effectiveness at either the organizational or individual levels. Further, there were no current empirical measures for effectiveness beyond the perceived enjoyment and importance of various components within the NUTMEG program. This is in stark contrast to the large emphasis evidence-informed interventions place on effectiveness (e.g., Weiss et al., 2013; Turnnidge & Côté, 2017). As such, it is important for practitioners to distinguish between program, coach, and athlete effectiveness and for researchers to provide reliable yet practical measures to best suit these areas. Further, it would be beneficial for practitioners if examples of effectiveness at the different levels were provided to assist in defining their success in a particular context.

Remain flexible and emphasize the needs of the program

Evaluation resources that can be used by program representatives independent from researchers should be available. There is a tendency for researchers to use evidence-informed measures that are reliable and valid. While this remains the gold standard of evaluation methods, it is not always feasible due to program constraints. As such, practitioners need to have access to clear and practical guidelines for evaluation to raise their awareness of the importance of program evaluation. By providing practical resources, practitioners may be motivated to implement specific guidelines into the program’s initial design and be better positioned for long-term success. Researchers must remain flexible and allow concessions for what is both feasible and acceptable within diverse communities (Cullen et al., 2006).

Additionally, Cullen and colleagues (2006) suggested that those involved with community programs are likely to have a better idea of community wants and needs. Therefore, by including both researchers and practitioners in the evaluation process from its initial design stages, both parties remain engaged in the process and can draw from each other’s strengths. Bridging the gap between research and the community is essential to increasing the feasibility of any sport program in the long term.

Conclusion

The RE-AIM framework provided the structure for the evaluation of the NUTMEG program. This paper conducted an evaluation based on data currently collected by the program. These results were then used to inform the recommendations provided to researchers and practitioners. In summary, the analysis identified the need for collaborative efforts to develop feasible evaluation tools for pre-existing community-based sports programs. The use of a flexible framework, like RE-AIM, can enhance the assessment of the impact that community-based programs can have on individuals by allowing for individualized measures to be developed across appropriate levels within the program (e.g., athletes, parents, coaches, stakeholders).

Programs such as NUTMEG are particularly important as they provide the personal and social benefits underserved children may not otherwise experience (Holt & Neely, 2011). By sharing the experience of evaluating a small, community-based program, this study increased awareness of the complexity of community partnerships, hopefully increasing the long-term sustainability of a program such as NUTMEG.

References

Bean, C., Kramers, S., Forneris, T. & Camiré, M. (2018). The implicit/explicit continuum of life skills development and transfer. Quest. 70(4), 456-470. https://doi.org/10.1080/00336297.2018.1451348

Bean, E., Whitley, M. A. & Gould, D. (2014). Athlete impressions of a character-based sports program for underserved youth. Journal of Sport Behaviour, 37(1), 3-23.

Benoit, C., Casey, L., Jansson, M., Phillips, R. & Burns, D. (2011). In for the long haul: Knowledge translation between academic and nonprofit organizations. In B. J. R. Leadbeater, E. M. Banister & E. A. Marshall (Eds.). Knowledge translation in context: Indigenous, policy, and community settings (pp. 15-34). University of Toronto Press.

Brunelle, J., Danish, S. & Forneris, T. (2007). The impact of a sport-based life skill program on adolescent prosocial values. Applied Developmental Science, 11(1), 43-55. https://doi.org/10.1080/10888690709336722

Camiré, M., Forneris, T., Trudel, P. & Bernard, D. (2011). Strategies for helping coaches facilitate positive youth development through sport. Journal of Sport Psychology in Action, 2(2), 92-99. https://doi.org/10.1080/21520704.2011.584246

Coakley, J. (2011). Youth sports: What counts as “positive development”? Journal of Sport and Social Issues, 35(3), 306-324. https://doi.org/10.1177/0193723511417311

Côté, J. & Fraser-Thomas, J. (2016). Youth involvement and positive development in sport. In P. R. E. Crocker (Ed.). Sport and exercise psychology (pp. 256-287). Pearson. 

Côté, J., Turnnidge, J. & Evans, M. B. (2014). The dynamic process of development through sport. Kinesiologia Slovenica, 20(3), 14-26.

Côté, J., Turnnidge, J., Murata, A., McGuire, C. & Martin, L. (2020). Youth sport research: Describing the integrated dynamics elements of the Personal Assets Framework. International Journal of Sport Psychology, 51(6), 562-578.

Cullen, A., Giles, T. & Rosenthal, J. (2006). Evaluating community-based child health promotion programs: A snapshot of strategies and methods. National Academy for State Health Policy.

Danish, S., Forneris, T., Hodge, K. & Heke, I. (2004). Enhancing youth development through sport. World Leisure, 46(3), 38-49. https://doi.org/10.1080/04419057.2004.9674365

Finch, C. F. & Donaldson, A. (2010). A sports setting matrix for understanding the implementation context for community sport. British Journal of Sports Medicine, 44(13), 973-978. https://doi.org/10.1136/bjsm.2008.056069

Gibson, K. (2017). Mixed-methods research in sport and exercise: Integrating qualitative research. In B. Smith & A.C. Sparkes (Eds.). Routledge Handbook of Qualitative Research in Sport and Exercise (pp. 382-396). Routledge International Handbooks.

Glasgow, R.E., Vogt, T.M. & Boles, S.M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327.

Gould, D., Flett, R. & Lauer, L. (2011). The relationship between psychosocial developmental and the sports climate experienced by underserved youth. Psychology of Sport and Exercise, 13(1), 80-87. https://doi.org/10.1016/j.psychsport.2011.07.005

Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W. & Robinson, N. (2006). Lost in knowledge translation: time for a map? Journal of Continuing Education in the Health Professions, 26(1), 13-24. https://doi.org/10.1002/chp.47

Herman, R. D. & Renz, D. O. (1997). Multiple constituencies and the social construction of non-profit organization effectiveness. Nonprofit and Voluntary Sector Quarterly, 26(2), 185-206. https://doi.org/10.1177/0899764097262006

High Five (2020). What is high five: Mission, vision, and principles. https://www.highfive.org

Hoekstra, F., Martin Ginis, K. A., Allan, V., Kothari, A. & Gainforth, H. (2018). Evaluating the impact of a network of research partnerships: A longitudinal multiple case study protocol. Health Research Policy Systems, 16(107), 1-11. https://doi.org/10.1186/s12961-018-0377-y

Holt, N. L. (Ed.). Positive youth development through sport. Routledge International Handbooks.

Holt, N. L. & Neely, K. C. (2011). Positive youth development through sport: A review. Revista Iberoamericana de Psicología del Ejercicio y el Deporte, (6)2, 299-316.

Jung, M., E., Bourne, J., E. & Gainforth, H., L. (2018). Evaluation of a community-based, family focused healthy weight initiative using the RE-AIM framework. International Journal of Behavioral Nutrition and Physical Activity, 15(13). https://doi.org/10.1186/s12966-017-0638-0

Lawrason, Turnnidge, J., Tomasone, J., Allan, V., Côté, J., Dawson, K. & Martin, L. J. (2021). Employing the RE-AIM Framework to Evaluate Multisport Service Organization Initiatives. Journal of Sport Psychology in Action, 12(2), 87-100. https://doi.org/10.1080/21520704.2020.1773592

Martinek, T., Schilling, T. & Johnson, D. (2001). Transferring personal and social responsibility of underserved youth to the classroom. The Urban Review, 33(1), 29-45. https://doi.org/10.1023/A:1010332812171

NUTMEG (2018). About us: What we do. https://nutmegsoccer.org

Pierce, S., Kendellen, K., Camiré, M. & Gould, D. (2016). Strategies for coaching for life skills transfer. Journal of Sport Psychology in Action, 9(1), 11-20. https://doi.org/10.1080/21520704.2016.1263982

Shaw, R. B., Sweet, S. N., McBride, C. B., Adair, W. K. & Martin Ginis, K. A. (2019). Operationalizing the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework to evaluate the collective impact of autonomous community programs that promote health and well-being. BMC Public Health, 19(1), 803. https://doi.org/10.1186/s12889-019-7131-4

Toivonen, H. M., Wright, P. M., Hagger, M. S., Hankonen, N., Laine, K. & Lintunen, T. (2021). Feasibility of a responsibility-based leadership training program for novice physical activity instructors. Frontiers in Psychology, 12, 648235–648235. https://doi.org/10.3389/fpsyg.2021.648235

Turnnidge, J. & Côté, J. (2017). Transformational coaching workshop: Applying a person-centered approach to coach development programs. International Sport Coaching Journal, 4(3), 314-325. https://doi.org/10.1123/iscj.2017-0046

Turnnidge, J., Côté, J. & Hancock, D.J. (2014). Positive youth development from sport to life: Explicit or implicit transfer? Quest, 66(2), 203-217. https://doi.org/10.1080/00336297.2013.867275

Verhagen, E., Voogt, N., Bruinsma, A. & Finch, C. F. (2013). A knowledge transfer scheme to bridge the gap between science and practice: An integration of existing research frameworks into a tool for practice. British Journal of Sports Medicine, 48(8), 698-701. https://doi.org/10.1136/bjsports-2013-092241

Weiss, M. R. (2016). Old wine in a new bottle: Historical reflections of sport as a context for youth development. In Holt, N. L. (Ed.). Positive youth development through sport (pp. 7-20). Routledge International Handbooks.

Weiss, M. R., Kipp, L. E., Phillips Reichter, A. & Bolter, N. D. (2020). Evaluating girls on the run in promoting positive youth development: Group comparisons on life skills transfer and social processes. Pediatric Exercise Science, 32(3), 172-182. https://doi.org/10.1123/pes.2019-0252

Weiss, M. R., Stuntz, C. P., Bhalla, J. A., Bolter, N. D. & Price M.S. (2013). ‘More than a game’: Impact of the first tee life skills program on positive youth development: Project introduction and yearly findings. Qualitative Research in Sport, Exercise, and Health, 5(2), 214-244. https://doi.org/10.1080/2159676X.2012.712997

Whitley, M. A., Forneris, T. & Barker, B. (2015). The reality of sustaining community-based sport and physical activity programs to enhance the development of underserved youth: Challenges and potential strategies. Quest. 67(4), 409-423. https://doi.org/10.1080/00336297.2015.1084340

Whitley, M. A., Forneris, T. & Barker, B. (2014). The reality of evaluating community-based sport and physical activity programs to enhance the development of underserved youth: Challenges and potential strategies. Quest. 66(2), 218-232. https://doi.org/10.1080/00336297.2013.872043

Williams, C., Neil, R., Cropley, B., Woodman, T. & Roberts, R. (2022). A systematic review of sport-based life skills programs for young people: The quality of design and evaluation methods. Journal of Applied Sport Psychology, 34(2), 409-435 https://doi.org/10.1080/10413200.2020.1792583