Investing in Program Evaluation and Impact Measurement

If you live in New York City, chances are you have met someone who has participated in a Fresh Air Fund program. Today Fresh Air alumni are everywhere, and they are often eager to share their experience and reflect on how The Fund has shaped their lives.  

Beginning in 2017, The Fresh Air Fund made a commitment to back up these powerful personal stories with formal program evaluation and impact measurement. The Fund brought on Anne-Klazien (Anna) Morrison to spearhead the initiative and lead the new Research and Evaluation (R&E) team. 

“With an organization that has been serving New York City children for as many years as The Fresh Air Fund,” Anna says, “there are real opportunities to step back and learn what’s working well and what needs to be tweaked or improved. We want to make sure that we’re making decisions with a fuller picture in mind.”

The R&E team targets three outcomes that are core to The Fund’s mission: communities served, program quality and program impact. 

Children are eligible to participate in The Fresh Air Fund’s summer programs if they live in New York City, attend city schools, and meet income guidelines and age requirements. The Fund’s Community Outreach Team works with families to identify the particular Fund programs that are the best fit for their children.

“We recognize that some communities in New York City are disproportionately under-resourced, and growing up in those communities can be associated with a greater risk to child well-being,” Anna says. “We have been especially sensitive to The Fund’s success in reaching children in these communities.”

The R&E team uses the ranking methodology of the Citizens’ Committee for Children of New York (CCC), which synthesizes data on economic security, health, housing and education in New York City to identify communities with children at highest risk, living in neighborhoods with low high school graduation rates, concentrated poverty, and other risk factors. About a third of Fresh Air Fund participants come from these “highest-risk” NYC communities, Anna says.

The Fresh Air Fund follows each child – from initial application, through the summer experience, to post-summer follow-up – in order to evaluate program quality and understand early indicators of program impact. The Fund gathers the same information from parents, volunteers and staff as well in order to gain a more complete picture.

“We survey the kids with fun, easy to use, anonymous surveys that ask them about what is working well and what needs to be better,” Anna explained.  The youth also report on any skills they developed through their experience. These skills range from moving up a swim level or getting certified in CPR to gaining the confidence to speak up in a group. One of the most commonly reported skills, Anna notes, is “making new friends.”

Surveys like these, combined with application and participation data, focus group discussions, and new partnerships with external research experts, have helped The Fresh Air Fund forge an evidence-based and best practices-informed path forward.  

This path has already revealed opportunities to sharpen The Fund’s impact. Some opportunities are straightforward, such as simplifying the application process for parents and guardians and supplying the books youth want in camp libraries. Other opportunities are more strategic, such as taking a closer look at teen-focused programming and identifying ways to better support new host families and the Fresh Air children visiting them. 

“As soon as summer ends, my colleagues are coming to me, excited to see the new data. They are already starting to plan for the following summer, and they’re really eager to get that information.”

Ultimately, Anna says, the best indicator of program quality is that many kids come back for another summer or enroll in The Fund’s school-year programs. “People vote with their feet,” Anna says. “A lot of our camp leaders and staff were Fresh Air Fund youth participants themselves.”

This past summer, 82 members of the camp staff were alumni of Fresh Air programs and three of the current six camp directors are also alumni. 

Fresh Air Fund staff work to ensure that The Fund’s programs create opportunities for youth participants to expand their perspectives, take on challenges and learn life skills, gain new friendships and mentoring relationships, and reflect on personal goals – all in a safe and fun atmosphere. Parents and guardians regularly report that The Fund’s programs have important, positive effects on their children – from confidence gained to positive goal-setting behavior – especially when the children participate in these programs for multiple years. 

Looking forward, a priority area for The Fresh Air Fund is to better understand how participation – especially repeat participation – shapes longer-term outcomes, such as success in school. Anna notes that social scientists are finding that a strong social-emotional foundation – which Fresh Air Fund programs seek to foster – is a critical ingredient for positive attitudes towards school and learning.

A 2018 pilot study, conducted by faculty from NYU’s Steinhardt School, found that graduates of The Fresh Air Fund’s three-year Career Awareness Program (CAP) had both stronger school attendance and higher on-time high school graduation rates than their non-participating school peers. CAP’s curriculum seeks to support the development of “21^st Century Skills” such as critical-thinking, collaboration and leadership.  

“In New York City,” Anna says, “there has been an effort in recent years to offer more summer programs that are tied to kids’ schools. I do think that this is a positive development, particularly in the sense that it simplifies planning for parents and guardians, and gives kids more summer options. However, this often means that children are spending summer in the same place with the same peers that they would all the time, and in the same communities. The Fresh Air Fund’s programs enable exposure to new places, new people and new experiences. The vast majority of our first-time participants have never left New York City without their family. This is a true adventure that can start a journey that impacts the rest of their lives.”

Learn more about The Fresh Air Fund’s programs and how they’re helping children from New York City’s low-income communities.