Evaluation Design

Designing Your Evaluation

STEP 1: Decide what you want to learn.

Work with your partners to decide:

  • The kind of information you want to collect.
  • What you hope to learn from the evaluation.
  • How you will use the information. Evaluation data can be used to improve the functioning of the program, convince administrators of the program’s value, or apply for funding.

Begin by reviewing the logic model you developed to guide your program planning. Look at the activities you included and what you hoped to achieve.

STEP 2: Decide on a research design for your outcome evaluation.

Pre-/Post-Test Comparison: Data are collected from participants before and after the program and then compared. This comparison can measure changes over time, but it cannot tell you whether the changes would have occurred without the program. Pre-/Post-test data can be either quantitative or qualitative. Quantitative data can be compared to assess changes in the group as a whole or changes in individuals.

Mid-term and End-of-Program Data Collection: Frequently, evaluation design and data collection are postponed until the end of a program. In these cases, evaluators typically survey or talk to participants about their experiences after the activity is over. This method gives participants time to reflect before sharing their opinions.

It can also be helpful to:

  • Interview participants or hold group discussions halfway through the program. This practice can help you assess what is working and not working.
  • Talk to people who leave the program prematurely to understand what you could do to increase participant retention.

STEP 3: Explore your options for collecting data.

Both process and outcome data can be collected using either quantitative or qualitative approaches.

Quantitative evaluation refers to something that can be measured numerically. Examples of quantitative data are the number of participants in a program, ratings on a participant satisfaction survey, and scores on instruments that assess knowledge or specific variables, such as loneliness and self-esteem.

Qualitative evaluation uses narrative data such as participants’ responses on an open-ended survey, comments collected during interviews or in a focus groups, field notes taken by an observer, journal entries, and notes based on videotapes of intergenerational programming.

Commonly used data-collection methods include:

Surveying participants: You can measure change over time by scoring the answers to close-ended questions, including multiple choice or true/false questions. You can also administer open-ended surveys, which allow participants to fill out a response in their own words, before and after a program. Narrative answers may show that participants’ attitudes changed over the course of the program. Open-ended questions might include:

  • Describe what you liked most and least about the program.
  • What are two things you learned about yourself?
  • What are two things you learned about another generation?

Conducting interviews or focus groups. Interviews and focus groups can provide information about outcomes, particularly if you ask questions related to:

  • Participants’ feelings about the program and what it means to them.
  • Activities participants liked and disliked and why.
  • What people learned.
  • Ways the program might be run differently.
  • Activities participants might like to do in the future.

Make sure to document particularly poignant quotes and/or stories from participants. These responses can add richness and depth to your evaluation and also be used in your public relations effort.

Using pre- and post-surveys to understand the impact of the program on specific variables. Standardized scales or measures are the gold standard for assessing impact. However, you are not conducting a research project, so you don’t need to worry about the true “validity” of a measure. See the list of frequently used measures in the box on the next page. All of these measures can be used both as a pre-test before you begin a longer term series of activities, and as a post-test after your program has concluded.

You do not need to use every measure. Think about the impact your programs or activities might have and see if there is a measure that will help you measure that impact. Don’t try to measure impact after a stand-alone event, aside from conducting a satisfaction survey.

Observing participants. You can observe interactions between residents and youth, either in person or on video, and record your observations on a checklist of desirable behaviors. Over the course of an intergenerational relationship, observers would expect to see an increase in behaviors that reflect that children and residents feel closer to one another. This kind of research project might appeal to a graduate student.

Reviewing records. Analyzing the information you collect routinely can provide insights into a program’s impact. For example, count the number of times residents attend any activity programs before and after an intergenerational program. You may find that you’ve made progress toward a program goal, like decreasing residents’ isolation. You can also review records of residents who do and do not attend intergenerational programs. However, be cautious about interpreting these comparisons, because other factors may account for changes in residents’ behavior. (See the Documenting Your Activities form.)