Frequently
Asked Questions
- How can I choose a tool that's right for me?
- What kinds of outcomes can be measured with the tools?
- What information about each tool does the Directory include?
- What can the Directory do?
- What can't the Directory do?
- What are the do's and don'ts for using a tool?
- How did we select tools to be in the Directory?
- Why did we develop the Directory?
- How did we find these tools?
- What resources can I use to get ready for outcomes evaluation?
- Where can I go for more specific academic evaluation tools?
How can I choose a tool that's right for me?
The Directory contains a wide variety of tools-encompassing different outcome areas, focusing on varying stages of change, and using differing methods and sources for gathering information. When considering the many tools available, take your time and be thoughtful. Just because a tool matches your outcome area of interest does not mean it is the right tool for you. For more help in finding the right tool, go to Finding a Good Fit Tool. This guide draws on workshop materials to provide you with reflection questions and suggestions to focus on how tools fit the purpose and audience of your measurement plan, your agency's capacity, and the youth in your programs.
Ask yourself the following questions:
- Do the items in this tool match our own definition of this outcome?
- Will this tool help us capture the changes that are realistic to expect given the amount of time our youth have been participating in our program?
- Does this tool collect information from the sources we feel are most important for our evaluation (e.g., teachers, staff, parents, youth)?
- Will this tool offer us the right amount of information on this outcome? Too much? Not enough depth?
- Does the tool come with support and/or software? Data entry, analysis and reporting can be very time-consuming!
- Is the tool a "ready to use" survey or will we need to customize? Do we have the time and experience to do this?
- Do we feel confident that we have the qualifications to use this tool, or are working with someone who does?
What kinds of outcomes can be measured with the tools?
The Directory is a searchable database containing detailed descriptions of 46 tested tools that are available to programs in 11 youth outcome areas. The table shows the 11 outcome areas and examples of elements of the outcome area that tools might measure.
| Outcome Area | Measures related to this outcome might examine: |
| Academic Skills | If your program is intentionally designed to promote the acquisition of specific and measurable academic skills, you are encouraged to work collaboratively with your local school system to coordinate testing efforts. In addition, you can find numerous academic measures through resources listed under 'Where can I go for more specific academic evaluation tools?'. |
| Adult Youth Relationships |
|
| Assets/Resiliency |
|
| Emotional Well Being |
|
| Healthy Lifestyles |
|
| Leadership Skills |
|
| Learning Orientation |
|
| Peer Relationships/Social Competence |
|
| Positive Behavior |
|
| Problem Solving/Decision Making |
|
| Self Concept |
|
What information about each tool does the Directory include?
On the tool Profile Page the Directory includes the following information about each tool:Features of the tool:
- Tool name
- Description
- Age group(s) that the tool applies to
- Respondent (i.e., who completes the tool's survey -- youth, parent, teacher, etc.)
- Whether the tool is available in Spanish
- The outcome areas the tool measures
- Whether the tool is free, or what it costs
- Whether the tool is ready to use
- If the tool comes with instructions for use
- If training is required or offered
- Whether the tool is available online
- If software or scoring or reporting services are available with the tool
- Scales or sub-scales included in the tool
- Sample items
- Number of items in the tool
- Whether the tool has been field tested
- Whether the tool has been tested for reliability or validity
- Whether there is comparative data available from when the tool has been used
- If the tool has been used in research or evaluation
- If the tool is linked with an approach (i.e., program theory or practice)
Usage guidelines:
- Whether the tool's researcher must be notified before the tool is used
- If there are pre-qualifications required and what (if any) qualifications are needed to use the tool
- Special notes or considerations
- Special instructions to follow when using the tool
- Research related to the tool
- Credits or acknowledgements to include if you use the tool
Identifying and Contact information:
- Author(s) or organizations that developed tool
- Year published
- Where to obtain the tool
- Where to get more information about the tool
- Where to get contact or permission information about the tool
- Where to get additional materials or research related to the tool
What can the Directory do?
It offers newly available tools and up-to-date information.
The Directory includes several tools that have been used in large scale after
school evaluations but have not been easily available -- until now. It also
offers up-to-date pricing, contacts and background on each tool, including
information that may NOT be available on product web sites!
It provides a wide variety of tools to choose from.
The variety of tools in the Directory gives programs many choices for finding
measures that meet their needs, interests and resources. For instance:
- Some tools in the Directory target one outcome, others examine numerous outcomes, and some also examine aspects of program quality.
- Some tools are "ready to use," while others require you to customize them to fit your outcomes of interest.
- Some tools focus on early outcomes, such as youth's experiences in the program, while others focus on "intermediate" outcomes that might be gained after longer participation in the program, and still others examine long term outcome areas that require much more time to show change-such as self-concept.
- Tools encompass various age groups and respondents (e.g. youth, staff, parent, teacher).
- Many tools listed in the Directory are available with services for data entry, analysis and reporting.
It gives you lots of ways to search and compare tools.
The Directory has been structured so that you can browse all tools or search by
outcome area or age groups. It includes a brief summary as well as a
full-length profile of each tool that meets your search criteria.
What can't the Directory do?
It's not a recommended list of tools.
All tools listed in this directory were found to meet certain selection
criteria (see Selection Criteria). However, it is important to note that the
directory is not a recommended list of tools. Only you can decide if a tool is
right for you.
It's not a guide to program quality assessment.
While several of the tools listed in this directory also examine aspects of
program quality, the Directory was not intended to be a resource for quality
assessment tools. For a guide to Quality Assessment tools, see "Measuring Youth
Outcome Program Quality: A Guide to Assessment Tools" published by the Forum
for Youth Investment, Impact Strategies. http://www.forumfyi.org/. See also,
www.gse.harvard.edu/hfrp/projects/afterschool/conference/
It's not a guide to family, community-level outcomes.
The purpose of the UMMB Directory was to identify tools that measure YOUTH
outcomes-not family outcomes, or community-level outcomes.
It's not a "How to Measure Outcomes" guide.
The Directory gives information about measurement tools, not how to develop or
use outcome measurement systems. Information related to designing and
conducting an outcome evaluation can be found in FAQ What resources
can I use to get ready for outcomes evaluation?
It's not a collection of ACTUAL tools.
The Directory includes detailed DESCRIPTIONS of tools-not copies of the tools
themselves. To preview or obtain copies of tools, you can click on the website
hyperlink provided or you can call the source of the tool using the information
provided.
What are the do's and don'ts for using a tool?
Changes to tool wording, ratings and scales
Every tool listed in the Directory has undergone field, reliability or validity
testing. This means that tremendous effort, care, and testing has gone into the
exact wording of questions, the options for responding or rating those
questions, and decisions about which sets of questions measure which outcomes
(scales). NEVER change the wording of any question, the rating options or add/
eliminate any questions from a scale unless there are instructions that
specifically tell you that this is allowed.
If a list of items/questions are said to be part of a "scale", "sub-scale" or
"item-set", then ALWAYS use ALL items/questions in that scale must be used
together. Do not use single items from any measure unless instructions
indicate that it is OK to do so.
Instruments from publishing companies
If you purchase a tool from a publishing house, then most likely copyright laws
protect the survey. This means that you may NOT alter the survey in any way or
even reproduce the survey in whole or part WITHOUT explicit written permission
from the publisher. Feel free to use the contact information in the profile to
ask questions.
Using individual scales from a survey
Many tools are comprised of multiple scales (i.e. sets of questions). In many
cases, it is fine to use a single scale. In fact, when a survey has been
designed to measure multiple outcomes, usually you can simply pick and choose
which outcome--and therefore, which scales-- to use. In other words, you don't
have to use the whole survey. However, a tool designed to focus on a single
outcome, (e.g. leadership), may contain several sub-scales that are designed to
work together to provide you with a comprehensive picture of that one outcome
area (e.g.sub scales might be: sets personal goals, challenges themselves leads
others, and stands up for their beliefs). For single outcome tools, it is
usually advisable to use all sub-scales.
Acknowledging your source
In cases where you are constructing your own survey using available scales, you
will ALWAYS need to acknowledge your source/s in your printed surveys and in
any reporting of results.
For example, you would include the following as a footnote:
This survey was adapted from the youth outcome measures developed by Drs. Bill
Peterson and Allison Titcomb, for Prepared and Engaged Youth Serving American
Communities: A National 4-H Impact Assessment Project. (2001), University of
Arizona Institute for Children, Youth and Families.
Please see the "Instructions for Credits and Acknowledgements" section of your selected tool to be sure you are using the correct wording.
Instructions that accompany the tool
If you have important questions about alternate uses or aspects of use that are
not covered in the instructions, you should contact the publisher, researcher
or organization that developed the tool. Contact information is included in the
profiles.
How did we select tools to be in the Directory?
To be considered for the Directory, tools had to meet the following selection criteria:
- Applies to one or more of the listed outcome areas
- Has undergone reliability and/or validity testing, or has been extensively field-tested and is being widely used.
- Is available for use by UNITED WAY programs.
In addition, tools meet at least one of the following preferred criteria:
- Free or affordable.
- Available online.
- Minimal qualifications needed to administer.
- Been used in afterschool settings or evaluations.
- Comes with support or software to help with scoring, interpretation and reporting.
- Has available data that allows programs to compare their findings with scores from large, and/or demographically diverse samples.
Why did we develop the Directory?
UNITED WAY has had a long-term interest in increasing the focus on measuring benefits to program participants. We believe what organizations learn from measurement can help improve programs, communicate their value to supporters, and enhance benefits for youth. With this Directory, we hope to encourage agencies to use research-based tools when measuring outcomes, both to enhance confidence in the results and to focus attention on those outcomes that are known to be important for youth development.
By their nature, the tools in this Directory focus on quantitative results. Qualitative approaches that derive data from observations, interviews, or verbal interactions and focus on the meanings and interpretations of the participants are valuable in their own right and can also be important complements to quantitative approaches.
Our work on the Directory began with an analysis of information about the outcomes of UNITED WAY affiliates' youth programs. We also scanned the standardized tools affiliates currently use to identify the outcomes each tool addresses. We concluded that a short list of key outcomes could meet the needs of many programs, and that there are available research-based tools that address these outcomes.
We hope the Directory will also be a vehicle through which funders of youth programs can focus their attention on those outcomes that sound research identifies as being most clearly associated with positive results. By working collaboratively with a variety of funders, we hope to simplify the messages programs receive about measurement and expand the resources available to them.
How did we find these tools?
A comprehensive scan of available measures in 11 youth outcome areas was conducted during the fall of 2005. Key sources for the scan were in education, psychology, youth development and after-school such as:
- Publications of large-scale after-school research studies.
- Available compendiums of youth outcome measures
- Instrument publishing houses.
- Key informant interviews with professionals in the field involved in research and measures.
- Web sites for intermediary after school organizations.
- General on-line searches.
The following resources yielded several appropriate tools that are included in the UWMB Guide to Youth Outcome Measures. Readers may want to consult these resources themselves when seeking other measurement instruments that fit their needs. This list is not a recommendation of the following sites, nor is it meant to be comprehensive.
- "Evaluating the National Outcomes." National Outcome Work Group. (ag.arizona.edu/fcs/cyfernet/nowg).
- "Indicators of Positive Development Conference." Mar. 2003. Child Trends. (www.childtrends.org/Files/ConferenceSummary.pdf).
- "Measurement Tools for Evaluating Out-of-School Time Programs: An Evaluation Resource." Nov. 2005. Harvard Family Research Project. (www.gse.harvard.edu/hfrp/projects/afterschool/resources/snapshot6).
- "Minnesota Charter Schools Evaluation: Resources." Center for Applied Research and Educational Improvement (CAREI) College of Education and Human Development, University of Minnesota (education.umn.edu/CAREI/Reports/MNCharter/resources.html#Goal).
- "Outcome Measurement in National Health and Human Service and Accrediting Organizations." 2003. United Way of America (national.unitedway.org/files/pdf/outcomes/natlorgsreportfinal.pdf).
- Pratt, C, A Katzev, R Ozretich, T Henderson, and W McGuigan . "Appendix of Measures for Building Results III: Measuring Outcomes for Oregon 's Children, Youth, and Families." 1998. Oregon State University , Family Policy Program. Prepared for the Oregon Commission on Children and Families: 530 Center St. NE, Suite 405 , Salem , Oregon 97301 . (503) 373-1283 (ccf.state.or.us).
What resources can I use to get ready for outcomes evaluation?
Training materials developed by UNITED WAY are available on our website. This list includes assorted guides, other websites, and publications that may be useful to readers who want to learn more about evaluating youth outcomes.
Chinman, Matthew, Pamela Imm, and Abraham Wandersman. "Getting to Outcomes
2004: Promoting Accountability through Methods and Tools for Planning,
Implementation, and Evaluation." 2004. RAND Corporation.
www.rand.org/publications/TR/TR101
Curnan, S P., and L A. Cava. "Getting Ready for Outcome Evaluation: Developing
a Logic Model." Community Youth Development Journal (2001).
www.cydjournal.org/2000Winter/hughes_S1.html
"Evaluation Forum." A list of evaluation resources/publications.
http://www.evaluationforum.com/publications
Hatry, H, J Cowan, and M Hendricks. "Analyzing Outcome Information." 2003. The
Urban Institute. 2003
www.urban.org/publications/310973.html
Harvard Family Research Project . Site includes multiple resources available
related to after school evaluation. Visit address below or call (617) 495-8594.
http://www.gse.harvard.edu/hfrp
Little, P, S Dupree , and S Deich. "Documenting Progress and Demonstrating
Results: Evaluating Local Out-of-School Time Programs." 2002. Harvard Family
Research Project and the Finance Project.
http://www.hfrp.org or
www.financeproject.org
"Measuring Program Outcomes: A Practical Approach." United Way of America .
1996.
701 North Fairfax St. Alexandria , Virginia 22314 . 800-772-0008. Item # 0989.
"Outcome Measurement Resource Network." United Way of America . 800-772-0008.
national.unitedway.org/outcomes/resources/What/OM_What.cfm
"Out-of-School Time Clearinghouse: Measuring and Using Results." The Finance
Project. See Other Resources. Includes publications and evaluation tool kits.
http://www.financeproject.org/irc/ost/measuring.asp
"Out-of-School Time Program Evaluation: Tools for Action." 2003. Northwest
Regional Educational Laboratory. Item #W2462.
www.nwrel.org/ecc/21century/publications/ost-tools.pdf
Weiss, C H. Evaluation. 2nd ed. Upper Saddle River , New Jersey : Prentice Hall, 1998.
Where can I go for more specific academic evaluation tools?
There are literally thousands of tools available to assess changes in youth academic skills. The list of resources below includes a few sources to get you started. After school and other youth serving programs interested in assessing academic skills are strongly encouraged to contact their local school system in order to coordinate testing efforts.
Publishers:
American Guidance Services (AGS)
www.agsnet.com
800-328-2560 651-287-7220
CTB
www.ctb.com
800-538-9547
831-393-0700
Harcourt Assessment
www.harcourtassessment.com
800-211-8378
McGraw Hill
www.mheducation.com/div_assessment
800-262-4729
Pearson Learning Group
www.pearsonlearning.com/dra
800-526-9907
Pro Ed Publishers
www.proedinc.com
800-897-3202
Riverside
www.riverpub.com
800-323-9540
Western Psychological Services (WPS)
www.wpspublish.com
800-648-8857
310-478-2061
Reviews & Listings of Assessment Tools (including academic
assessment)
Buros Institute of Mental Measurements.
buros.unl.edu/buros
Educational Testing Service.
www.ets.org.
Maddox, Taddy. "Tests, Fifth Edition:
A Comprehensive Reference for Assessments in Psychology, Education and
Business." 2003. Pro-Ed.
www.proedinc.com
Some Other Resources for Academic Assessment in Afterschool:
Assess 21 . 2004. Partnership for 21st Century Skills.
www.21stcenturyskills.org/assess21.
National Partnership for Quality Afterschool Learning . Southwest Educational
Development Laboratory.
www.sedl.org/afterschool
United Way wishes to thank Wendy Surr and Beth M. Miller of Miller Surr Research Associates who helped design the Directory, conducted the research, and provided invaluable advice about its format and use. We would also like to thank and acknowledge the many researchers and others who helped make the Directory possible through providing background information and/or generously making their tools more accessible to youth serving programs. Finally, special thanks goes to Joan Huang and Piri Miller, research assistants, who helped conduct research and collect detailed information on hundreds of measurement tools.