Tips & Tools for Data Collection
“One of the greatest mistakes is to judge policies and programs by their intentions rather than their results.”
Data collection is how we gather information about – and evidence of – progress towards programmatic outcomes. The original purpose of data collection and evaluation in the social impact sector was to assist in improving the quality of programs. However, in recent years, program evaluation has come to focus much more on proving whether a program or initiative works and is actually having a measurable impact in the way it is intended. While important, this focus on accountability and monitoring has unfortunately overshadowed what is the most important reason to collect data and evaluate impact: program improvement and learning.
Choosing which data collection methods are best for your organization is complicated, and there are many factors to consider. To assess which of the possible methods are most appropriate, you’ll want to consider the purpose of your evaluation, the feasibility of your data collection efforts, and the quality data collected.
Purpose of Your Evaluation. When assessing the purpose of your evaluation, you will want to consider how the methods align with your outcomes and indicators and the key questions you are trying to answer:
- Do you need to collect qualitative or quantitative data?
- Are you measuring the % of people who do something, or trying to understand the “why” of something?
- What types of data is appropriate for your audience?
- What types of data are you expected to collect?
Feasibility of the Methods. When assessing the feasibility of your evaluation, you will want to consider the resources and capacity you’ll need to carry out data collection:
- Consider the time needed
- How soon do you need the data collected?
- How often will you need to collect data?
- Consider the resources (materials, financial, human) needed
- Do you have (or can you acquire) what you need to collect the data? (Survey platform license, space to conduct focus groups)?
- Do you have a budget?
- How much staff time can be allocated to data collection?
- Do you have staff with the appropriate skills to collect data?
- Consider your access to your target population
- Do you have a member list you can access easily? Or do you need to employ others to help you access wider populations?
Quality of Data. When assessing the methods for your evaluation, you will want to consider the quality of the data:
- Validity is the degree to which the data collection method accurately captures what it intends to measure.
- Avoid systematic bias or distortion in the data
- Maintain sampling validity (e.g., that the sample you collect data on is as representative of the population as possible)
- Ensure instrument validity (e.g., that the survey or interview instrument is capturing what you intend for it to capture)
- Reliability is the degree to which a data collection method produces stable and consistent results.
- Maintain consistency in data collection procedures in order to generate valid trend data over time.
Check out this Overview of Methods which describes the strengths and weaknesses of various approaches.
Below are a few additional resources to use when developing your data collection plan: