For a more general explanation of the weighting process, take a look at this article. Below are some more technical FAQs about Polco’s weighting practices.
Q: Why should data be weighted?
A: The purpose of weighting is to make results more representative of the entire community. Those who respond to a specific survey effort are not always representative of the total population; there are some groups that are often underrepresented (such as young adults and renters). Weighting looks at the demographics of respondents and adjusts the results to bring them more in line with demographic proportions for the entire community. For this reason, weighting is a best practice in the survey research industry and is recommended for the most accurate, representative data.
Q: How exactly does weighting work?
A: We compare certain demographic variables (age, race, gender, ethnicity, housing type, and housing tenure) between the survey respondents and the entire community. We then use an iterative statistical weighting technique called raking, which assigns a weight value to each survey respondent, to correct for discrepancies and align the demographic characteristics of our survey respondents more closely with the known characteristics for the whole population.
Q: Where does demographic data come from?
A: Respondent demographics are either self-reported (via demographic questions on a survey) or from a third-party source, such as voter registration files. If both types are available, we use the self-reported demographic information. This respondent data is then compared to community-wide demographic data from the U.S. Census or the most recent American Community Survey in the weighting process.
For the self-reporting component, we automatically include the relevant demographic questions on Assess benchmark surveys conducted by our in-house survey research team (such as The NCS, The NLES, etc.). For surveys you conduct on Polco through the Engage module, demographic questions must be added manually using the “Suggested Demographic Questions” modal.
Q: What’s an example of weighting in action?
A: Let’s say that according to Census data, 49% of the entire community identify as male. However, based on our survey results, only 35% of our respondents identified as male. Weighting can be used to adjust the data to correct for this discrepancy, applying higher weights to the responses of the underrepresented group (in this example, males) and slightly lower weights to those who responded at higher proportions. With weighting, the results may be adjusted to represent 46% males; while this isn’t a perfect match for the community’s proportions, it’s closer than the raw data was! In this way, the voices of otherwise underrepresented groups are amplified, bringing the results more in line with the actual proportions of the entire community.
Q: How does automatic weighting differ from weighting conducted on benchmark or custom survey assessments?
A: Both weighting processes use the same algorithm to analyze demographic data collected from the survey, compare the survey results to the most recent Census data, and apply weights to adjust results accordingly for improved representation. Both processes look at the same demographic characteristics. The differences mainly exist in process and timing: For automatic weighting on Engage surveys, you must add the corresponding demographic questions through the “Suggested Demographic Questions” feature on the survey builder page. In addition, the weighting is automatically run once per day, for each day the survey is open, as soon as at least 10 responses have been submitted. For survey assessments conducted by Polco’s in-house research team (including both benchmark surveys and custom survey projects), the demographic questions are added to the survey by the survey research team. The weighting algorithm is run when the survey has closed (after all online and paper survey data have been combined into a single dataset, if applicable).
Q: How does Polco help engage hard-to-reach demographic populations?
A: For benchmark and custom survey assessments conducted by our in-house research team, our random sampling process generally includes oversampling multi-family dwelling units, as these are generally proxies for renters and lower-income individuals (a few groups that we’ve found tend to respond to surveys at lower rates). By including more hard-to-reach individuals in our sample from the start, we hope to hear from more participants within these demographic groups and thus reduce the dependency on weighting (although most survey efforts require both to mitigate under-response bias). For Engage surveys, we offer representivity graphs to help you gauge your survey’s reach, as well as guidance on how to target additional outreach based on that information. Weighting can correct for underrepresentation of certain demographics as well, to a certain degree. Together, all of these methods bring more diverse voices into your community engagement opportunities and resulting data.
Q: Does weighting skew the data?
A: Weighting does not skew the data; in contrast, it helps correct for non-response biases that may be present in raw data and provides more representative and accurate data overall. Polco’s data scientists take great care to never overmanipulate the data. Weights assigned to each individual response are capped at a certain limit, to ensure that no individual response receives too much weight. Additionally, resident input is never discounted simply because of demographic identity. Weighting allows all voices to be heard, with the goal of achieving full representivity (proportionate to each community), rather than allowing results to reflect only the loudest voices or the most predominant respondent groups.