Imagine working for the Department of Motor Vehicles, not as a desk clerk or driving test administrator, but at the central office in Salem. Imagine that you were tasked with ascertaining how best to allocate fuel tax revenue. This revenue comes from the tax on fuel purchased for cars and trucks, marine boats, off-highway vehicles and other motorized vehicles. How would you go about distributing the revenue to the agencies responsible for these vehicles?
A survey to determine the fuel used in each category might be a good option, but where to begin? The Oregon State University Survey Research Center is ready to help. Since 1973 the Survey Research Center (SRC) has been working with Oregon State faculty and state government agencies to help them conduct and analyze surveys.
In addition to working with more than 15 different agencies in Salem including the Oregon Department of Transportation, Oregon State Marine Board and Oregon Department of Fish and Wildlife, the SRC has worked with Oregon State University faculty and staff on a variety of projects. Examples range from investigations in applied economics to surveys of students graduating from the College of Science.
"It's fascinating because I learned so much about the state for each survey we work with," said Virginia Lesser, head of the Survey Research Center.
The art of survey design
The process of survey design involves first working with the client to determine the budget and the population to be surveyed. Throughout the entire process there is constant back and forth with the client to figure out exactly what they want and what is possible.
Sometimes clients come in looking to survey a specific segment of the population, such as people who use their service. However, there is often no way of knowing exactly who comprises that population. An important component in designing surveys is to determine and accurately define the population from which the results are gathered. In order to obtain measures of confidence around any estimates, probability sampling must be used.
With probability sampling, every member of the population has an equal chance of being surveyed because they are drawn from a random distribution.
In contrast, non-probability sampling involves selecting from a non-random distribution, such as those who seek out a survey to participate in. Due to the lack of a random selection of individuals, it's impossible to identify biases or compute other measurements. Therefore, data collected using non-probability sampling should be interpreted with caution due to the impact of selection bias.
"So the challenge I have is working with a client to figure out how to get the opinions they need when they don't know who the people are that use their service," Lesser said.
Once the population has been selected, there are three primary components to designing a survey: writing the questions, selecting the probability samples and correctly analyzing the survey data. Writing questions is a bit of an art. Good survey questions should be both unbiased and flow well within a questionnaire. The sample selection depends on the population one is looking to reach and the sampling design. The statistical analysis methods must account for the sampling design and any survey errors, such as nonresponse.
The future of survey distribution
Survey administration methods have evolved significantly over the last several years. Phone-based surveys are not used as much as they were 20 years ago due to the low response rates. Web surveys have become more prominent.
Today the Survey Research Center conducts most of its probability-based surveys using a hybrid web and mail approach, combining the usability of the internet with the higher quality and response rates of mail. This ensures reaching a wider array of demographics than either form would alone.
A key advantage of mail-based surveys is that one can easily mail out surveys to every address or a random sample of addresses in an area. The list of addresses used to select the probability samples for SRC surveys covers nearly 100% of all households in the US. The hybrid web and mail approach provides the opportunity for all selected households to respond either by mail or by web. Thus not all households need to have internet access.
Another method to contact survey participants uses email addresses. Email-based surveys may also be used in cases where the SRC or the client knows the email addresses for the population and know that all potential participants have access to email, for example surveys of Oregon State faculty and students.
Increasingly, more surveys are being conducted using non-probability online panels. These panels are composed of people volunteering to complete surveys in exchange for remuneration or other benefits. They have become popular in recent years due to how cheaply and easily they can be used to get relatively large samples. However, the individuals from an online panel who complete a survey are not a random sample of the population. These respondents choose to actively participate because of personal interest, or because of some sort of incentive, such as a gift card.
Due to their self-selecting nature, estimates from non-probability panel surveys tend to be biased as compared to results from probability sampling methods. Relying on non-probability panels for unbiased information would be like relying on a jury entirely made up of people who really want to be on it.
In addition to its work helping others conduct surveys, the Survey Research Center staff also conduct original research. The main focus of their research is on comparing the effectiveness of different survey techniques, studying how even minute details such as envelope color and size, incentives and content of cover letters impact response rates.
Survey response rates continue to decrease and the research conducted by the SRC examines methods to improve response rates in their surveys. Recently, one of Lesser’s graduate students has been working on how to combine probability and non-probability samples, and thus how to integrate the two approaches.
In our ever-changing social and technological landscape, keeping a finger on the pulse of public opinion is vital. The behind-the-scenes work of designing survey questions and researching survey methods is just as vital. For the past 50 years, the Survey Research Center has done both, through their original research and the services they provided to their clients within and without Oregon State University.
