Whether it is after a webinar or an organization soliciting information, we all fill out our fair share of surveys.
According to Wuyang Hu, professor and AEDE Honors Coordinator, the responses recorded in a survey may not always reflect reality, particularly surveys asking folks “what do you plan to do” instead of “what have you done.”
Hu studies how to improve survey data quality and particularly, how to reduce the difference between information obtained from a survey and through reality. He applies these techniques to examine a variety of issues related to consumer food choices and agribusiness owner’s decision making. Hu says better designed questions yield more precise answers and that in order to reduce bias, survey questions should make people think harder in order to provide deep answers.
One example from a survey Hu plans to send to Ohio producers, growers, and small agribusiness owners, asks “how much revenue have you lost during COVID from March 2020 to March 2021,” instead of “how much do you think COVID has impacted your business?”
The first question is better defined and can generate a dollar figure that actually shows different levels of impact on businesses instead of a generic answer such as “a lot” that may be expected as a result of asking the second question. Hu says posing questions about a person’s actual willingness to pay also generates a truer response.
“Saying you support something is much different than agreeing to financially back that support,” says Hu. “And when you remind respondents that financial support of a cause would be in addition to existing financial investment, their tune changes.”
In 2018, Hu and a group of OSU students designed a study that asked students if they would be willing to support The Ohio State University becoming a certified Bee Campus. The survey detailed how the Xerces Society, a non-profit environmental organization made up of general civilians, offers a Bee Campus certification program that recognizes, supports, and encourages pollinator conservation on U.S. college and university campuses.
Who doesn’t want to help pollinators such as bees, especially on an agricultural campus?
Turns out most students surveyed were in favor. But when money to fund the initiative came into the equation, students were less supportive. Then when students were told that the cost to support the program would be an additional fee to their already-paid student fees, there was less support.
Hu and students plan to produce an additional version of the survey and ask students to give money up front to back the certification process.
“We anticipate that the percentage of students we actually collect money from will be less than the amount of money that was previously committed on paper, says Hu.
This research shows that survey responses have consequences and that the more we think about reality and the reality of financial support, the less willing we are to green light projects, initiatives and causes. Performing cost-benefit analysis relies on precise and realistic estimates of the costs and benefits. Hu, a behavioral and experimental economist at heart believes the bottom line is this: money matters a lot.