Survey Design 101

Everyone at UF is familiar with the time of semester (actually right around the corner) when the students taking Introduction to Statistics 2 hit the Facebook group pages and post survey links, urging fellow students to click on the link and fill out the questions so they can properly study t-tests. These survey questions are pretty simple, and the surveys themselves are short: “Year? Gender? How many alcoholic beverages do you consume per week?”

There’s also advertising and marketing students posting surveys and students involved in campus organizations looking to scout out what others want in terms of entertainment. The point is, we here at UF are familiar with surveys. We can spot the ones that are easy to take, simple to understand and have the potential to make an impact — all while being unbiased and to the point.

Which means we are quick to call out those that do not fit that criteria.

On Thursday night, the Trump administration released a “Mainstream Media Accountability Survey” to see what the American people think of media bias — one big complaint of our current president. From a social science and statistical viewpoint, this survey lacks an incredible number of basic requirements. It is biased, using loaded language we learn quickly in a survey design class not to use. It has leading questions, which basically tell you what to answer. It has questions with false information and no sources: One points out a poll that may or may not exist and be in negation with a few other polls out there. Another discusses how the media uses “slurs” instead of “facts,” but then calls the Affordable Care Act by its colloquial name, “ObamaCare,” which kind of negates the whole point of the question. It has confusingly worded questions with double negations that would make any English teacher out there cringe. (“Do you believe that contrary to what the media says, raising taxes does not create jobs?” — read that just once, and try to figure out what exactly it’s saying.)

The questions about media outlets only focus on CNN, MSNBC and Fox News (the first one Trump earlier tried to discredit as fake news; the latter two are known to skew left and right, respectively) — leaving out other networks and outlets and not even mentioning newspapers.

And quite alarmingly, the survey refers to the Republican Party as “our Party” and its beliefs as “our movement,” which sounds a little too “1984” for comfort.

From a design standpoint, this survey, simply put, sucks. A stats 2 student could tell you this.

We get it: Conservatives feel like mainstream media is against them. But that’s not a new thought. We don’t need a survey to hammer that point across. People who believe there is intense bias in the media will believe it and vice-versa. So hopefully, this survey will do nothing but reaffirm the opinions of opposite sides. But there’s a bit of danger behind a poorly designed survey — the people with no strong opinion may be led to believe false information. There’s the danger of referring to this poorly designed survey and taking its results as truth.

If the Trump administration really wanted to release a survey to garner what the American people thought of media accountability, it would have designed it properly. Because, like we said before and like every student who took stats 2 knows, there is a way to do it, and it’s not hard. It’s not just a mistake; it’s not that the Trump team hasn’t taken a survey design course — this is a deliberate move. For what exactly? We can’t tell just yet, but hold onto that stats 2 knowledge.