Speaking Notes

PADM 5502

October 1, 2009

Dr. Neubauer

WHERE WE ARE:

• The first installment of the research prototype is due October 1. 
• Our midterm exam will be October 8.  It will include questions related to all the things we are discussing in class. This maps approximately to the first 7 chapters of our textbook.

Chapter 7 -- Data Collection: Questions and Questionnaires

Poorly written survey instruments are a huge waste. Worse of all, the resulting data is likely to be useless.

The survey should be derived from the research model. It is necessary to operationalize every variable (and to have a reason for asking every question). This is not a fishing expedition. Some "repetition" is probably good. But every question has a reason.

OPEN ENDED QUESTIONS may capture some QUALITATIVE DATA. It may be necessary to try to convert written answers into categories. Two or more RATERS should be involved and INTER RATER RELIABILITY becomes an issue. If professional raters cannot agree regarding how to code or categorize responses, there is a problem.

Open ended questions are often useful in the DISCUSSION part of you paper. These questions give respondents an opportunity to express things that would be missed if there were no open ended questions. Few people are likely to answer open ended questions. But that is okay.

MAKE IT EASY for respondents. This is not about saving trees. The appearance of the survey instrument should be clear and attractive.

Don't ask them to do very different things like rank-order ten items. It is not necessary and the data produced is not likely to be very useful.

Use FILTER QUESTIONS so people don't have to answer irrelevant CONTINGENCY QUESTIONS.

Group similar questions together. Avoid requiring MENTAL GYMNASTICS.

Avoid double and triple negatives! Having said that, think about the POLARITY of multiple questions intended to measure the same variable. The polarity probably should not always be the same.

If possible, do a PILOT study, or at least get someone who knows nothing about your research to look at the survey instrument. Beyond a point you quickly become blind to your own mistakes.

BE SURE LIKERTS ARE STATEMENTS AND NOT QUESTIONS. PEOPLE CANNOT AGREE OR DISAGREE WITH A QUESTION.

Avoid DOUBLE BARRELED QUESTIONS. Ask one thing at a time. I have never seen a case in which one question could serve to help operationalize two different variables.

Avoid AMBIGUITY. If the question is unclear the responses will be garbage.

Avoid "red flag" words. They tend to produced biased responses.

In my opinion, give people the option of being undecided or not having an opinion. You are trying to capture what is already there. FORCING people to have an opinion is creating "data" that did not exist.

THINK ABOUT THE ORDER OF THE QUESTIONS. If the questions intended to measure the DV are at the end people may quit before getting to the end. If asking one question is likely to influence how people will answer another, order the questions to minimize this effect, if possible.

It is hard to know how and where to ask the income question. The age question can also be an issue for some people.

It is difficult to ask about ILLEGAL or controversial beliefs and behaviors. That is why IRB wants to see the survey instrument.

Avoid jargon and abbreviations that may not be familiar to respondents.

Give the "other" option. It can be offensive not to. If they don't want to answer, they won't. Don't assume, for example, that everyone has a religious affiliation.

A phrase like, "Native American" can be confusing to some respondents.

Ordinal ranges should be complete and not overlapping.

Don't hit respondents with a hard or controversial question immediately.

Be careful how you collect the surveys in a group. Consider providing envelopes for the completed surveys to be "hidden" in.

CONSIDER THE CONTEXT in which you collect your data. If your survey is about attitudes toward abortion you probably should not collect your data at an abortion protest event.

Think about YOUR OWN SAFETY or the safety of those working for you collecting data.

It is okay to give a small "reward" for voluntary participation but the reward should not be something that the respondent cannot refuse. If it is, then the reward can become almost a kind of coercion to participate.

Don't "use" participants. Be respectful. They are not objects. They are indeed, "human subjects" in the very best sense of that phrase. If you appear to place no value in their responses then perhaps you should not be doing this research at all.