Surveys
From BANwiki
Contents |
[edit] Introduction
A survey is simply a systematic study designed to gather information or views. Ideally, sample sizes will be large enough to be significant (statistically speaking) and the survey design will allow for easy compilation of the data obtained (for example, by designing questionnaires with closed questions). An important consideration in using surveys is whether there need to be baseline and follow-up surveys (“before and after”). For example, collecting data on essential characteristics of companies within a specified sector (such as number of employees, turnover, installed capacity) before the intervention and conducting the same survey one or two years later would produce some information on the development of the sector during the intervention period.
The important feature of surveys is the use of interviews and questionnaires in data collection. An interview typically occurs whenever a researcher and respondent are face-to-face or communicating via some technology like telephone or computer. There are three subtypes of interviews: unstructured, which allows spontaneous communication in the course of the interview or questionnaire administration; structured, where the researcher is highly restricted on what can be said; and semi structured, which restricts certain kinds of communication but allows freedom on discussion of certain topics.
Although surveys can be a cost-effective type of research, survey suffers from inherent weaknesses. The biggest weakness is that all surveys are essentially exploratory. You can make inferences, but not at the level of cause-and effect and ruling out rival hypotheses, as you may with experimental or quasi-experimental research. Other survey weaknesses include:
- Reactivity: respondents tend to give socially desirable responses that make them look good or seem to be what the researcher is looking for
- Sampling frame: it can be difficult to access the proper number and type of people who are needed for a representative sample of the target population
- Non-response Rate: a lot of people won't participate in surveys, or drop out
- Measurement Error: surveys are often full of systematic biases, and/or loaded questions
[edit] Designing surveys and questionnaires
Initial planning of the survey design and survey questions is extremely important in conducting survey research. Once surveying has begun, it is undesirable to amend the basic research questions since the questions need to be constant to standardize the data set.
A questionnaire is a set of questions in a logical sequence on one or several specific issues, though it is not unusual to include check questions, often out of sequence, intended to identify when respondents are deliberately skewing their answers. The design should allow for a systematic analysis and documentation of the responses, particularly if the sample is quite large. Questionnaires often serve as supporting tools for interviews and surveys, and are often used to gather basic data prior to focus group discussions. Questions can be closed (“Have the services delivered by the BMO been good or bad”) or open (“What do you think about the services?”) questions. The following issues should be considered in designing questionnaires for surveys: respondent attitude, the nature of the questions, the cost of conducting the survey, and the suitability of the survey to your research questions.
- Respondent attitude: When developing your questionnaire, it is important to put yourself into your target respondent’s shoes. Think about how you might react when approached by a surveyor or when you are receiving a phone call. When developing your questionnaire, it is important to choose the method you think will work for your research, but also one in which you have confidence. Ask yourself what kind of survey you, as a respondent, would be most apt to answer. The easiest rule for this is to make your instrument simple and straightforward.
- Nature of questions: It is important to consider the relationship between the interview method that you use and the questions that you ask. For instance, certain types of questions are difficult to answer over the telephone. Think of the problems you would have in attempting to record responses, as in closed questions over the telephone, especially if a scale of more than five points is used. Responses to open-ended questions would also be difficult to record and report in telephone interviews.
- Cost: Along with decisions about the nature of the questions you ask, expense issues may also affect your decision making when planning a survey. The population under consideration, the geographic distribution of this sample population, and the type of questionnaire used all affect costs.
- Ability of instrument to meet needs of research question: Finally, there needs to be a logical link between your survey instrument and your research questions. If it is important to get a large number of responses from a broad sample of the population, you obviously will not choose to do a drop-off written survey or an in-person oral survey.
If you believe that your respondents are likely to use e-mail and the internet, there are a number of excellent web driven survey sites such as Survey Monkey which can help with clear design and can calculate all the results and even generate charts for you.
[edit] Types of questions
Closed questions limit respondents' answers to the survey. The participants are allowed to choose from either a pre-existing set of dichotomous answers, such as yes/no, true/false, or multiple choice with an option for "other" to be filled in, or ranking scale response options. The most common of the ranking scale questions is called the Likert scale question. This kind of question asks the respondents to look at a statement (such as "The most important issue facing our business community in Tanzania in year 2006 is multiplicity of taxes") and then "rank" this statement according to the degree to which they agree ("I strongly agree, I somewhat agree, I have no opinion, I somewhat disagree, I strongly disagree"). Experience suggests that an odd number of options tends to generate a high proportion of answers in the middle – using an even number of choices forces respondents to come donw on one side or the other of ‘average’.
Open questions do not give respondents answers from which to choose, but are phrased so that the respondents are encouraged to explain their answers and reactions to the question with a sentence, a paragraph, or even a page or more, depending on the survey. Open questions are like "What do you think is the most important business issue facing the business community in East Africa?" If you want the respondents to focus on their answer, the question can be phrased as ”Do you think that the most important issue facing the business community is taxation? Explain your answer below” (though you should note that writing questions in this way is likely to introduce bias into the answers).
It’s important to remember that you do not have to use closed or open questions exclusively. Many researchers use a combination of closed and open questions; often researchers use closed questions in the beginning of their survey, and then allow for more expansive answers once the respondent has some background on the issue and is "warmed-up. The advantages of using the two types of questioning are well described in the section below.
In a structured interview, the interview would be restricted only to asking the questions in the questionnaire; the research required by business membership organisations is likely to get the most interesting results if semi-structured interviews are used. In this instance, the interviewer asks the questions as written but, depending on the answer, can probe further to understand why the respondent is answering in a particular way and to explore the topic more broadly. This requires interviewers who are experienced in the particular subject and also requires more careful analysis since the results are no longer simply adding up all the individual answers.
[edit] Advantages of closed questions
Closed questions have a number of advantages:
- Closed questions are more easily analysed. Every answer can be given a number or value so that a statistical interpretation can be assessed. Closed questions are also better suited for computer analysis. If open questions are analysed quantitatively, the qualitative value of information is reduced. Because of the simplicity of closed questions, this kind of loss is not a problem.
- Closed questions can be more specific, thus more likely to communicate similar meanings. Because open questions allow respondents to use their own words, it can sometimes be difficult to compare the meanings of the responses.
- In large-scale surveys, closed questions require less time from the interviewer, the participant and the researcher, and so is a less expensive survey method. The response rate is higher with surveys that use closed questions than with those that use open questions.
[edit] Advantages of open questions
The use of open questions offers the following advantages:
- Open questions allow respondents to include more information, including feelings, attitudes and understanding of the subject. This allows researchers to access better the respondents' true feelings on an issue. Closed questions, because of the simplicity and limit of the answers, may not offer the respondents enough choices to reflect their real feelings accurately. Closed questions do not allow the respondent to explain that they do not understand the question or do not have an opinion on the issue.
- Open questions cut down on two types of response error: respondents are not likely to forget the answers from which they have to choose if they are given the chance to respond freely; open questions do not allow respondents to disregard reading the questions and just "fill in" the survey with all the same answers (such as filling in the "no" box on every question).
- Because they allow for obtaining extra information from the respondent, such as demographic information (current employment, age, gender, etc), surveys that use open questions can be used more readily for secondary analysis by other researchers than can surveys that do not provide contextual information about the survey population.
[edit] Issues to consider when designing questions
In order to write effective questions, you need to consider four important factors: directness, simplicity, specificity and discreteness.
- Questions should be written in a straightforward, direct language that is not caught up in complex rhetoric or syntax, or in a discipline's jargon. Questions should be specifically tailored for a group of respondents.
- Questions should be kept short and simple. Respondents should not be expected to learn new, complex information in order to answer questions.
- Specific questions are usually better than general ones. Research shows that the more general a question, the wider is the range of interpretation among respondents. To keep specific questions brief, you can use longer introductions that provide the context, background, and purpose of the survey so that this information is not required in the questions.
- Avoid questions that are overly personal or direct, especially when dealing with sensitive issues.
Note that surveys often have problems with unclear wording, confusing design, questions that are difficult to understand, and typos that could easily be corrected by a small pilot test. Pilot test your survey before releasing it. Even the feedback of one other person can be valuable to help clarify and improve the survey. If you can, try to get a handful of respondents to pilot test. A greater number (30 or more) of respondents makes it possible to conduct statistical analyses on the reliability of questions before doing the main survey.
