Guidelines for Surveys on NCF ----------------------------- by Alana Boltwood (ad097@freenet.carleton.ca) Statistics Administrator National Capital FreeNet, Ottawa, Canada (This file was last updated October 15, 1995. It is available at rleton.ca/freeport/freenet/stats/survey on WWW.) (Most of these points apply to all surveys, online and off. Feel free to distribute this information electronically, with content unchanged and credit to the author. Permission required for print publication.) All user surveys conducted on behalf of the National Capital FreeNet should be approved by the Executive Director of NCF. Before a research project is approved, it should be reviewed by someone with technical knowledge of survey research methods. The NCF Board of Directors should be kept informed of major research projects done on NCF's behalf. Surveys which would not appear to be conducted on behalf of NCF do not require approval of the Executive Director or the Board. However, anyone planning to conduct research via a newsgroup or other online forum is encouraged to consult a survey research expert. A well-designed survey will get more numerous and useful responses from the online community. Researchers are also encouraged to consult the NCF Statistics Administrator (address above) to ensure their survey is not collecting information already available elsewhere. The following are some practical and ethical points to consider when planning your research project. They do not cover all the technicalities which NCF is likely to ask about before approving a survey conducted on its behalf. 1. What do you want to find out? Clearly define the objectives of your research project. At every stage, ask yourself whether your plans will meet your objectives. Researchers who do not do this usually waste their time and money. 2. Who do you want to talk to? Define the "population of interest" and find a distribution method that will reach all of them and nobody else. If there are a lot of people in your population, take a random sample instead of surveying everyone. Consult an expert about sampling. Note that FreeNet or Internet users are *not* representative of the general Ottawa, Canadian or world population. Note also that no-one is obligated to complete any online survey, so respondents are "self-selecting". An online survey is only useful if: - your population of interest is part of the online community, and - people's reasons for choosing to answer your survey are not related to the characteristics you are measuring (e.g. a survey asking "Is your online time very limited?" will get fewer "yes" responses than it should) or - you will report qualitative information, rather than numerical results that claim to be representative of a population. Note also that surveys are subject to the usual rules prohibiting excessive cross-posting or repeated posting to newsgroups. Type 'go help-manners' for more information on how "spam" is treated on NCF. 3. How do you want to talk to your population of interest? You could use newsgroups and e-mail to arrange phone or in-person interviews. You could ask open-ended questions by email. You could ask multiple-choice questions using the survey processor developed for NCF (email Jim Elder, aa456, for information), or HTML forms on a Web page. Be sure that the Net is the best medium for meeting your objectives. 4. What will you tell them? Respondents should know who you are, the purpose of your survey, and whether their responses are confidential. They also appreciate knowing when and where the results will be available. 5. What will you ask them? Writing the questionnaire is a very important and very difficult part of a survey; get professional advice. Your questions should be short, simple, clear, specific, grammatical and correctly spelled. Ensure that question content and ordering does not cause bias. Every question should be essential for meeting your survey's objectives. Every objective should be satisfied by the responses to one or more questions. Only ask for information you can realistically analyse. 6. How will you avoid bias? Your methods, questions, analysis and interpretation can affect your results in subtle ways. 7. How will you minimize response burden? There are too many surveys out there and people are answering fewer of them. If your questions are long, confusing or uninteresting, you will have few responses. Lay out the questionnaire so it is easy to read and fill in. Don't conduct surveys too often, and don't make them too long. 8. Have you pre-tested your survey with a few respondents? Make sure your questionnaire is understandable and analysable before you ask everyone to answer it. You can't go back and re-do it afterwards. 9. How will you keep responses confidential? Ensure responses are stored in a safe place, especially if they have user IDs or comment text. User IDs can be encrypted; ask Jim Elder (aa456) or Andrew Patrick (aa118). 10. Who will analyse your data? Make sure you have a number-cruncher available, with the time, skills and software to analyse all your data. Your analysis should not reveal the identity of respondents, unless they agreed in advance to the release of their identities. 11. Who will interpret the analysis and write the report? Make sure you have a good writer who can explain the numbers. Know the audience for your report; if in doubt, make it short and non-technical. 12. Who must approve your report? If your survey is on behalf of NCF, you should determine in advance whether staff or the Board will influence what you can report. The online community traditionally prefers openness. 13. How will you release your results? It is courteous to announce and post results on NCF, in appropriate newsgroups or menus. They should be available in plain-text format because not everyone on NCF can download or view graphics. (Note that some statistical software can still produce graphs made of ASCII characters; they may not be fancy but everyone can read them.) Graphics and Web pages make for a more powerful presentation, but to fewer people; provide both text and graphics if possible. 14. How will your survey be followed up? The report should specify a suitable newsgroup for discussion of results. Follow up to see if the new information is being used effectively. You should evaluate whether your survey met its objectives, and whether it should be repeated later on. Good luck with conducting an effective research study.