Survey design and implementation

Effective survey design facilitates the gathering of actionable and relevant information. By asking the right questions in the right way to the right people, you increase the value and validity of the results. However, there are many potential pitfalls in designing large-scale surveys - the most fundamental of which is a lack of clarity about what the survey is really seeking to examine.

All too often, OPM has encountered draft abstracts which betray such muddled thinking. Instead of focusing on the pivotal issues, where public opinion or user experience really matters, they are overlong, overcomplicated and overambitious. The first step in effective survey design, therefore, is to define the key indicators - which may involve a process of consultation and negotiation with a range of stakeholders to get to the root of the issue. Once these indicators are defined, it’s also important to look at the language in which the survey is written, avoiding leading questions and ensuring that it is not too complex for the mass of respondents.

One way to accelerate the drafting process is to use existing instruments - surveys and questions sets that have been applied previously or elsewhere. There are a number of benefits to this beyond time and money. Most importantly, it enables comparisons against other areas or countries, and against previous studies. For example, with household surveys, it can enable benchmarking domestic income or education levels against similar areas in neighbouring states, or help show how those same factors have changed over a number of years. Here, OPM’s experience can be invaluable: having designed and conducted surveys around the globe, we can quickly identify relevant instruments to adopt or adapt.

Survey design also involves defining the sampling approach to be taken. This is not simply a question of establishing how many responses are required for the survey to be representative, but also agreeing what geographic and demographic factors need to be taken into account. Such decisions must reflect the survey’s aims and scope, and a snapshot survey can afford to be less rigorous than a policy evaluation, where unscientific sampling could skew the basis for the whole policy.

All of these issues are standard considerations in OPM’s survey design. We have worked with more than governments and donors over decades to design quantitative surveys that feed into policy and decision-making.