As elections approach in Germany, the number of surveys on
voting intentions increases. Every week you can read some wild speculations on
why party X lost one point, while another gained one. If you are tired of journalists who interpret nothing but noise in the data, you might want to have a look at
the diagrams below. They show you the long term trend in voting intentions for
each major party as a moving average of individual survey results (including
surveys from all major research agencies). Confidence intervals for every survey
estimate are also shown. The data is up to date - last included survey was published on June 18.
The plenitude of tools and specialisations that
agencies offer to potential clients on the research market is stunning. If you
listen to marketing speeches of the research industry, you will be tempted to
believe that any question can be answered by research, at any time, anywhere,
for any target population, and at reasonable costs. You want to know if the
packaging of your product should be gradually bluer? You want to measure the
long term return on investment of hiring external head hunters with absolute
precision? You want to estimate customer satisfaction with a service that does
not even exist yet? You might even have more esoteric issues that I cannot
imagine right know? Just ask a research agency about the feasibility of your
research project. The answer which is most likely to get is: yes we can. This
might be the consequence of a competitive market. If one agency says “no we
can’t” a potential client easily goes to another one. However, this does not
mean that the conclusions drawn from overstretched projects are any better than
speculations. But naïve trust in numbers makes dubious research a profitable
undertaking for some agencies. Actually, many clients do not buy knowledge and
insights but ‘numbers’ - no matter how they came about. To some people
‘numbers’ have the aura of precision and truth, at least as long as they
broadly fit into one’s own presumptions on an issue. This is surely a poor
understanding of what research is good for, but a popular one nonetheless.
Basic methodological issues like sampling, statistical inference or
psychometric considerations in research design are ignored way too often in
practice. Ignorance rules in many businesses which buy research services. So,
if you are on the client side and if you really want to gain insights, than focus
on methodologically justifiable projects and give up on projects that are
likely to produce nothing but useless ‘numbers’. And if you are really in the
need for ‘numbers’, just throw some dices. This will deliver equivalently useless
‘numbers’. But it is significantly cheaper than commissioning research.
