Answer:
Quality awareness among users and producers are, of course, of great importance for statistical surveys as well as in any other branch. For statistical surveys, however, quality has many facets, some of which can be evaluated by means of a so-called customer satisfaction survey. For instance the user as well as the producer of a statistical survey easily obtains the timeliness accuracy. For other aspects of survey quality, more sophisticated methods must be used to assess the quality. Most important is the accuracy of the survey.

Accuracy is an indicator of the degree, to which the user can rely on the results of a survey. The accuracy should meet the needs of the users and the inevitable inaccuracies should be properly reported, thus enabling the users to make their own judgements as to whether the quality of the data support intended uses or not.

Perfect survey quality is merely a theoretical ideal. Inaccuracies come from errors in the survey. Sources of errors are the sampling procedure (sampling errors) and other steps of the survey (nonsampling errors) particularly nonresponse and measurement errors.

It is crucial that the producer obtains a fair assessment of the accuracy of the survey by means of evaluation efforts like the computing of a confidence intervals and nonsresponse rates, to mention the two most common measures. Thus, the producer’s perceived accuracy is obtained. It is equally important that this is communicated to the users to affect their perceived accuracy. This is done by means of a quality declaration.

Generally, the assessment of quality measures of random errors, i.e. sampling quality, is readily obtained by the computation of confidence intervals (or a point estimates together with their coefficients of variation, CV) that measure the very uncertainty associated with the sampling error, taking advantage of the sampling theory. Unfortunately, there is no unified theory that evaluates response quality in a similar way. Actually the quality awareness varies with the different sources of errors. This may be described by the quality awareness ladder, introducing the following four steps of quality awareness:

Step 4:   Quantitative measure of inaccuracy (confidence interval) available
Step 3:   Quantitative measure of indicator of issue (nonresponse rate) available
Step 2:   Vague awareness of quality issue occurs (as often for measurement error)
Step 1:   No awareness of quality issue (sometimes detected “by accident”)

Obviously, the quality awareness varies from one issue to another. Sampling errors represent the highest step of quality awareness, whereas indicators or judgments normally evaluate nonsampling errors. There is no single measure that brings together all the uncertainty of a statistical survey. However, much work in the realm of survey methods development today is devoted to the bringing up of the awareness of quality issues up the ladder, and much of this work has been successful, though far from completed in terms of a single “total error quality indicator”. Especially, works on measurement error models and variance due to imputation makes it possible to provide quality information beyond that of rates.

Statistical agencies have developed, and are using, quality guidelines and/or policies to help staff have quality in their mind during all steps of surveys and to provide a framework for correctly informing the users of data quality.

Christianson A and Polfeldt T (1995): Evaluation and Improvement of Response Quality at Statistics Sweden – A Research Approach. In Statistics Sweden: European Harmonization, National Decentralization and Quality. Proceedings of the 1^st International Conference on Methodological Issues in Official Statistics. Stockholm, June 12-13 1995.

Christianson A and Tortora R D (1995) Issues in Surveying Businesses: An International Survey. Chapter 14 of Cox B G, Binder D A, Nanjamma Chinnappa B, Christianson A, Colledge M J, and Kott P S. Business Survey methods. Wiley ISBN 0-471-59852-6.
The Survey Statistician, no. 54, pages 16-17, July 2006
 

2. What is a tolerable nonresponse rate?
Anders Christianson
Answer:
This is a somewhat controversial question. Historically, the view upon what is a tolerable nonrespons rate, was stricter, say 40 years ago, than it is today. Some agencies even applied so called minimum permformance standards, implying that survey results were suppressed when standards were not met. Improved weighting methods and a hardening survey climate has made the view upon what is a tolerable nonresponse rate more liberal. However, there is still a common agreement in two respects:
First, it depends on the purpose of the survey if the nonresponse rate should be considered acceptable or not. Some decisions demand a higher degree of accuracy than others do. It is the survey users' responsibilty to take the uncertainty of nonresponse into account when she or he makes decisions.

Thus, it s the survey producer's (if other than the user) obligation to communicate a fair account of this uncertainty to the users. There are also limits as to when a probability sample still is a probability sample when the nonresponse increases, thus jeopardizing the basis of inferense from a sample to the population, from which it is drawn.

So, there is no straightforward answer to the question in terms of a specific percentage. An interesting discussion on this, giving different points of view, is to be found in the July 2002 issue of the Survey Statistician in the "discussion corner" starting with an article by the title Avoid the Need to Impute.

The Survey Statistician, no. 52, page 16, July 2005

Robert F. Belli, University of Nebraska

Answer:
The quality of retrospective reports depends on the quality of autobiographical remembering. Errors arise because all remembering is reconstructive rather than reproductive, and the accuracy of memory reconstructions depend (among other things) on the degree to which information is encoded when events occur, how well information is stored over time, and the extent to which the conditions at retrieval provide effective retrieval cues. With regard to retrospective reports, survey researchers only have some degree of control on the conditions of retrieval. As my work has shown, providing more effective retrieval cues in questionnaires can enhance the quality of retrospective reports. One line of studies has been based on using episodic cues to help participants overcome potential source monitoring errors, as can happen when people remember thinking about voting, or their usual voting behavior, as evidence that they had voted in the last election (Belli, Traugott, Young, & McGonagle, 1999; Belli & Moore, 2004). Another line of studies focuses on calendar-based interviewing methodologies, which permit an ability to utilize retrieval cues that exist within the structure of autobiographical memory to an extent greater than that permissible in traditional standardized question-list (Q-list) methods (Belli, 1998). Studies which have compared calendar-based to standardized Q-list interviewing methodologies have shown higher quality retrospective reports with the calendar-based approaches (Belli, 2004; Belli, Shay, & Stafford, 2001; van der Vaart, 2004). In many cases, additional costs in terms of increased interviewing time are negligible, and there have been no observed increase in interviewer variance, despite the more conversational and flexible nature of calendar-based interviewing (Belli, Lee, Stafford, & Chou, 2004). Finally, calendar-based interviewing has demonstrated its advantages in comparison to Q-list approaches with 2-year and life course reference periods, in both face-to-face and telephone modes, and in both paper and pencil and computer-assisted interviewing data collection methods.

Belli, R. F. (1998). The structure of autobiographical memory and the event history calendar: Potential improvements in the quality of retrospective reports in surveys. Memory, 6, 383-406.

Belli, R. F. (2004, August) Improving the Quality of Retrospective Reports: Calendar Interviewing Methodologies. Paper presented at the sixth international conference on logic and methodology, Amsterdam, the Netherlands.

Belli, R. F., Lee, E. H., Stafford, F. P., & Chou, C-H. (2004). Calendar and question-list survey methods: Association between interviewer behaviors and data quality. Journal of Official Statistics, 20, 185-218.

Belli, R. F., & Moore, S. E. (2004). An experimental comparison of question formats used to reduce vote overreporting. Manuscript submitted for publication.

Belli, R.F., Shay, W. L., & Stafford, F. P. (2001). Event history calendars and question list surveys: A direct comparison of interviewing methods. Public Opinion Quarterly, 65, 45-74.

Belli, R. F., Traugott, M. W., Young, M., & McGonagle, K. A. (1999). Reducing vote overreporting in surveys: Social desirability, memory failure, and source monitoring. Public Opinion Quarterly, 63, 90-108.

van der Vaart, W. (2004). The time-line as a device to enhance recall in standardized research interviews: A split ballot study. Journal of Official Statistics, 20, 301-317.

The Survey Statistician, no. 51, pages 19-20, January 2005