Answer: 
The following excerpt from Biemer and Lyberg (2003) provides some information on this question:
“Specification error occurs when the concept implied by the survey question and the concept that should be measured in the survey differ. When this occurs, the wrong parameter is being estimated in the survey and, thus, inferences based upon the estimate may be erroneous. Specification error is often caused by poor communication between the researcher, data analyst, or survey sponsor and the questionnaire designer.

For example, in an agricultural survey, the researcher or sponsor may be interested in the value of a parcel of land if it were sold at fair market value. That is, if the land were put up for sale today, what would be a fair price for the land? However, the survey question may simply ask “For what price would you sell this parcel of land?” Thus, instead of measuring the market value of the parcel, the question may instead be measuring how much the parcel is worth to the farm operator. There may be quite a difference in these two values. The farm operator may not be ready to sell the land unless offered a very high price for it - a price much higher than market value. Since the survey question does not match the concept (or construct) underlying the research question, we say that the question suffers from specification error.

To take this example a step further, suppose the survey analyst is only interested in the value of the parcel without any of the capital improvements that may exist on it such as fences, irrigation equipment, air fields, silos, out buildings and so on. However, the survey question may be mute on this point. For example, it may simply ask "What do you think is the current market value of this parcel of land?" Note that this question does not explicitly exclude capital improvements made to the land and thus, the value of the land may be inflated by these improvements without the knowledge of the researcher. A more appropriate question might be, “What do you think is the current market value of this parcel of land? Do not include any capital improvements in your estimate such as fences, silos, irrigation equipment, and so on.”

The question, “What do you think is the current market value of this parcel of land?” is not necessarily a poorly worded question. Rather, it is the wrong question to ask considering the research objectives. A questionnaire designer who does not clearly understand the research objectives and how data on land values will be used by agricultural economists and other data users may not recognize this specification error. For that reason, identifying specification errors usually requires that the questions be reviewed thoroughly by the research analyst or someone with a good understanding of the concepts that need to be measured in order to properly address the research objectives. The research analyst should review each question relative to the original intent as it relates to the study objectives and determine whether the question adequately reflects that intent. For the land values example, the agricultural economist or other analyst who will use the data on land values would be the best person to check the survey questionnaire for specification errors. In general, detecting specification error usually requires a review of the survey questions by researchers who are responsible for analyzing the data to address the research objectives and who know best about what concepts should be measured in the survey.

Note that in some disciplines (for example, econometrics), specification error means including the wrong variables in a model, such as a regression model, or leaving important variables out of the model. In our terminology, specification error does not refer to a model, but a question on the questionnaire.”

It should also be noted that specification errors are more common in business and institutional surveys than in household surveys.

The Survey Statistician, no. 51, pages 20-21, January 2005

2. What purposes can web surveys be used for?
Robert D. Tortora, Gallup, Europe

Answer:
Web surveys can be used in some cases to make inferences to some populations and in some cases as a qualitative research tool. The design of the web survey is the critical factor. For instance if it is the case that a client can provide a reasonably complete email list of the target population then a web survey is a methodology that can be used to make statistical inferences. Some examples of realistic email lists for web surveys include lists of employees for a employee satisfaction survey and lists of purchasers at a e-commerce website to measure customer loyalty. Another design that can be used for inferential purposes involves two modes of data collection – a Random Digit Dial (RDD) screening survey to identify eligible respondents that are then asked to go to a website to complete the survey. One important part of web surveys that are designed for inferential purposes includes using an access code so respondents can only complete the survey once.

Knowledge Networks ( http://www.knowledgenetworks.com/ ) uses another web survey design. They use a RDD survey to recruit households to join their panel. They provide recruited households with a web device that allows these households to complete web surveys. Their website has various case studies and white papers that discuss the quality of the data collected in their panel.

Harris Interactive has created a large panel of Internet users. This panel is not representative of Internet users. They do this by conducting parallel surveys, asking their panel members and respondents to an RDD survey the same set of questions. They employee propensity scoring (Rosenbaum and Rubin, 1983) methods to adjust the results of web surveys of their panel members. 

Many web surveys involve a non-random selection of respondents. Many are opt-in surveys where the potential for a self-selection bias exists. These surveys may be implemented as “pop up” windows at a web site. In other cases a web site is open for interviewing and potential respondents may receive email invitations or see invitations (with the URL of the website) at various other websites or in other media. In other cases panels of web users can be created and surveyed as the need arises. These types of design provide qualitative data and generally should not be used for inferential purposes.

Reference
Rosenbaum, P.R. and Rubin, D.B., 1983. “The Central Role of the Propensity Score in Observational Studies for Casual Effects.” 
Biometrika 70 (1): 41-55.

The Survey Statistician, no. 50, pages 15-16, July 2004

 3. Now that cell phones are so frequently used: What is the current status of telephone surveys?

Editorial note: The use of telephone surveys has varied between countries depending on the degree to which households have had access to a telephone in their homes. The progress of the new technology had also progressed differently. We have therefore asked experts from three different parts of the world to respond to this question.

Dennis Trewin, Australian Bureau of Statistics

An important aspect of all surveys is to have a good sample frame. This has always been problematic with random digit dialing, even more so with the increasing availability of cell phones. For surveys where households are visited more than once (e.g. a monthly labour force survey), area frameworks can be used with the first interview conducted using face to face interviews. A telephone number can be obtained, perhaps a cell phone, for subsequent interviews. We have found the additional cost associated with the first interview being face to face to be a very worthwhile investment in terms of the improved quality.

For one off surveys, random digit dialing is then a possibility if a telephone framework is not available. This could include cell phone numbers. Of course, "probabilities" would have to be carefully assessed if we include cell phone numbers as it is more likely that households have multiple phones.

I would also caution that all the survey errors and costs be carefully evaluated and compared with the costs of face to face interviewing. The cost of an area frame that is established to support a full range of household surveys, as is often the case for larger survey organisations, can be affordable on a per survey basis as the costs can be amortised over many surveys. It may well be that random digit dialing is a false economy. Certainly in Australia we have not found it an attractive option even though nearly all households have telephones.

Edith de Leeuw, MethodikA Amsterdam (with thanks to Fred Bronner, Albert Emmering, Ger Snijkers, and André Zijdenbos)

The very first telephone surveys were short 10-minute surveys with very simple questions; this was back in 1970. From this simple beginning the telephone survey evolved into a scientific data collection method and became a serious threat to the face-to-face method between 1980-1990. Now in the 21-st century, the question is raised if telephone surveys still have a future. Changes in technology and in society are threatening the validity of telephone surveys. The use of answering machines and other screening devices makes it more difficult to contact respondents, the growing telephone SPAM makes it more difficult to convince respondents to cooperate, and the growing number of mobile (cell) phones is a special challenge for coverage and sampling. 

At present the number of cell phone-only persons is limited to special groups (e.g., students), who were always difficult to reach in standard consumer research. Most households still have a fixed-line in their main dwelling, and only during weekends and in summer, when people are away does it pay to incorporate cell phones in standard telephone surveys. A good example is the Finish Labour Force survey in the month July. Incorporating cell phones asks an adaptation of the methodology and will increase surveys costs. For instance, a cell phone is personal a fixed line is a household, which has implications for sampling. For business-to-business surveys cell phones are less of a problem.

Telephone surveys still have a future and survey methodologists are working hard to overcome the challenges by adapting old methods and developing new methods. Telephone surveys are necessary because face-to-face surveys are too costly, especially in sparsely populated areas and are only used in special cases where interviewers have to perform extra tasks (e.g., observe behaviour or administer tests as in health surveys). Web surveys are still limited to special groups only. Telephone surveys are flexible and combine the personal extras of interviewers with lower costs. Especially in mixed mode designs, telephone surveys will be indispensable for the coming time. It will be used as major mode in household surveys, but also as a prenotification or reminder in web-surveys of individuals and businesses and in electronic data exchange procedure for establishment surveys. Furthermore, it is an excellent selection or screening tool for internet or access panels. 

Recommended reading: Gad Nathan (2001), Telesurvey methodologies for household surveys-A review and some thoughts for the future. 
Survey Methodology, 27, 1, 7-31

Mike Brick, Westat, USA

In the United States and Canada researchers use telephones for both sampling households and as a mode for conducting interviews. The increasing prevalence of cell phones has different but substantial effects on both of these uses that are discussed below. 

Since Waksberg (1978) first introduced an efficient and valid probability method for random digit dialing (RDD), all RDD methods sample only landline telephone numbers. Blumberg, Luke, and Cynamon (2004) found that in the first half of 2003 only 3% of U.S. adults lived in households with only cell phones. However, this percentage is likely to grow substantially over time and result in greater noncoverage. A related problem associated with the increased use of cell phones is a recently implemented regulation that allows people to switch from landline service to a cell phone and keep the same telephone number. Although a large number of persons may not choose this option, it could still make RDD sampling even more difficult. Furthermore, with cell phones in over 60% of households already, RDD response rates may already be suffering deleterious effects. If cell phone users in households with landlines primarily use their cell phones, it may be more difficult to contact and interview these persons on their landlines.

The effects of the proliferation of cell phones on the use of the telephone as a mode of data collection are less clear. One possibility is that people may be more available and willing to be interviewed from their cell phones. While cell phones may be perceived by respondents as providing a more convenient or private option, government agencies have concerns about the confidentiality of these interviews because of the ability to intercept these conversations. Since a cell phone interview could be done while the respondent is also doing another activity such as driving a car, the interviewer must pay some attention to the concurrent activites of the respondent beyond what is currently required for interviews over landlines. An ethical issue arises because the person receiving a call on a cell phone is responsible for the charges in the U.S. Cell phone users who do not wish to participate in the survey are still responsible for the cost of calls made by the survey organization to that phone. This issue may be resolved by revisions in the costing structure in the U.S., but until then, alternatives such as monetary incentives may be necessary.

The increase in cell phone usage presents serious challenges to both RDD sampling methods and the use of the telephone as a mode of data collection. Research to address these challenges has begun, but much work and ingenuity are required. One direction of research is the renewed interest in mixed mode data collection. The dynamics of the technological changes in telephony is likely to continue to require frequent modifications in survey methodology.

References:
Blumberg, S., Luke, J., and Cynamon, M. (2004). Has cord-cutting cut into random-digit-dialed health surveys? The prevalence and impact of wireless substitution. Proceedings of the Eights Conference on Health Survey Research Methods, Atlanta, GA.

Waksberg, J. (1978). Sampling methods for random digit dialing. Journal of the American Statistical Association, 73, 40-46.

 

The Survey Statistician, no. 50, pages 16-18, July 2004

 4. What is the possible impact of question ordering and of adding questions to the end of a questionnaire?
Gary Shapiro, Westat

Answer:
Maurius Cronje provided a very interesting discussion of the effect of questionnaire wording and order in his article in the July 2003 Survey Statistician (p. 27-31). Not only can the answer to a question be affected by whether or not some other question precedes it, as discussed in the article, but it can even be affected by whether or not other questions come later in the survey. Shapiro (1987) gave three examples of such an occurrence. 

The most dramatic example involved the National Health Interview Survey. This survey is conducted every week by the U. S. Bureau of Census for the National Center for Health Statistics. The survey is conducted face-to-face by well-trained interviewers who generally work on the survey for several years. Among other things, the survey always asks about acute health conditions noticed by the respondent within the two weeks prior to the week of interview. (An acute health condition is defined as a condition which has lasted less than three months and which has involved either medical attention or restricted activity.) For two years, 1973 and 1974, there were at least 55 additional questions asked about the reported acute health conditions. Table 1 is an abbreviated version of the table provided in the 1987 paper. The table shows that the number of acute conditions per 100 persons declined by 20.3 percent from 1972 to 1973 and increased by 20.7 percent from 1974 to 1975. 

Table 1 Number of Acute Conditions per 100 Persons per Year

The 1987 paper noted that the results were not obtained under controlled experimental conditions, but that it is highly unlikely that health conditions truly changed in 1973 and 1974. It is not possible to determine the cause of the decline in the years when the supplementary questions were asked, but the paper stated that there were two plausible explanations. First, the interviewer did not want to burden him/herself, or to burden the respondent, with more questions and therefore classified some respondents incorrectly. Second, the presence of the long supplement caused interviewers to rush through the questions to complete the interview more quickly, resulting in less complete reporting by respondents.

References
Shapiro, G. (1987). Interviewer-Respondent bias resulting from adding supplemental questions. Journal of Official Statistics, 3, (2), 155-168. 

The Survey Statistician, no. 50, page 19, July 2004