Probabilistic survey questions and incorrect answers: Retirement income replacement rates

Journal of Economic Behavior & Organization 82 (2012) 267–280



Contents lists available at SciVerse ScienceDirect


Journal of Economic Behavior & Organization
journal homepage: www.elsevier.com/locate/jebo




Probabilistic survey questions and incorrect answers:
Retirement income replacement rates
Peter van Santen a,b,∗ , Rob Alessie a,b , Adriaan Kalwij b,c
a
University of Groningen, The Netherlands
b
Network for Studies on Pensions, Ageing and Retirement (Netspar), Tilburg, The Netherlands
c
Utrecht University, The Netherlands




a r t i c l e i n f o a b s t r a c t

Article history: We study responses to subjective retirement income replacement rate expectations ques-
Received 22 June 2011 tions in a survey of Dutch employees. One out of three respondents is unable to provide
Received in revised form 24 January 2012 probabilities satisfying the requirements of a cumulative distribution function. We show
Accepted 10 February 2012
that using probabilistic survey questions yields an endogenous sample selection when
Available online 17 February 2012
these individuals are removed from the sample, biasing the results toward more pes-
simistic expectations and excess uncertainty in the replacement rate. These biases are most
JEL classification:
D91
prevalent for less-educated individuals.
H55 © 2012 Elsevier B.V. All rights reserved.
D31

Keywords:
Subjective expectations
Selection bias
Retirement income uncertainty




1. Introduction

The standard life-cycle consumption model with uncertain (pension) income predicts that consumption during one’s
working life is positively related to what individuals expect to receive as income, and negatively related to income uncertainty
(Caballero, 1990). In most empirical work in economics (see e.g., Feldstein, 1974; Attanasio and Rohwedder, 2003; Kapteyn
et al., 2005), individual-specific expectations and uncertainty are not available, leading authors to assume static, rational
expectations. To avoid making such strict assumptions regarding the expectation formation process, several studies suggest
using individuals’ subjective expectations of future income (Dominitz and Manski, 2006; Guiso et al., 2009). Of particular
interest, and the ones used in this study, are probabilistic questions of the type suggested by Dominitz and Manski (1997)
and Manski (2004) which allow the researcher to elicit the subjective cumulative distribution function of an individual’s
pension income. Manski (2004) provides an overview of the use of probabilistic questions, which have become increasingly
popular in recent years and have been used to assess the likelihood of general events (such as inflation and social security
benefits) as well as person-specific events (such as mortality and one’s economic situation). Manski suggests two reasons
why eliciting expectations in a probabilistic way is better than eliciting expectations from vaguely defined answer categories
(e.g., an event is “very likely” or “not too likely”). First, the numerical scale allows comparisons among individuals. Second,
the consistency of a respondent’s answers can be checked using the laws of probability.


∗ Corresponding author at: Faculty of Economics and Business, University of Groningen, P.O. Box 800, 9700 AV Groningen, The Netherlands.
Tel.: +31 50 3636328; fax: +31 50 3637337.
E-mail addresses: (P. van Santen), (R. Alessie), (A. Kalwij).

0167-2681/$ – see front matter © 2012 Elsevier B.V. All rights reserved.
doi:10.1016/j.jebo.2012.02.007

, 268 P. van Santen et al. / Journal of Economic Behavior & Organization 82 (2012) 267–280


This article contributes to the empirical literature by investigating whether incorrect (or inconsistent) answers to proba-
bilistic survey questions, and their subsequent removal from the sample, lead to endogenous sample selection. For example,
less-educated and less financially literate persons may be less likely to answer such questions in a meaningful way—that
is, their answers may not satisfy certain laws of probability—and may also have a different retirement income replacement
rate. Simply excluding these observations when analyzing the determinants of the subjective replacement rate or subjective
uncertainty, as is commonly done in other papers (e.g., Dominitz and Manski, 2006), can therefore result in endogenous
sample selection and bias the parameter estimates.
The quality of subjective expectations, elicited using probabilistic survey questions, is examined in other papers.1 Hurd
and McGarry (1995, 2002) investigate the validity of subjective survival probabilities and find that individuals are well able
to predict their own mortality, underlining Manski’s (2004) conclusion that probabilistic survey questions are informative.
Dominitz (1998) shows that next-year income expectations are able to predict subsequent realizations reasonably well.
In addition, data on expectations and subsequent realizations are used in Dominitz (2001), Das and van Soest (1997), Das
and Donkers (1999), and Stephens (2004). The overviews of Hurd (2009) and Pesaran and Weale (2006) also emphasize the
predictive power of subjective probabilities.
However, Dominitz and Manski (1996, 1997) signal some evidence that not all respondents answer correctly. Dominitz
and Manski (1996) use survey software that automatically signals mistakes in the probabilities entered by the respondent,
after which the respondent must correct the mistake, but still allows the researcher to keep track of them. The authors
find that 7 percent of respondents violate the monotonicity of answers, and 40 percent provide answers incompatible
with the (previously elicited) median of the subjective distribution of future income. Dominitz and Manski (1997) report
21 percent item non-response, 8 percent providing constant probabilities over the thresholds, and 5 percent violating
monotonicity. Kleinjans and van Soest (2010) show that two common fears associated with probabilistic questions, namely,
non-response and focal points (e.g., answering 0 percent, 50 percent, or 100 percent), do not affect the determinants of
retirement expectations, but that individuals round off probabilities instead. Manski and Molinari (2010) investigate the
extent of rounding in more detail and find heterogeneity in answering patterns, with a small fraction (11 percent of the
respondents) always rounding up to multiples of 50. We do not address the issues of rounding or focal points. More closely
related to our study are Dominitz and Manski (2006), who compare the sample statistics of non-respondents to those in
their final sample, using data from the Survey of Economic Expectations (SEE). Non-respondents, defined as providing either
missing values, incomplete, non-valid, or unusable answers for estimating the subjective distribution, in the SEE are more
likely to be female, less likely to be non-Hispanic whites, less likely to be labor force participants, less likely to be married,
and less likely to be high school or college graduates. Furthermore, the probability of non-response is non-monotonic in age.
The degree of non-response, Dominitz and Manski (2006) claim, compares favorably to that from the Health and Retirement
Study, and hence the authors conclude that selection effects are not an issue. By contrast, our empirical findings provide
strong evidence of endogenous sample selection effects when omitting incorrect answers from an analysis of expected
retirement income replacement rates.
We use responses to the Dutch Pension Barometer survey, described in detail in Section 2, which follows the Dominitz
and Manski (1997, 2006) approach. By eliciting points on the (subjective) distribution function of future pension income,
these questions allow the researcher to compute estimates of the expected replacement rate, that is, the ratio of expected
pension income to current income, as well as its standard deviation, which can be interpreted as uncertainty regarding the
replacement rate. We find that about one-third of the respondents are unable to answer correctly, and that the incidence
of violations correlates with observable background variables, such as education, income, and gender. De Bresser and van
Soest (2011) find similar results using the same data source, but use only the sample with correct responses, thus implicitly
assuming exogenous sample selection. A new finding is that excluding those individuals for whom it is not possible to com-
pute the expectation or standard deviation of future pension income results in endogenous sample selection. The resulting
biases are quantified by predicting both the expected replacement rate and the standard deviation of the replacement rate
using a linear model without correcting for selection effects and using a Heckman model that does correct for possible
endogenous selection effects. This quantification shows that ignoring endogenous sample selection yields a downward bias
in the predicted expected replacement rate and an upward bias in the predicted uncertainty (standard deviation) of the
replacement rate. These biases are largest for less-educated individuals.
The paper is organized as follows. Section 2 discusses the data and the exact wording of the survey questions. Section
3 examines the incidence of violations of the laws of probability and how it relates to individual characteristics. Section
4 discusses the computation of the expected value and standard deviation of the replacement rate and relates these to
individual characteristics. It also quantifies the consequences for the parameter estimates of ignoring endogenous sample
selection. Finally, Section 5 presents our conclusions.

2. Data

Since the summer of 2006, CentERdata has been collecting data on the pension benefit expectations of Dutch households
with the Pension Barometer survey. CentERdata, affiliated with Tilburg University, specializes in data collection via (Internet)


1
See the April/May 2011 special issue in the Journal of Applied Econometrics for a collection of papers using subjective expectations.