2016 Annual Evaluation Review, Linked Document D 1
Analyzing the Determinants of Project Success:
A Probit Regression Approach
1. This regression analysis aims to ascertain the factors that determine development project
outcome. It is intended to complement the trend analysis in the performance of ADB-financed
operations from IED’s project evaluations and validations. Given the binary nature of the project
outcome (i.e., successful/unsuccessful), a discrete choice probit model is appropriate to empirically test
the relationship between project outcome and a set of project and country-level characteristics.
2. In the probit model, a project rated (Y) successful is given a value 1 while a project rated
unsuccessful is given a value of 0. Successful projects are those rated successful or highly successful.
The probability 𝑝𝑖 of having a successful rating over an unsuccessful rating can be expressed as: 1
𝑥 ′𝛽 2
𝑖
𝑝𝑖 = 𝑃𝑟𝑜𝑏 (𝑌𝑖 = 1|𝑿) = ∫−∞ (2𝜋)−1/2 exp(−𝑡2 ) 𝑑𝑡 = Φ(𝒙𝑖 ′ 𝛽)
where Φ is the cumulative distribution function of a standard normal variable which ensures 0≤ 𝑝𝑖 ≤ 1,
𝒙 is a vector of factors that determine or explain the variation in project outcome and 𝛽 is a vector of
parameters or coefficients that reflects the effect of changes in 𝒙 on the probability of success. The
relationship between a specific factor and the outcome of the probability is interpreted by the means
of the marginal effect which accounts for the partial change in the probability. 2 The marginal effects
provide insights into how the explanatory variables change the predicted probability of project success.
3. The development project outcome is determined by several factors—some are complex and
unobservable. These are the elements of vector 𝒙 representing the independent variables in the model.
Table 1 describes the variables, including how each was specified in the econometric model. The
project-level characteristics include: (i) timing of loan approval; (ii) type of lending modality; (iii)
estimated project cost and environmental safeguard classification (as proxies for project complexity);
(iv) indicators of project administration (i.e., percentage of loan amount cancelled, percentage of cost
overrun, and implementation delay); (v) sector classification; and (vi) a dummy category based on year
of approval to control for unobserved time effects (e.g., improvement in the system over time, in
general). Other project-level characteristics that indicate the quality of project management (e.g., staff
information, quality at entry of projects, and other project administration variables) were not included
due to lack of data.
4. Project performance also depends on the conditions in the country in which the project is being
implemented. The most important of these are the economic and policy environment as well as the
political stability of a country. To control for these country-level effects, gross domestic product
averaged over the project implementation period is included to account for the economic environment,
an average corruption perceptions index to account for the policy environment, and a political stability
index for the political environment. The regional location of a development project is included to
control for regional differences that may affect the probability of project success but which are not
captured by the country-level variables.
5. The probit regression analysis includes a sample of 586 projects validated by IED from 2000 to
2015. 3 The sample consists of project completion report validation reports (PVRs) and project or
1
The inverse standard normal distribution of the probability is modeled as a linear combination of predictors. See W.H. Greene.
2011. Econometric Analysis, Prentice Hall. 7th ed. Upper Saddle River, New Jersey, USA.
2
The interpretation of the coefficients in probit regression is not as straightforward as the interpretations of linear or logit
regression coefficients. These coefficients relate the change in the z-score or probit index to a one-unit change in the predictor.
3
A total of 638 project completion reports were validated by IED from 2000 to 2015. However, only 586 projects were included
in the regression analysis. Of the 52 projects not included, 26 were approved before 1990 (country-level data are sparse in
years before 1990), 9 projects have a regional classification (no country-specific data), while 17 projects (mostly in Pacific
developing member countries) were dropped because of missing data.
, 2 Analyzing the Determinants of Project Success: A Probit Regression Approach
program performance evaluation reports (PPERs) evaluated using a four-category system of rating
projects. A project is rated successful based on an aggregated assessment of four evaluation criteria:
relevance, effectiveness, efficiency, and sustainability. Since 2012, these have been weighted equally.
For the purpose of this regression analysis, equal weighting was retroactively applied to projects that
were validated before 2012. The descriptive statistics of the variables used in the econometric analysis
are shown in Table 2.
Table 1. Description of the Variables Used in the Model
Variable Description
Dependent Variable
Project Success Rating A binary variable that takes a value of 1 if the project is rated successful or highly successful
and 0 otherwise
Independent Variables
Project Characteristics
Timing of Loan Approvala A dummy binary variable that takes a value of 1 if the project is approved during the
months of November and December, and 0 if otherwise
Estimated Project Cost Includes government counterpart financing, the ADB loan, and cofinancing ($ million)
Loan Amount Cancelled Percentage of the approved loan amount
Cost Overrun Percentage of the estimated project cost: (actual project cost - estimated project
cost)/estimated project cost
Implementation Delay In years
Lending Modalityb A dummy binary variable that takes a value of 1 for an investment-based (project) and 0 for
a policy-based (program)
Environmental Safeguardsc A dummy categorical variable that takes a value of 1 if a project is classified under a
specified category and 0 otherwise. The categories are A, B, C, and FI. The base category is
A.
Sectorsd A dummy categorical variable that takes a value of 1 if a project is classified under a
specified sector and 0 if otherwise. The sectors are (i) core infrastructure (the base sector),
which includes transport, energy, ICT, and water and other urban infrastructure services; (ii)
agriculture, natural resources and rural development;
(iii) education; (iv) finance; (v) health; (vi) industry and trade; (vii) public sector
management; and (viii) multisector.
Regional Location A dummy categorical variable that takes a value of 1 if a project is located in a specified
Asian region and 0 otherwise. The regions are (i) East Asia (base region),
(ii) Central and West Asia, (iii) Pacific Asia, (iv) South Asia, and (v) Southeast Asia.
Year of Loan Approval A dummy categorical variable that takes a value of 1 if a project is approved within the
specified years of approval and 0 otherwise. The period groupings are 1990-1996, 1997-
2003, and 2004-2013. The base period is 1997-2003.
Country Characteristics
Gross Domestic Product Natural log of the gross domestic product (GDP), averaged over the project implementation
period
Corruption Perceptions Indexe Corruption perceptions index averaged over the project implementation period
Political Stability Indexf Political stability index averaged over the project implementation period
a
A disproportionate number of loan approvals taking place toward the end of the year.
b
Activities which have both investment and policy components, such as sector development programs and multisector
operations, are classified as “programs” to avoid double counting.
c
ADB classifies projects with environmental risks in three categories: A (high risk), B (medium risk), and C (low or no risk). A
separate category exists for investment of funds through a financial intermediary with unknown risk at the initial stage.
d
The sector classification follows the 2014 ADB project classification system. Only the primary sector classification is included in
the regression analysis to avoid double counting.
e
The Corruption Perceptions Index produced by Transparency International is based on expert opinions of public sector
corruption. A poor score is an indication of prevalent bribery, lack of punishment for corruption, and public institutions that
don’t respond to citizens’ needs.
f
The Political Stability and Absence of Violence/Terrorism Index measures perceptions of the likelihood of political instability
and/or politically-motivated violence, including terrorism. It is one of the six aggregate worldwide governance indicators (WGIs)
based on 31 underlying data sources reporting the perceptions of governance of a large number of survey respondents and
expert assessments worldwide. For details, see D. Kaufmann, A. Kraay and M. Mastruzzi. 2010. The Worldwide Governance
Indicators: A Summary of Methodology, Data and Analytical Issues. World Bank Policy Research Working Paper No. 5430
(http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1682130).
Analyzing the Determinants of Project Success:
A Probit Regression Approach
1. This regression analysis aims to ascertain the factors that determine development project
outcome. It is intended to complement the trend analysis in the performance of ADB-financed
operations from IED’s project evaluations and validations. Given the binary nature of the project
outcome (i.e., successful/unsuccessful), a discrete choice probit model is appropriate to empirically test
the relationship between project outcome and a set of project and country-level characteristics.
2. In the probit model, a project rated (Y) successful is given a value 1 while a project rated
unsuccessful is given a value of 0. Successful projects are those rated successful or highly successful.
The probability 𝑝𝑖 of having a successful rating over an unsuccessful rating can be expressed as: 1
𝑥 ′𝛽 2
𝑖
𝑝𝑖 = 𝑃𝑟𝑜𝑏 (𝑌𝑖 = 1|𝑿) = ∫−∞ (2𝜋)−1/2 exp(−𝑡2 ) 𝑑𝑡 = Φ(𝒙𝑖 ′ 𝛽)
where Φ is the cumulative distribution function of a standard normal variable which ensures 0≤ 𝑝𝑖 ≤ 1,
𝒙 is a vector of factors that determine or explain the variation in project outcome and 𝛽 is a vector of
parameters or coefficients that reflects the effect of changes in 𝒙 on the probability of success. The
relationship between a specific factor and the outcome of the probability is interpreted by the means
of the marginal effect which accounts for the partial change in the probability. 2 The marginal effects
provide insights into how the explanatory variables change the predicted probability of project success.
3. The development project outcome is determined by several factors—some are complex and
unobservable. These are the elements of vector 𝒙 representing the independent variables in the model.
Table 1 describes the variables, including how each was specified in the econometric model. The
project-level characteristics include: (i) timing of loan approval; (ii) type of lending modality; (iii)
estimated project cost and environmental safeguard classification (as proxies for project complexity);
(iv) indicators of project administration (i.e., percentage of loan amount cancelled, percentage of cost
overrun, and implementation delay); (v) sector classification; and (vi) a dummy category based on year
of approval to control for unobserved time effects (e.g., improvement in the system over time, in
general). Other project-level characteristics that indicate the quality of project management (e.g., staff
information, quality at entry of projects, and other project administration variables) were not included
due to lack of data.
4. Project performance also depends on the conditions in the country in which the project is being
implemented. The most important of these are the economic and policy environment as well as the
political stability of a country. To control for these country-level effects, gross domestic product
averaged over the project implementation period is included to account for the economic environment,
an average corruption perceptions index to account for the policy environment, and a political stability
index for the political environment. The regional location of a development project is included to
control for regional differences that may affect the probability of project success but which are not
captured by the country-level variables.
5. The probit regression analysis includes a sample of 586 projects validated by IED from 2000 to
2015. 3 The sample consists of project completion report validation reports (PVRs) and project or
1
The inverse standard normal distribution of the probability is modeled as a linear combination of predictors. See W.H. Greene.
2011. Econometric Analysis, Prentice Hall. 7th ed. Upper Saddle River, New Jersey, USA.
2
The interpretation of the coefficients in probit regression is not as straightforward as the interpretations of linear or logit
regression coefficients. These coefficients relate the change in the z-score or probit index to a one-unit change in the predictor.
3
A total of 638 project completion reports were validated by IED from 2000 to 2015. However, only 586 projects were included
in the regression analysis. Of the 52 projects not included, 26 were approved before 1990 (country-level data are sparse in
years before 1990), 9 projects have a regional classification (no country-specific data), while 17 projects (mostly in Pacific
developing member countries) were dropped because of missing data.
, 2 Analyzing the Determinants of Project Success: A Probit Regression Approach
program performance evaluation reports (PPERs) evaluated using a four-category system of rating
projects. A project is rated successful based on an aggregated assessment of four evaluation criteria:
relevance, effectiveness, efficiency, and sustainability. Since 2012, these have been weighted equally.
For the purpose of this regression analysis, equal weighting was retroactively applied to projects that
were validated before 2012. The descriptive statistics of the variables used in the econometric analysis
are shown in Table 2.
Table 1. Description of the Variables Used in the Model
Variable Description
Dependent Variable
Project Success Rating A binary variable that takes a value of 1 if the project is rated successful or highly successful
and 0 otherwise
Independent Variables
Project Characteristics
Timing of Loan Approvala A dummy binary variable that takes a value of 1 if the project is approved during the
months of November and December, and 0 if otherwise
Estimated Project Cost Includes government counterpart financing, the ADB loan, and cofinancing ($ million)
Loan Amount Cancelled Percentage of the approved loan amount
Cost Overrun Percentage of the estimated project cost: (actual project cost - estimated project
cost)/estimated project cost
Implementation Delay In years
Lending Modalityb A dummy binary variable that takes a value of 1 for an investment-based (project) and 0 for
a policy-based (program)
Environmental Safeguardsc A dummy categorical variable that takes a value of 1 if a project is classified under a
specified category and 0 otherwise. The categories are A, B, C, and FI. The base category is
A.
Sectorsd A dummy categorical variable that takes a value of 1 if a project is classified under a
specified sector and 0 if otherwise. The sectors are (i) core infrastructure (the base sector),
which includes transport, energy, ICT, and water and other urban infrastructure services; (ii)
agriculture, natural resources and rural development;
(iii) education; (iv) finance; (v) health; (vi) industry and trade; (vii) public sector
management; and (viii) multisector.
Regional Location A dummy categorical variable that takes a value of 1 if a project is located in a specified
Asian region and 0 otherwise. The regions are (i) East Asia (base region),
(ii) Central and West Asia, (iii) Pacific Asia, (iv) South Asia, and (v) Southeast Asia.
Year of Loan Approval A dummy categorical variable that takes a value of 1 if a project is approved within the
specified years of approval and 0 otherwise. The period groupings are 1990-1996, 1997-
2003, and 2004-2013. The base period is 1997-2003.
Country Characteristics
Gross Domestic Product Natural log of the gross domestic product (GDP), averaged over the project implementation
period
Corruption Perceptions Indexe Corruption perceptions index averaged over the project implementation period
Political Stability Indexf Political stability index averaged over the project implementation period
a
A disproportionate number of loan approvals taking place toward the end of the year.
b
Activities which have both investment and policy components, such as sector development programs and multisector
operations, are classified as “programs” to avoid double counting.
c
ADB classifies projects with environmental risks in three categories: A (high risk), B (medium risk), and C (low or no risk). A
separate category exists for investment of funds through a financial intermediary with unknown risk at the initial stage.
d
The sector classification follows the 2014 ADB project classification system. Only the primary sector classification is included in
the regression analysis to avoid double counting.
e
The Corruption Perceptions Index produced by Transparency International is based on expert opinions of public sector
corruption. A poor score is an indication of prevalent bribery, lack of punishment for corruption, and public institutions that
don’t respond to citizens’ needs.
f
The Political Stability and Absence of Violence/Terrorism Index measures perceptions of the likelihood of political instability
and/or politically-motivated violence, including terrorism. It is one of the six aggregate worldwide governance indicators (WGIs)
based on 31 underlying data sources reporting the perceptions of governance of a large number of survey respondents and
expert assessments worldwide. For details, see D. Kaufmann, A. Kraay and M. Mastruzzi. 2010. The Worldwide Governance
Indicators: A Summary of Methodology, Data and Analytical Issues. World Bank Policy Research Working Paper No. 5430
(http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1682130).
