Downloaded from orbit.dtu.dk on: déc. 17, 2021
A simulation study to evaluate the performance of five statistical monitoring methods
when applied to different time-series components in the context of control programs
for endemic diseases
Lopes Antunes, Ana Carolina; Jensen, Dan; Hisham Beshara Halasa, Tariq; Toft, Nils
Published in:
P L o S One
Link to article, DOI:
10.1371/journal.pone.0173099
Publication date:
2017
Document Version
Publisher's PDF, also known as Version of record
Link back to DTU Orbit
Citation (APA):
Lopes Antunes, A. C., Jensen, D., Hisham Beshara Halasa, T., & Toft, N. (2017). A simulation study to evaluate
the performance of five statistical monitoring methods when applied to different time-series components in the
context of control programs for endemic diseases. P L o S One, 12(3), [e0173099].
https://doi.org/10.1371/journal.pone.0173099
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright
owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
You may not further distribute the material or use it for any profit-making activity or commercial gain
You may freely distribute the URL identifying the publication in the public portal
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
, RESEARCH ARTICLE
A simulation study to evaluate the
performance of five statistical monitoring
methods when applied to different time-
series components in the context of control
programs for endemic diseases
Ana Carolina Lopes Antunes1*, Dan Jensen2, Tariq Halasa1, Nils Toft1
a1111111111
a1111111111 1 Division for Diagnostics and Scientific Advice—Epidemiology, National Veterinary Institute–DTU,
a1111111111 Bülowsvej 27, Frederiksberg C, Denmark, 2 Section for Production and Health, Department of Large Animal
a1111111111 Science–KU, Grønnegårdsvej 8, Frederiksberg C, Denmark
a1111111111
*
Abstract
OPEN ACCESS
Disease monitoring and surveillance play a crucial role in control and eradication programs,
Citation: Lopes Antunes AC, Jensen D, Halasa T,
Toft N (2017) A simulation study to evaluate the as it is important to track implemented strategies in order to reduce and/or eliminate a spe-
performance of five statistical monitoring methods cific disease. The objectives of this study were to assess the performance of different statis-
when applied to different time-series components
tical monitoring methods for endemic disease control program scenarios, and to explore
in the context of control programs for endemic
diseases. PLoS ONE 12(3): e0173099. what impact of variation (noise) in the data had on the performance of these monitoring
doi:10.1371/journal.pone.0173099 methods. We simulated 16 different scenarios of changes in weekly sero-prevalence. The
Editor: Hiroshi Nishiura, Hokkaido University changes included different combinations of increases, decreases and constant sero-preva-
Graduate School of Medicine, JAPAN lence levels (referred as events). Two space-state models were used to model the time series,
Received: September 8, 2016 and different statistical monitoring methods (such as univariate process control algorithms–
Shewart Control Chart, Tabular Cumulative Sums, and the V-mask- and monitoring of the
Accepted: February 15, 2017
trend component–based on 99% confidence intervals and the trend sign) were tested. Perfor-
Published: March 6, 2017
mance was evaluated based on the number of iterations in which an alarm was raised for a
Copyright: © 2017 Lopes Antunes et al. This is an given week after the changes were introduced. Results revealed that the Shewhart Control
open access article distributed under the terms of
Chart was better at detecting increases over decreases in sero-prevalence, whereas the
the Creative Commons Attribution License, which
permits unrestricted use, distribution, and opposite was observed for the Tabular Cumulative Sums. The trend-based methods detected
reproduction in any medium, provided the original the first event well, but performance was poorer when adapting to several consecutive events.
author and source are credited. The V-Mask method seemed to perform most consistently, and the impact of noise in the
Data Availability Statement: The data used in this baseline was greater for the Shewhart Control Chart and Tabular Cumulative Sums than for
study are publicly available at the following link: the V-Mask and trend-based methods. The performance of the different statistical monitoring
https://figshare.com/s/8760d1be0d738e57292b
methods varied when monitoring increases and decreases in disease sero-prevalence. Com-
(DOI: 10.6084/m9.figshare.4272260).
bining two of more methods might improve the potential scope of surveillance systems, allow-
Funding: The authors would like to thank the Pig
ing them to fulfill different objectives due to their complementary advantages.
Research Centre – SEGES for providing part of the
data used in this study, and the Danish Food and
Agriculture Administration for funding the project.
Competing interests: The authors have declared
that no competing interests exist.
PLOS ONE | DOI:10.1371/journal.pone.0173099 March 6, 2017
, Statistical methods for monitoring endemic diseases and control programs
Introduction
Surveillance and monitoring systems are critical for the timely and effective detection of
changes in disease status. Over the last decade, several studies have applied different statistical
monitoring methods for detecting outbreaks of (re-)emerging diseases in the context of syn-
dromic surveillance in both human and veterinary medicine [1–3]. Different types of models
(such as linear models, logistic regression and time-series models) have been implemented in
the context of syndromic surveillance in order to evaluate the performance and implementa-
tion of these methods [4].
However, it may not be possible to make generalizations about the performance of these
methods when used for monitoring endemic diseases and control programs. In this case, the
availability of control measures (such as vaccination or health-management programs) results
in lower incidence rates for endemic diseases than for (re)-emerging diseases. The dynamics of
disease spread and immunity within a population from previous exposure also contribute to a
lower incidence, resulting in slow and gradual changes in incidence and prevalence for endemic
diseases [5]. It is important to follow-up on implemented control strategies in order to reduce
and/or eliminate a specific disease [6]. Unexpected changes (such as an increase in disease prev-
alence or a failure to achieve a target value of disease prevalence within a certain period of time)
indicate that the implemented strategies should be revised. When a control program fails to
achieve certain goals, it can have a devastating impact on herds with susceptible animals.
In previous work, we assessed the performance of univariate process control algorithms
(UPCA) in monitoring changes in the burden of endemic diseases based on sentinel surveil-
lance [7]. However, these methods were not tested in the context of voluntary disease control
and monitoring programs. In such cases, the frequency of testing depends on the monetary
value of the animal and not just on the impact of the disease [6]. Programs for monitoring
endemic diseases include the Danish Porcine Reproductive and Respiratory Syndrome Virus
(PRRSV) monitoring program. Despite disease-control efforts, PRRSV has contributed to eco-
nomic losses since its first diagnosis in 1992 [8]. Monitoring of PRRSV is primarily based on
serological testing within the Specific Pathogen Free System (SPF System) [9]. The frequency
of testing depends upon the health status of the herd within this system. As a consequence, the
number of samples is not constant and it is necessary to use methods with a more dynamic
structure, allowing the parameters to change over time, thus taking into account the variation
in sample size. Previous studies have also discussed the influence of variation in the number of
samples (i.e. the noise present in data) on the performance of different monitoring methods
[7,10].
State-space models have a flexible structure, allowing parameters to be updated for each
time step [11]. In addition, they can be decomposed, and changes in the components (such as
trends and seasonal patterns) can be monitored for inference [12]. While state-space models
have been used to monitor influenza in humans [13–15] as well as and for herd-management
decisions [16–19], it has not yet been determined how useful these techniques are for monitor-
ing endemic diseases.
The objectives of this study were to assess the performance of different statistical monitor-
ing methods for endemic disease control programs, and to explore what impact of variation
(noise) in the data had on the performance of these statistical monitoring methods. The simu-
lation study was motivated by the Danish PRRSV monitoring program.
Two state-space models were chosen for this study based on their ability to monitor changes
in different time-series components [11]. Five different statistical monitoring methods were
evaluated for each model: three UPCA used in process-control monitoring [20], and two meth-
ods for monitoring changes based on the trend component of the time series.
PLOS ONE | DOI:10.1371/journal.pone.0173099 March 6, 2017
A simulation study to evaluate the performance of five statistical monitoring methods
when applied to different time-series components in the context of control programs
for endemic diseases
Lopes Antunes, Ana Carolina; Jensen, Dan; Hisham Beshara Halasa, Tariq; Toft, Nils
Published in:
P L o S One
Link to article, DOI:
10.1371/journal.pone.0173099
Publication date:
2017
Document Version
Publisher's PDF, also known as Version of record
Link back to DTU Orbit
Citation (APA):
Lopes Antunes, A. C., Jensen, D., Hisham Beshara Halasa, T., & Toft, N. (2017). A simulation study to evaluate
the performance of five statistical monitoring methods when applied to different time-series components in the
context of control programs for endemic diseases. P L o S One, 12(3), [e0173099].
https://doi.org/10.1371/journal.pone.0173099
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright
owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
You may not further distribute the material or use it for any profit-making activity or commercial gain
You may freely distribute the URL identifying the publication in the public portal
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
, RESEARCH ARTICLE
A simulation study to evaluate the
performance of five statistical monitoring
methods when applied to different time-
series components in the context of control
programs for endemic diseases
Ana Carolina Lopes Antunes1*, Dan Jensen2, Tariq Halasa1, Nils Toft1
a1111111111
a1111111111 1 Division for Diagnostics and Scientific Advice—Epidemiology, National Veterinary Institute–DTU,
a1111111111 Bülowsvej 27, Frederiksberg C, Denmark, 2 Section for Production and Health, Department of Large Animal
a1111111111 Science–KU, Grønnegårdsvej 8, Frederiksberg C, Denmark
a1111111111
*
Abstract
OPEN ACCESS
Disease monitoring and surveillance play a crucial role in control and eradication programs,
Citation: Lopes Antunes AC, Jensen D, Halasa T,
Toft N (2017) A simulation study to evaluate the as it is important to track implemented strategies in order to reduce and/or eliminate a spe-
performance of five statistical monitoring methods cific disease. The objectives of this study were to assess the performance of different statis-
when applied to different time-series components
tical monitoring methods for endemic disease control program scenarios, and to explore
in the context of control programs for endemic
diseases. PLoS ONE 12(3): e0173099. what impact of variation (noise) in the data had on the performance of these monitoring
doi:10.1371/journal.pone.0173099 methods. We simulated 16 different scenarios of changes in weekly sero-prevalence. The
Editor: Hiroshi Nishiura, Hokkaido University changes included different combinations of increases, decreases and constant sero-preva-
Graduate School of Medicine, JAPAN lence levels (referred as events). Two space-state models were used to model the time series,
Received: September 8, 2016 and different statistical monitoring methods (such as univariate process control algorithms–
Shewart Control Chart, Tabular Cumulative Sums, and the V-mask- and monitoring of the
Accepted: February 15, 2017
trend component–based on 99% confidence intervals and the trend sign) were tested. Perfor-
Published: March 6, 2017
mance was evaluated based on the number of iterations in which an alarm was raised for a
Copyright: © 2017 Lopes Antunes et al. This is an given week after the changes were introduced. Results revealed that the Shewhart Control
open access article distributed under the terms of
Chart was better at detecting increases over decreases in sero-prevalence, whereas the
the Creative Commons Attribution License, which
permits unrestricted use, distribution, and opposite was observed for the Tabular Cumulative Sums. The trend-based methods detected
reproduction in any medium, provided the original the first event well, but performance was poorer when adapting to several consecutive events.
author and source are credited. The V-Mask method seemed to perform most consistently, and the impact of noise in the
Data Availability Statement: The data used in this baseline was greater for the Shewhart Control Chart and Tabular Cumulative Sums than for
study are publicly available at the following link: the V-Mask and trend-based methods. The performance of the different statistical monitoring
https://figshare.com/s/8760d1be0d738e57292b
methods varied when monitoring increases and decreases in disease sero-prevalence. Com-
(DOI: 10.6084/m9.figshare.4272260).
bining two of more methods might improve the potential scope of surveillance systems, allow-
Funding: The authors would like to thank the Pig
ing them to fulfill different objectives due to their complementary advantages.
Research Centre – SEGES for providing part of the
data used in this study, and the Danish Food and
Agriculture Administration for funding the project.
Competing interests: The authors have declared
that no competing interests exist.
PLOS ONE | DOI:10.1371/journal.pone.0173099 March 6, 2017
, Statistical methods for monitoring endemic diseases and control programs
Introduction
Surveillance and monitoring systems are critical for the timely and effective detection of
changes in disease status. Over the last decade, several studies have applied different statistical
monitoring methods for detecting outbreaks of (re-)emerging diseases in the context of syn-
dromic surveillance in both human and veterinary medicine [1–3]. Different types of models
(such as linear models, logistic regression and time-series models) have been implemented in
the context of syndromic surveillance in order to evaluate the performance and implementa-
tion of these methods [4].
However, it may not be possible to make generalizations about the performance of these
methods when used for monitoring endemic diseases and control programs. In this case, the
availability of control measures (such as vaccination or health-management programs) results
in lower incidence rates for endemic diseases than for (re)-emerging diseases. The dynamics of
disease spread and immunity within a population from previous exposure also contribute to a
lower incidence, resulting in slow and gradual changes in incidence and prevalence for endemic
diseases [5]. It is important to follow-up on implemented control strategies in order to reduce
and/or eliminate a specific disease [6]. Unexpected changes (such as an increase in disease prev-
alence or a failure to achieve a target value of disease prevalence within a certain period of time)
indicate that the implemented strategies should be revised. When a control program fails to
achieve certain goals, it can have a devastating impact on herds with susceptible animals.
In previous work, we assessed the performance of univariate process control algorithms
(UPCA) in monitoring changes in the burden of endemic diseases based on sentinel surveil-
lance [7]. However, these methods were not tested in the context of voluntary disease control
and monitoring programs. In such cases, the frequency of testing depends on the monetary
value of the animal and not just on the impact of the disease [6]. Programs for monitoring
endemic diseases include the Danish Porcine Reproductive and Respiratory Syndrome Virus
(PRRSV) monitoring program. Despite disease-control efforts, PRRSV has contributed to eco-
nomic losses since its first diagnosis in 1992 [8]. Monitoring of PRRSV is primarily based on
serological testing within the Specific Pathogen Free System (SPF System) [9]. The frequency
of testing depends upon the health status of the herd within this system. As a consequence, the
number of samples is not constant and it is necessary to use methods with a more dynamic
structure, allowing the parameters to change over time, thus taking into account the variation
in sample size. Previous studies have also discussed the influence of variation in the number of
samples (i.e. the noise present in data) on the performance of different monitoring methods
[7,10].
State-space models have a flexible structure, allowing parameters to be updated for each
time step [11]. In addition, they can be decomposed, and changes in the components (such as
trends and seasonal patterns) can be monitored for inference [12]. While state-space models
have been used to monitor influenza in humans [13–15] as well as and for herd-management
decisions [16–19], it has not yet been determined how useful these techniques are for monitor-
ing endemic diseases.
The objectives of this study were to assess the performance of different statistical monitor-
ing methods for endemic disease control programs, and to explore what impact of variation
(noise) in the data had on the performance of these statistical monitoring methods. The simu-
lation study was motivated by the Danish PRRSV monitoring program.
Two state-space models were chosen for this study based on their ability to monitor changes
in different time-series components [11]. Five different statistical monitoring methods were
evaluated for each model: three UPCA used in process-control monitoring [20], and two meth-
ods for monitoring changes based on the trend component of the time series.
PLOS ONE | DOI:10.1371/journal.pone.0173099 March 6, 2017
