Chapter 2: Nature of Data and Statistical Modelling
1. What is Data?
Data are a collection of facts which are usually obtained as the result of experiences,
observations, or experiments.
Data may consist of numbers, words, images, etc.
Data is the lowest level of abstraction (from which information and knowledge are derived).
Data is the source of information and knowledge.
Data quality and data integrity are critical to analytics.
2. Analytics-Ready Data
a) Data source refers to the originality and appropriateness of the storage medium where
reliability the data is obtained
should always look for the original source/creator of the data to
eliminate/mitigate the possibilities of data misrepresentation and data
transformation caused by the mishandling of the data as it moved from the
source to destination through one or more steps and stops along the way.
b) Data content Data content provide by respondent must accuracy
accuracy
c) Data means that the data are easily and readily obtainable
accessibility ” Access to data may be tricky, especially if the data is stored in more than
one location and storage medium and need to be merged/transformed
while accessing and obtaining it
d) Data security means that the data is secured to only allow those people who have the
and data authority and the need to access it and to prevent anyone else from
privacy reaching it
e) Data richness Capture relevant data needed in order to develop a predictive model to
analyze
f) Data means that the data are accurately collected and combined/ merged
consistency
g) Data means that the data should be up-to-date (or as recent/new as it needs to be)
currency/data for a given analytics model.
timeliness It also means that the data is recorded at or near the time of the event or
observation so that the time-delayrelated misrepresentation (incorrectly
remembering and encoding) of the data is prevented. Because accurate
analytics rely on accurate and timely data, an essential characteristic of
analytics-ready data is the timeliness of the creation and access to data
elements.
h) Data requires that the variables and data values be defined at the lowest (or as
granularity low as required) level of detail for the intended use of the data.
i) Data validity means that the variables in the data set are all relevant to the study being
and data conducted.
relevancy
3. Taxonomy of data
, a) Categorical data represent the labels of multiple classes used to divide a variable into
specific groups.
Examples of categorical variables include race, sex, age group,
and educational level.
The categorical data may also be called discrete data, implying that it
represents a finite number of values with no continuum between them.
Even if the values used for the categorical (or discrete)
variables are numeric, these numbers are nothing more than
symbols and do not imply the possibility of calculating
fractional values.
b) Nominal data contain measurements of simple codes assigned to objects as labels,
which are not measurements. For example, the variable marital status
can be generally categorized as (1) single, (2) married, and (3)
divorced.
c) Ordinal data contain codes assigned to objects or events as labels that also represent
the rank order among them.
For example, the variable credit score can be generally categorized as
(1) low, (2) medium, or (3) high.
d) Numeric data represent the numeric values of specific variables.
Examples of numerically valued variables include age, number of
children, total household income (in U.S. dollars), travel distance
(in miles), and temperature (in Fahrenheit degrees).
Numeric values representing a variable can be integer (taking only
whole numbers) or real (taking also the fractional number).
e) Interval data are variables that can be measured on interval scales.
A common example of interval scale measurement is temperature on
the Celsius scale. In this particular scale, the unit of measurement is
1/100 of the difference between the melting temperature and the
boiling temperature of water in atmospheric pressure; that is, there is
not an absolute zero value.
f) Ratio data include measurement variables commonly found in the physical
sciences and engineering.
Mass, length, time, plane angle, energy, and electric charge are
examples of physical measures that are ratio scales.
4. Data Preprocessing
Real-world data is usually dirty, misaligned, overly complex, and inaccurate.
1. What is Data?
Data are a collection of facts which are usually obtained as the result of experiences,
observations, or experiments.
Data may consist of numbers, words, images, etc.
Data is the lowest level of abstraction (from which information and knowledge are derived).
Data is the source of information and knowledge.
Data quality and data integrity are critical to analytics.
2. Analytics-Ready Data
a) Data source refers to the originality and appropriateness of the storage medium where
reliability the data is obtained
should always look for the original source/creator of the data to
eliminate/mitigate the possibilities of data misrepresentation and data
transformation caused by the mishandling of the data as it moved from the
source to destination through one or more steps and stops along the way.
b) Data content Data content provide by respondent must accuracy
accuracy
c) Data means that the data are easily and readily obtainable
accessibility ” Access to data may be tricky, especially if the data is stored in more than
one location and storage medium and need to be merged/transformed
while accessing and obtaining it
d) Data security means that the data is secured to only allow those people who have the
and data authority and the need to access it and to prevent anyone else from
privacy reaching it
e) Data richness Capture relevant data needed in order to develop a predictive model to
analyze
f) Data means that the data are accurately collected and combined/ merged
consistency
g) Data means that the data should be up-to-date (or as recent/new as it needs to be)
currency/data for a given analytics model.
timeliness It also means that the data is recorded at or near the time of the event or
observation so that the time-delayrelated misrepresentation (incorrectly
remembering and encoding) of the data is prevented. Because accurate
analytics rely on accurate and timely data, an essential characteristic of
analytics-ready data is the timeliness of the creation and access to data
elements.
h) Data requires that the variables and data values be defined at the lowest (or as
granularity low as required) level of detail for the intended use of the data.
i) Data validity means that the variables in the data set are all relevant to the study being
and data conducted.
relevancy
3. Taxonomy of data
, a) Categorical data represent the labels of multiple classes used to divide a variable into
specific groups.
Examples of categorical variables include race, sex, age group,
and educational level.
The categorical data may also be called discrete data, implying that it
represents a finite number of values with no continuum between them.
Even if the values used for the categorical (or discrete)
variables are numeric, these numbers are nothing more than
symbols and do not imply the possibility of calculating
fractional values.
b) Nominal data contain measurements of simple codes assigned to objects as labels,
which are not measurements. For example, the variable marital status
can be generally categorized as (1) single, (2) married, and (3)
divorced.
c) Ordinal data contain codes assigned to objects or events as labels that also represent
the rank order among them.
For example, the variable credit score can be generally categorized as
(1) low, (2) medium, or (3) high.
d) Numeric data represent the numeric values of specific variables.
Examples of numerically valued variables include age, number of
children, total household income (in U.S. dollars), travel distance
(in miles), and temperature (in Fahrenheit degrees).
Numeric values representing a variable can be integer (taking only
whole numbers) or real (taking also the fractional number).
e) Interval data are variables that can be measured on interval scales.
A common example of interval scale measurement is temperature on
the Celsius scale. In this particular scale, the unit of measurement is
1/100 of the difference between the melting temperature and the
boiling temperature of water in atmospheric pressure; that is, there is
not an absolute zero value.
f) Ratio data include measurement variables commonly found in the physical
sciences and engineering.
Mass, length, time, plane angle, energy, and electric charge are
examples of physical measures that are ratio scales.
4. Data Preprocessing
Real-world data is usually dirty, misaligned, overly complex, and inaccurate.
