LECTURE NOTES
ON
DATA PREPARATION AND ANALYSIS
(BCSB13)
Prepared by,
G. Sulakshana, Assistant Professor, CSE Dept.
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
INSTITUTE OF AERONAUTICAL ENGINEERING
(Autonomous)
Dundigal, Hyderabad- 500043
, MODULE -I
DATA GATHERING AND PREPARATION
BIG DATA ANALYTICS:
The volume of data that one has to deal has exploded to unimaginable levels in the past decade,
and at the same time, the price of data storage has systematically reduced. Private companies
and research institutions capture terabytes of data about their users‘ interactions, business, social
media, and also sensors from devices such as mobile phones and automobiles. The challenge of
this era is to make sense of this sea of data. This is where big data analytics comes into picture.
Big Data Analytics largely involves collecting data from different sources, mange it in a way
that it becomes available to be consumed by analysts and finally deliver data products useful to
the organization business.
,Big Data Analytics - Data Life Cycle: Traditional Data Mining Life Cycle:
In order to provide a framework to organize the work needed by an organization and deliver
clear insights from Big Data, it‘s useful to think of it as a cycle with different stages. It is by no
means linear, meaning all the stages are related with each other. This cycle has superficial
similarities with the more traditional data mining cycle as described in CRISP methodology.
CRISP-DM Methodology:
The CRISP-DM methodology that stands for Cross Industry Standard Process for Data Mining
is a cycle that describes commonly used approaches that data mining experts use to tackle
problems in traditional BI data mining. It is still being used in traditional BI data mining teams.
Take a look at the following illustration. It shows the major stages of the cycle as described by
the CRISP-DM methodology and how they are interrelated.
CRISP-DM was conceived in 1996 and the next year, it got underway as a European Union
project under the ESPRIT funding initiative. The project was led by five companies: SPSS,
Terradata, Daimler AG, NCR Corporation, and OHRA (an insurance company). The project
was finally incorporated into SPSS. The methodology is extremely detailed oriented in how a
data mining project should be specified.
Let us now learn a little more on each of the stages involved in the CRISP-DM life cycle −
Business Understanding − This initial phase focuses on understanding the project
objectives and requirements from a business perspective, and then converting this
knowledge into a data mining problem definition. A preliminary plan is designed to
, achieve the objectives. A decision model, especially one built using the Decision Model
and Notation standard can be used.
Data Understanding − The data understanding phase starts with an initial data
collection and proceeds with activities in order to get familiar with the data, to identify
data quality problems, to discover first insights into the data, or to detect interesting
subsets to form hypotheses for hidden information.
Data Preparation − The data preparation phase covers all activities to construct the final
dataset (data that will be fed into the modeling tool(s)) from the initial raw data. Data
preparation tasks are likely to be performed multiple times, and not in any prescribed
order. Tasks include table, record, and attribute selection as well as transformation and
cleaning of data for modeling tools.
Modeling − In this phase, various modeling techniques are selected and applied and their
parameters are calibrated to optimal values. Typically, there are several techniques for
the same data mining problem type. Some techniques have specific requirements on the
form of data. Therefore, it is often required to step back to the data preparation phase.
Evaluation − At this stage in the project, you have built a model (or models) that
appears to have high quality, from a data analysis perspective. Before proceeding to
final deployment of the model, it is important to evaluate the model thoroughly and
review the steps executed to construct the model, to be certain it properly achieves the
business objectives.
A key objective is to determine if there is some important business issue that has not
been sufficiently considered. At the end of this phase, a decision on the use of the data
mining results should be reached.
Deployment − Creation of the model is generally not the end of the project. Even if the
purpose of the model is to increase knowledge of the data, the knowledge gained will
need to be organized and presented in a way that is useful to the customer.
Depending on the requirements, the deployment phase can be as simple as generating a
report or as complex as implementing a repeatable data scoring (e.g. segment allocation)
or data mining process.
In many cases, it will be the customer, not the data analyst, who will carry out the deployment
steps. Even if the analyst deploys the model, it is important for the customer to understand
upfront the actions which will need to be carried out in order to actually make use of the created
models.
ON
DATA PREPARATION AND ANALYSIS
(BCSB13)
Prepared by,
G. Sulakshana, Assistant Professor, CSE Dept.
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
INSTITUTE OF AERONAUTICAL ENGINEERING
(Autonomous)
Dundigal, Hyderabad- 500043
, MODULE -I
DATA GATHERING AND PREPARATION
BIG DATA ANALYTICS:
The volume of data that one has to deal has exploded to unimaginable levels in the past decade,
and at the same time, the price of data storage has systematically reduced. Private companies
and research institutions capture terabytes of data about their users‘ interactions, business, social
media, and also sensors from devices such as mobile phones and automobiles. The challenge of
this era is to make sense of this sea of data. This is where big data analytics comes into picture.
Big Data Analytics largely involves collecting data from different sources, mange it in a way
that it becomes available to be consumed by analysts and finally deliver data products useful to
the organization business.
,Big Data Analytics - Data Life Cycle: Traditional Data Mining Life Cycle:
In order to provide a framework to organize the work needed by an organization and deliver
clear insights from Big Data, it‘s useful to think of it as a cycle with different stages. It is by no
means linear, meaning all the stages are related with each other. This cycle has superficial
similarities with the more traditional data mining cycle as described in CRISP methodology.
CRISP-DM Methodology:
The CRISP-DM methodology that stands for Cross Industry Standard Process for Data Mining
is a cycle that describes commonly used approaches that data mining experts use to tackle
problems in traditional BI data mining. It is still being used in traditional BI data mining teams.
Take a look at the following illustration. It shows the major stages of the cycle as described by
the CRISP-DM methodology and how they are interrelated.
CRISP-DM was conceived in 1996 and the next year, it got underway as a European Union
project under the ESPRIT funding initiative. The project was led by five companies: SPSS,
Terradata, Daimler AG, NCR Corporation, and OHRA (an insurance company). The project
was finally incorporated into SPSS. The methodology is extremely detailed oriented in how a
data mining project should be specified.
Let us now learn a little more on each of the stages involved in the CRISP-DM life cycle −
Business Understanding − This initial phase focuses on understanding the project
objectives and requirements from a business perspective, and then converting this
knowledge into a data mining problem definition. A preliminary plan is designed to
, achieve the objectives. A decision model, especially one built using the Decision Model
and Notation standard can be used.
Data Understanding − The data understanding phase starts with an initial data
collection and proceeds with activities in order to get familiar with the data, to identify
data quality problems, to discover first insights into the data, or to detect interesting
subsets to form hypotheses for hidden information.
Data Preparation − The data preparation phase covers all activities to construct the final
dataset (data that will be fed into the modeling tool(s)) from the initial raw data. Data
preparation tasks are likely to be performed multiple times, and not in any prescribed
order. Tasks include table, record, and attribute selection as well as transformation and
cleaning of data for modeling tools.
Modeling − In this phase, various modeling techniques are selected and applied and their
parameters are calibrated to optimal values. Typically, there are several techniques for
the same data mining problem type. Some techniques have specific requirements on the
form of data. Therefore, it is often required to step back to the data preparation phase.
Evaluation − At this stage in the project, you have built a model (or models) that
appears to have high quality, from a data analysis perspective. Before proceeding to
final deployment of the model, it is important to evaluate the model thoroughly and
review the steps executed to construct the model, to be certain it properly achieves the
business objectives.
A key objective is to determine if there is some important business issue that has not
been sufficiently considered. At the end of this phase, a decision on the use of the data
mining results should be reached.
Deployment − Creation of the model is generally not the end of the project. Even if the
purpose of the model is to increase knowledge of the data, the knowledge gained will
need to be organized and presented in a way that is useful to the customer.
Depending on the requirements, the deployment phase can be as simple as generating a
report or as complex as implementing a repeatable data scoring (e.g. segment allocation)
or data mining process.
In many cases, it will be the customer, not the data analyst, who will carry out the deployment
steps. Even if the analyst deploys the model, it is important for the customer to understand
upfront the actions which will need to be carried out in order to actually make use of the created
models.
