SCHOOL OF COMPUTING

DEPARTMENT OF COMPUTER SCIENCE ENGINEERING

UNIT – I – SCSA1504 – BIG DATA ANALYTICS FOR AI

1
UNIT 1 INTRODUCTION TO BIG DATA 9 Hrs.

Introduction - Challenges of Conventional Systems, What is Business Analytics,

Business Analytics lifecycle, Why Big Data Analytics, Defining Big data,
Characteristics of Big Data(V’s), Business Analytics phases: Data Acquisition- Data
Cleaning - Data Manipulation - Data Analysis (Statistical and Analytical methods) to
make sense of data - Data Visualization

1.1 CHALLENGESOFCONVENTIONALSYSTEMS
Introduction to ConventionalSystems
What is Conventional System?
 The system consists of one or more zones each having either manually operated call
points or automatic detection devices, or a combination of both.
 Big data is huge amount of data which is beyond the processing capacity of conventional
data base systems to manage and analyze the data in a specific time interval.
Difference between conventional computing and intelligent computing
 The conventional computing functions logically with a set of rules and calculations while
the neural computing can function via images, pictures, and concepts.
 Conventional computing is often unable to manage the variability of data obtained in the
real world.
 On the other hand, neural computing, like our own brains, is well suited to situations that
have no clear algorithmic solutions and are able to manage noisy imprecise data. This
allows them to excel in those areas that conventional computing often finds difficult.
Comparison of Big Data with ConventionalData
Big Data Conventional Data
Huge data sets Data set size in control.
Unstructured data such as text, video, and Normally structured data such asnumbers and
audio. categories, but it can take other forms aswell.
Hard-to-perform queries and analysis Relatively easy-to-perform queries and
analysis.
Needs a new methodology for analysis. Data analysis can be achieved by using
conventional methods.
Need tools such as Hadoop, Hive, Tools such as SQL, SAS, R, and Excel
Hbase, Pig, Sqoop, and so on. alone may be sufficient.
The aggregated or sampled or filtered data. Raw transactional data.

2
Used for reporting, basic analysis, and Used for reporting, advanced analysis, and
text mining. Advanced analytics is only in a predictive modeling.
starting stage in big data.
Big data analysis needs both programming Analytical skills are sufficient for
skills (such as Java) and analytical skills to conventional data; advanced analysis tools
perform analysis. don’t require expert programing skills.
Petabytes/exabytes of data. Millions/billions of accounts.
Billions/trillions of transactions. Megabytes/gigabytes of data.
Thousands/millions of accounts. Millions of transactions
Generated by big financial institutions, Generated by small enterprises and small
Facebook, Google, Amazon, eBay, banks.
Walmart, and so on.

List of challenges of ConventionalSystems

The following list of challenges has been dominating in the case Conventional systems in real
time scenarios:
1. Uncertainty of Data Management Landscape
2. The Big Data Talent Gap
3. The talent gap that exists in the industry Getting data into the big data platform
4. Need for synchronization across data sources
5. Getting important insights through the use of big data analytics
Other Three challenges of Conventional systems
 Three Challenges That big data face.
• Data
• Process
• Management
Data Challenges – Volume
 The volume of data, especially machine-generated data, is exploding,how fast that data is
growing every year, with new sources of data that are emerging.
 For example, in the year 2000, 800,000petabytes (PB) of data were stored in the world,
and it is expected to reach 35 zetta bytes (ZB) byB2020 (according to IBM).
 Social media plays a key role: Twitter generates 7+ terabytes (TB) of data every day.
Facebook, 10 TB.
 Mobile devices play a key role as well, as there were estimated 6 billion mobile phones
in 2011.
3
 The challenge is how to deal with the size of Big Data.
• Variety, Combining Multiple Data Sets
• More than 80% of today’s information is unstructured and it is typically too big to
manage effectively.
• Today, companies are looking to leverage a lot moredata from a wider variety of
sources both inside and outside the organization.
• Things like documents, contracts, machine data, sensor data, social media, health
records, emails, etc. The list is endless really.
• A lot of this data is unstructured, or has a complex structure that’s hard to represent
in rows and columns.
Processing
 More than 80% of today’s information isunstructured and it is typically too big to
manage effectively.
 Today, companies are looking to leverage a lot more data from a wider variety of sources
both inside and outside the organization.
 Things like documents, contracts, machine data, sensor data, social media, health
records, emails, etc. The list is endless really.
Management
 A lot of this data is unstructured, or has acomplex structure that’s hard to represent in
rows and columns.
 Big Data ChallengesThe challenges include capture, duration, storage, search, sharing,
transfer,analysis, and visualization.
 Big Data is trend to larger data setsdue to the additional information derivable from
analysis of a single large set of related data, as compared to separate smaller sets with the
same total amount of data, allowing correlations to be found to"spot business trends,
determine quality of research, prevent diseases, link legal citations, combat crime, and
determine real-time roadway traffic conditions.”

4
Business Analytics Life cycle

1.2 Business Analytics Life cycle

What is business analytics?

Business analytics (BA) is a set of disciplines and technologies for solving business problems
using data analysis, statistical models and other quantitative methods. It involves an iterative,
methodical exploration of an organization's data, with an emphasis on statistical analysis, to
drive decision-making.

Data-driven companies treat their data as a business asset and actively look for ways to turn it
into a competitive advantage. Success with business analytics depends on data quality, skilled
analysts who understand the technologies and the business, and a commitment to using data
to gain insights that inform business decisions.

How business analytics works

Before any data analysis takes place, BA starts with several foundational processes:

 Determine the business goal of the analysis.

 Select an analysis methodology.

 Get business data to support the analysis, often from various systems and sources.

 Cleanse and integrate data into a single repository, such as a data

warehouse or data mart.

5
1.3 Three types of Business Analytics
Types of business analytics

Different types of business analytics include the following:

 descriptive analytics, which tracks key performance indicators (KPIs) to

understand the present state of a business;

 predictive analytics, which analyzes trend data to assess the likelihood of future
outcomes; and

 prescriptive analytics, which uses past performance to generate recommendations

for handling similar situations in the future.

6
 

1. Descriptive analytics:

 This first type of analytics provides the facts stating what has happened. It is
the simplest type that “allows you to condense big data into smaller, more
useful nuggets of information.”

 The objective is to summarize the results and to understand what is going on.

7
  Descriptive analytics helps a business learn from its past behavior and how it
will impact the future.

 It provides information that helps to understand the performance of the

business on an overall aggregate level.

 It is also an important step in explaining the current raw data to the various
stakeholders.

 This is more like a management information system (MIS), where an MIS

gathers data from multiple online systems, analyzes the information, and
reports data to aid in management decision-making. The key techniques used
are data aggregation and data mining to summarize the past data of
understanding the data’s underlying behavior and not for any predictions.

 These are the start of the data analytics value chain and are the most valuable
to uncover any patterns.
 A simple example of descriptive analytics is to assess credit risk: we can
predict a consumer’s likely financial riskiness by seeing their balance amount
against the credit limit. It is also used to analyze the sales cycle of a store.
Also, it can be used to categorize the customers based on their product
preferences, purchase transactions, how often they purchase. In the context of
social media, descriptive analytics offers an overview of the performance
metrics: the total of posts, mentions, followers, comments, page views,
reviews, the average time is taken to respond, and so on.

2. Predictive analytics:

 Predictive analytics uses the gathered data and descriptive and diagnostic analytics
results to tell what is likely to happen in the future on a granular level.

 This is where the earlier steps’ insights can be used into actionable insights for
decision-making. Its use involves forecasting the future, predicting the market

8
 trends, changing customer behaviors, and analyzing competitors to optimize and
build strategies to maximize the business results.

 The predictions are made by analyzing the past data, detecting patterns, casual
relationships in the data, and then extrapolating them in the future. For instance, a
bank to predict which customer is likely to default will need all the past data about
which customers have defaulted to predict. The inferential statistics, training
algorithms for regression, classification, and segmentation come under this type of
business analytics. It uses the techniques to segment the data into groups, apply
clustering methods, heuristic rules, decision trees to project future outcomes.

 The predictive analysis can also be used to generate, test, and evaluate hypotheses.
It is useful to understand whether a set of features are explaining or predicting
other features. For example, it can validate a person’s hypothesis inhibiting from a
certain region, age group, gender defaults in its credit card payments. This is
especially useful when some of the features are actions determined by the business
decision-makers. One of the applications of prescriptive analysis is sentiment
analysis.

 Predictive analytics is extensively used in every industry: finance, healthcare,

social media, sports, energy, manufacturing. One of the most frequent applications
is in retail, where the retailers are always using predictive analytics to predict and
improve their sales positions. Amazon’s recommendation engines are a classic
example where on making one purchase, the engine shows the list of other similar
items that the buyers have purchased

3. Prescriptive analytics:

 These analytics reveal why you should take a particular action. Prescriptive analytics
enable optimization, simulation, decision modeling and provide the best possible
analysis for business decisions and actions.

 Building on predictive analytics, prescriptive analysis is the next step that helps in
exploiting the future. It essentially tells the business what should be done. Using

9
 simulation and optimization, it advises on the possible outcomes and suggests actions
that can maximize the key business metrics. The focus here is on how to make it
happen.

 It can be defined as a business optimization data analytics that provides insights on

“what should a business do” to solve a problem. It explores several possible outcomes
and suggests actions depending on the results of descriptive, diagnostic, and
predictive analytics of the data. The prescriptive analysis uses a feedback system that
constantly learns, updates the relationship between the action and the results.

 It does not predict one possible future but rather multiple future outcomes. It is an
advanced analytics concept based on optimization and simulation. Optimization helps
understand how to achieve the best outcome and identify the data uncertainties to
make better decisions. The other approach of prescriptive analytics is a simulation in
which all the key performance areas are combined to design the correct metric goals.
This ensures whether the key performance metrics are included in the solution.

 The prescriptive analysis is performed when scenario analysis simulates the future
under various sets of assumptions and combines it with different optimization
techniques. It uses statistical models and machine learning algorithms to estimate the
probabilities, optimizing and recommending actions. A prescriptive model can
recommend the best course of action for any pre-specified outcome as it can predict
the possible results based on a different choice of action. Waymo, Google’s self-
driving car, is an example of prescriptive analytics. Recommendation engines are a
use case of prescriptive analysis.

 Prescriptive analytics can be applied to almost any industry where the population is to
be targeted or grouped. It also has its applications in marketing, financial markets, and
the transportation industry. In social media, grouping the customers under one bucket
based on their characteristics can use prescriptive analytics to optimize the offering to
each group. Similarly, it can be applied in the transportation sector to save time and
resources by optimizing the best routes. In the financial markets, the researchers
heavily rely on statistical modeling to maximize the returns and manage risk and
profitability.
10
1.4 Benefits of business analytics

Business analytics benefits impact every corner of your organization. When data across
departments consolidates into a single source, it syncs up everyone in the end-to-end process.
This ensures there are no gaps in data or communication, thus unlocking benefits such as:

Data-driven decisions: With business analytics, hard decisions become smarter—and by

smart, that means that they are backed up by data. Quantifying root causes and clearly
identifying trends creates a smarter way to look at the future of an organization, whether it be
HR budgets, marketing campaigns, manufacturing and supply chain needs, or sales outreach
programs.

Easy visualization: Business analytics software can take unwieldy amounts of data and turn
it into simple-yet-effective visualizations. This accomplishes two things. First, it makes
insights much more accessible for business users with just a few clicks. Second, by putting
data in a visual format, new ideas can be uncovered simply by viewing the data in a different
format.

Modeling the what-if scenario: Predictive analytics creates models for users to look for
trends and patterns that will affect future outcomes. This previously was the domain of
experienced data scientists, but with business analytics software powered by machine
learning, these models can be generated within the platform. That gives business users the
ability to quickly tweak the model by creating what-if scenarios with slightly different
variables without any need to create sophisticated algorithms.

Go augmented: All of the points above consider the ways that business data analytics
expedite user-driven insights. But when business analytics software is powered by machine
learning and artificial intelligence, the power of augmented analytics is unlocked. Augmented
analytics uses the ability to self-learn, adapt, and process bulk quantities of data to automate
processes and generate insights without human bias.

1.5 Business Analytics Components

11
1. Define Objective

This is the foremost step. Without having a clear understanding of business goals, questions
we need to answer, and problems we ought to solve, none of the following steps will deliver.
This also helps us to translate business objectives into analytics objective and map data
requirements.

2. Data Aggregation

The process of having a centralized location for the data, extracting and loading the relevant
data by putting relevant filters, and creating subsets of data is the core aspect of Data
Aggregation. This is where the data is transformed depending upon the business requirement.
Also data pertaining from various sources are combined to have one large dataset. The format
of the data is also sometimes changed at this step to make it compatible with the tool being
used to achieve the objectives.

3. Data Cleaning

Data Cleaning is an extremely important component of business analytics because the data in
its raw form sometimes is not directly usable. As the other components of Business Analytics

12
use mathematics, statistics, and computer programming, the data must be compatible with
these streams of study.

For example, for applying statistics, the data mustn’t have any extreme values (also known as
outliers), while for mathematics, there should no blank cells or missing values (as matrix
operations become difficult) while for programming, the concept of type casting plays a role
where the data is made sure to be in the right format (i.e. correct class or data type).

Also, the concepts of multicollinearity and curse of dimensionality come in play as the
business analyst has to make sure that there are no implicit or explicit duplicate columns. The
importance of getting rid of the unnecessary columns can only be understood once a good
grasp of statistics is there. Other aspects include the resampling of data (under-sampling,
oversampling, hybrid-sampling) removal of duplicate rows, etc.

4. Analytical Methodology

Having a detailed understanding of the different type of analytics out there dominate this
component as this is where the analysts have to identify the method with which they will go
to achieve their end goal. If the end goal is to understand what is the present situation of the
business then that requires a different set of methods while if there is a need to identify which
has happened in the past or what can happen in the future, then a different technique is
required. Here, having the know-how of various procedures, methods, and algorithms is
important, and knowing what to use, when makes one business analyst stand apart from the
other.

13
5. Evaluation and Validation

Once the results come out, the next task is to understand if the result stands true given a
different situation or not. This is where is predictive models are used and their evaluation and
validation are conducted whereas, for other methods, various simulation techniques are put to
use to identify the most plausible outcome, thus providing provide a very reliable result. Here
also, the business analyst needs to learn a range of techniques to identify the shortcoming in
their method, work on it, improve it, and make their insights stable and valuable.

6. Reporting and Data Visualization

Perhaps the most important and often overlooked component of the discipline is the aspect of
communicating the results in an easy to understand way. This requires innovation and
creativity and is the reason that this field is open to all and not only to mathematicians,
statisticians, or computer programmers. To quickly make people understand the complex
insights discovered over weeks or even months, reports or presentations are created that have
simple tables, bullet points, etc. On top of all this, visualizing the data plays a major role here
as it allows the people in the leadership positions to quickly view where the organization is
coming from and perhaps where they are headed. Business Analytics ought to know the
various ways of visualizing the data, the transformation required to be done on the data to
make it possible, and finding innovative ways to string together different information in a
smooth storytelling method.

1.5 Difference Between Business Intelligence vs Business Analytics

Business Intelligence is the process comprising of technologies and strategies incorporated by

the enterprise industries to analyze the existing business data which provides past (historical),
current and predictive events of the business operations. Business Analytics is the process of
technologies and strategies used to continue exploring and to extract the insights and
performance from the past business information to drive successful future business planning.
14
Following is the difference between Business Intelligence and Business Analytics are as
follows:

1. Business Intelligence uses past and current data whereas Business Analytics uses past
data to extract insights and run the business operations that drive the customer needs
and increase productivity.
2. Business Intelligence mostly concentrates on reporting the analyzed data whereas
Business Analytics concentrates on multiple tools that perform different operational
applications using different tools.
3. Business Intelligence almost comes under Business Analytics where Business
Analytics contains Business Intelligence, data warehousing, information management,
enterprise applications and governance, risk and security compliances.
4. Business Intelligence is the way of analyzing the existing data whereas Business
Analytics will have Business Intelligence reports acts as inputs for the analytics to
process the extracted information in a more sophisticated way to visualize the
analyzed data.
5. Business Intelligence uses statistical analysis, predictive analysis, and predictive
modeling to set the current trends and figure out the reasons for current outcomes or
happenings whereas Business Analytics have no control over huge amounts of Data to
retrieve, analyze, report and publish the data.
6. Business Intelligence consists more as User Interface Dashboards to carry out the
analysis and operations whereas Business Analytics has a lot of tools to work upon
and that also needs some software application knowledge to carry out the tasks to be
done.
7. Business Intelligence gives insights or information about the data itself rather than
making extra transformations or conversions to give data insights and on the other
side Business Analytics involves the way of problem-solving by enabling the
technologies by transforming the raw form of data into a meaningful way to convey
the solution in an easy way.
8. Business Intelligence can be applied more to structured data from enterprise
applications such as Financial Software Systems or Enterprise Resource Planning
(ERP) to get insight from the past financial information or from the past financial

15
 transactions and in the areas of supply chain and operations. Business Analytics can
be applied to both unstructured and semi-structured data by transforming them into
some meaningful data before analyzing it to get insights from that data.
9. Business Intelligence consumes the data in the same format to get insights out of it
whereas Business Analytics transforms or breaks the existing data into different forms
or elements and studying them as a whole to get some insights out of it.
10. In Business Intelligence, the data can be produced in the form of Dashboards, reports
or pivot tables for different users like executives, managers and for analysts
respectively whereas Business Analytics uses past Business Intelligence capabilities
and information to help the customers’ highly productive in getting their jobs done.
11. Business Intelligence is the content of data what you are having with you whereas
Business Analytics is the way how you are using or operating on that data to get your
insights of out of that data.
12. Business Intelligence is all about accessing the big data Business Analytics is the use of
different latest technological methodologies to handle the big data.
13. Business Intelligence is used to run the businesses effectively whereas Business
Analytics is the way of changing the business to make it more productive and
operations effective.
14. As Business Intelligence being the subset of Business Analytics and the benefits of
Business Analytics are causing BA to get more popular and drawing attention from
business users to get more useful things out of it.
15. Business Intelligence incorporates different tools and methodologies for use in the
stages of data analysis in which the common types of tools include, data
reporting, Real-Time analysis, Mapping Analysis, Online Analytical Processing,
Dashboarding, etc., and Business Analytics incorporates different stages and phases of
analyses such as SWOT analysis, use case modelling, predictive modelling, data
modelling user stories, requirement analysis, functional requirement and non-functional
requirement analysis etc.,

Business Intelligence and Business Analytics Comparison Table

16
1.6 INTRODUCTION TO BIGDATA PLATFORM
Data andInformation
 Data are plainfacts.
 The word "data" is plural for"datum."
 Data is nothing but facts and statistics stored or free flowing over a network, generally
it's raw andunprocessed.
 When data are processed, organized, structured or presented in a given context so as to
make them useful, they are calledInformation.
 It is not enough to have data (such as statistics on theeconomy).
 Data themselves are fairly useless, but when these data are interpreted and processed to
determine its true meaning, they become useful and can be calledInformation.
For example: When you visit any website, they might store you IP address, that is data,
in return they might add a cookie in your browser, marking you that you visited the
website, that is data, your name, it's data, your age, it's data.
What is Data?
 The quantities, characters, or symbols on which operations are performed by a computer,
 which may be stored and transmitted in the form of electrical signals andrecorded on
magnetic, optical, or mechanical recording media.
3 Actions on Data
 Capture
 Transform
 Store
BigData
 Big Data may well be the Next Big Thing in the IT world.
 Big data burst upon the scene in the first decade of the 21st century.
 The first organizations to embrace it were online and startup firms.
 Firms like Google, eBay, LinkedIn, and Facebook were built around big data from the
beginning.
 Like many new information technologies,
 big data can bring about dramatic cost reductions,
 substantial improvements in the time required to perform a computing task, ornew
product and service offerings.
17
 Walmart handles more than 1 million customer transactions every hour.
 Facebook handles 40 billion photos from its user base.
 Decoding the human genome originally took 10years to process; now it can be achieved
in one week.
What is Big Data?
 Big Data is also data but with a huge size.
 Big Data is a term used to describe a collection of data that is huge in size and yet
growing exponentially with time.
 In short such data is so large and complex that none of the traditional data management
tools are able to store it or process it efficiently.
No single definition; here is from Wikipedia:
 Big data is the term fora collection of data sets so large and complex that it becomes
difficult to process using on-hand database management tools or traditional data
processing applications.
Examples of Bigdata: Following are some the examples of Big Data
 The New York Stock Exchange generates about one terabyte of new trade data per day.
 Other examples of Big Data generation include
• stock exchanges,
• social media sites,
• jet engines,
• etc.
Types of Big Data
BigData could be found in threeforms:
1. Structured
2. Unstructured
3. Semi-structured
What is Structured Data?
 Any data that can be stored, accessed and processed in the form of fixed format is termed
as a 'structured' data.
 Developed techniques for working with such kind of data (where the format is well
known in advance) and also deriving value out of it.
 Foreseeing issues of today:when a size of such data grows to a huge extent, typical sizes
18
 are being in the rage of multiple zetta bytes.
 1021 bytes equal to 1 zettabyte or one billion terabytes forms a zettabyte.
 That is why the name Big Data is given and imagines the challenges involved in its
storage and processing?
 Data stored in a relational database management system is one example of a 'structured'
data.
 An 'Employee' table in a database is an example of Structured Data:

Employee_ID Employee_Name Gender Department Salary_In_lacs

2365 Rajesh Kulkarni Male Finance 650000

3398 Pratibha Joshi Female Admin 650000

7465 Shushil Roy Male Admin 500000

7500 Shubhojit Das Male Finance 500000

7699 Priya Sane Female Finance 550000

Unstructured Data
 Any data with unknown form or the structure is classified as unstructured data.
 In addition to the size being huge,un-structured data poses multiple challenges in terms
of its processing for deriving value out of it.
 A typical example of unstructured data isa heterogeneous data source containing a
combination of simple text files, images, videos etc.
 Now day organizations have wealth of data available with them but unfortunately,they
don't know how to derive value out of it since this data is in its raw form or unstructured
format.
Example of Unstructured data:The output returned by 'GoogleSearch'
Semi-structured Data
 Semi-structured data can contain both the forms of data.
 Semi-structured data as a structured in formbut it is actually not defined with e.g. a table
definition in relational DBMS.
 Example of semi-structured data isa data represented in an XML file.
 Personal data stored in an XML file.
<rec>

19
 <name>Prashant Rao</name>
<sex>Male</sex>
<age>35</age>
</rec>
<rec>
1.7 Characteristics of BD OR 3Vs of Big Data
 Three Characteristics of Big Data V3s:
• VolumeData quantity
• VelocityData Speed
• VarietyData Types
Growth of Big Data

Storing Big Data

 Analysing your data characteristics
• Selecting data sources for analysis
• Eliminating redundant data
• Establishing the role of NoSQL
 Overview of Big Data stores
• Data models: key value, graph, document, column-family
• Hadoop Distributed File System (HDFS)
• Hbase

20
 • Hive
Processing Big Data
 Integrating disparate data stores
• Mapping data to the programming framework
• Connecting and extracting data from storage
• Transforming data for processing
• Subdividing data in preparation for Hadoop MapReduce
 Employing Hadoop MapReduce
• Creating the components of Hadoop MapReduce jobs
• Distributing data processing across server farms
• Executing Hadoop MapReduce jobs
• Monitoring the progress of job flows
Why Big Data?
 Growth of Big Data is needed
• Increase of storage capacities
• Increase of processing power
• Availability of data(different data types)
• Every day we create 2.5 quintillion bytes of data; 90% of the data in
the world today has been created in the last two years alone
 Huge storage needs in Real Time Applications
• FB generates 10TB daily
• Twitter generates 7TB of data Daily
• IBM claims 90% of today’s stored data was generated in just the last
two years.
How Is Big Data Different?
1. Automatically generated by a machine (e.g. Sensor embedded in an engine)
2. Typically, an entirely new source of data(e.g. Use of the internet)
3. Not designed to be friendly(e.g. Text streams)
4. May not have much values – Need to focus on the importantpart
Big Data sources
 Users
 Application

21
 Systems
 Sensors
 Moved to
 Large and growing files (Big data files)
Risks of Big Data
 Will be so overwhelmed (Need the right people and solve the right problems)
 Costs escalate too fast (Isn’t necessary to capture 100%)
 Many sources of big data is privacy (self-regulation, Legal regulation)
Leading Technology Vendors

Example Vendors Commonality

IBM – Netezza • MPParchitectures

EMC – Greenplum • CommodityHardware

Oracle – Exadata • RDBMS based

• Full SQLcompliance
Big data is a term for data sets that are so large or complex that traditional data
processing applications are inadequate.
Challenges include
 Analysis,
 Capture,
 Data Curation,
 Search,
 Sharing,
 Storage,
 Transfer,
 Visualization,
 Querying,
 Updating
Information Privacy.
 The term often refers simply to the use of predictive analytics or certain other
 Advanced methods to extract value from data, and seldom to a particular size of data set.
 ACCURACY in big data may lead to more confident decision making, and better
22
 decisions can result in greater operational efficiency, cost reduction and reduced risk.
 Big data usually includes data sets with sizes beyond the ability of commonly
usedsoftware tools to capture, curate, manage, and process data within a tolerable
elapsedtime. Big data "size" is a constantly moving target.
 Big data requires a set of techniques and technologies with new forms of integration
toreveal insights from datasets that are diverse, complex, and of a massive scale
List of Big Data Platforms
 Hadoop
 Cloudera
 Amazon Web Services

1.8 Challenges of Big Data

The following are the five most important challenges of the Big Data
Meeting the need for speed
 In today’s hypercompetitive business environment, companies not only have to find and
analyze the relevant data they need, they must find it quickly.
 Visualization helps organizations perform analyses and make decisions much more
rapidly, but the challenge is going through the sheer volumes of data and accessing the
level of detail needed, all at a high speed.
 The challenge only grows as the degree of granularity increases. One possible solution is
hardware. Some vendors are using increased memory and powerful parallelprocessing to
crunch large volumes of data extremely quickly
Understanding the data
 It takes a lot of understanding to get data in the RIGHT SHAPE so that you can
usevisualization as part of data analysis.
Addressing data quality
 Even if you can find and analyze data quickly and put it in the proper context for
theaudience that will be consuming the information, the value of data for DECISION-
MAKING PURPOSES will be jeopardized if the data is not accurate or timely.
 This is a challenge with any data analysis.
Displaying meaningful results
 Plotting points on a graph for analysis becomes difficult when dealing with extremely
23
 large amounts of information or a variety of categories of information.
 For example, imagine you have 10 billion rows of retail SKU data that you’re trying to
compare. The user trying to view 10 billion plots on the screen will have a hard
timeseeing so many data points.
 By grouping the data together, or “binning,” you can more effectively visualize the data.
Dealing with outliers
 The graphical representations of data made possible by visualization can communicate
trends and outliers much faster than tables containing numbers and text.
 Users can easily spot issues that need attention simply by glancing at a chart. Outliers
typically represent about 1 to 5 percent of data, but when you’re working with massive
amounts of data, viewing 1 to 5 percent of the data is rather difficult
 We can also bin the results to both view the distribution of data and see the outliers.
 While outliers may not be representative of the data, they may also reveal previously
unseen and potentially valuable insights.
 Visual analytics enables organizations to take raw data and present it in a meaningful
way that generates the most value. However, when used with big data, visualization is
bound to lead to some challenges.

DATA CONVERSION
 We can convert or transform our data from ratio to interval to ordinal to nominal.
However, we cannot convert or transform our data from nominal to ordinal to interval to
ratio.
 Scaled data can be measured in exact amounts.
 For example, 60 degrees, 12.5 feet, 80 Miles per hour
 Scaled data can be measured with equal intervals.
 For example, between 0 and 1 is 1 inch, between 13 and 14 is also 1 inch
 Ordinal or ranked data provides comparative Amounts Example:
 1stPlace, 2ndPlace, 3rdPlace but Not equalintervals
1st Place 2nd Place 3rd Place
19.6 feet 18.2 feet 12.4 feet

DATA SELECTION
24
 Another Example that handles the question as:
What is the average driving speed of teenagers on the freeway?
a) Scaled
b) Ordinal
Scaled – Speed:- Speed can be measured in exact amounts with equal intervals.
Example :
60degrees 12.5feet 80 Miles per hour
 Ordinal or ranked data provides comparativeamounts.
Forexample, 1stPlace 2ndPlace 3rdPlace
 Percentiles provide comparativeamounts.
In this case, 93% of all hospital have lower patient satisfaction scores than Eastridge hospital.
31% have lower satisfaction scores than Westridge Hospital.
Thus the nature of data and its value have great influence on data insight in it.
1.9 ANALYTIC PROCESS AND TOOLS
• There are 6 analytic processes or phases:

1. Deployment
2. Business Understanding
3. Data Exploration
25
4. Data Preparation
5. Data Modeling
6. Data Evaluation
Step 1: Deployment
• Here we need to:
– plan the deployment and monitoring and maintenance,
– we need to produce a final report and review the project.
• In this phase,we deploy the results of the analysis. This is also known as reviewing the
project.
Step 2: Business Understanding
• Business Understanding
– The very first step consists of business understanding.
– Whenever any requirement occurs, firstly we need to determine the business objective,
– assess the situation, determine data mining goals and then produce the project plan as per the
requirement.
• Business objectives are defined in this phase.
Step 3: Data Exploration
• The second step consists of Data understanding.
– For the further process, we need to gather initial data, describe and explore the data and
verify data quality to ensure it contains the data we require.
– Data collected from the various sources is described in terms of its application and the need
for the project in this phase.
– This is also known as data exploration.
• This is necessary to verify the quality of data collected.
Step4:Data Preparation
• From the data collected in the last step,we need to select data as per the need, clean it,
construct it to get useful information and then integrate it all.
• Finally, we need to format the data to get the appropriate data.
• Data is selected, cleaned, and integrated into the format finalized for the analysis in this
phase.
Step5:DataModeling
– we need to select a modelling technique, generate test design, build a model and assess the
26
 model built.
– The data model is build to analyze relationships between various selected objects in the data,
– Test cases are built for assessing the model and modelists and implemented on the data in this
phase.
• Where processing is hosted?
– Distributed Servers / Cloud (e.g. AmazonEC2)
• Where data isstored?
– Distributed Storage (e.g. Amazon S3)
• What is the programming model?
– Distributed Processing (e.g. MapReduce)
• How data is stored &indexed?
– High-performance schema-free databases ([Link])
• What operations are performed on data?
– Analytic / Semantic Processing
• Big data tools for HPC and supercomputing
– MPI
• Big data tools onclouds
– MapReduce model
– Iterative MapReduce model
– DAG model
– Graph model
– Collective model
• Other BDA tools
– SaS
– R
– Hadoop
Thus the BDA tools are used throughout the BDA applications development.
ANALYSIS AND REPORTING
INTRODUCTION TO ANALYSIS ANDREPORTING
What is Analysis?
• The process of exploring data and reports
– in order to extract meaningful insights,
27
– which can be used to better understand and improve business performance.
• What is Reporting?
– Reporting is“the process of organizing data into informational summaries
– in order to monitor how different areas of a business are performing.”
COMPARING ANALYSIS WITHREPORTING
• Reporting is “the process of organizing data into informational summaries in order to
monitor how different areas of a business are performing.”
• Measuring core metrics and presenting them — whether in an email, a slide deck, or online
dashboard — falls under this category.
• Analytics is “the process of exploring data and reports in order to extract meaningful
insights, which can be used to better understand and improve business performance.”
• Reporting helps companies to monitor their online business and be alerted to when data falls
outside of expected ranges.
• Good reporting
• should raise questions about the business from its end users.
• The goal of analysisis
• to answer questions by interpreting the data at a deeper level and providing actionable
recommendations.
• A firm may be focused on the general area of analytics (strategy, implementation,
reporting,etc.)
– but not necessarily on the specific aspect of analysis.
• It’s almost like some organizations run out of gas after the initial set-up-related
activities and don’t make it to the analysis stage

A reporting activity deliberately proposes Analysis activity.

28
 CONTRAST BETWEEN ANALYSIS ANDREPORTING
The basis differences between Analysis and Reporting are as follows:
Analysis Reporting
Provides what is needed Provides what is asked for
Is typically customized Is Typically standardized
Involves a person Does not involve a person
Is extremely flexible Is fairly Inflexible
• Reporting translates raw data into information.
• Analysis transforms data and information into insights.
• reporting shows you what is happening
• while analysis focuses on explaining why it is happening and what you can do about it.
 Reports are like Robots n monitor and alter you and where as analysis is like parents - c an
figure out what is going on (hungry, dirty diaper, no pacifier, , teething, tired, ear infection,
etc).
 Reporting and analysis can go hand-in-hand:
 Reporting provides no limited context about what is happening in the data. Context is critical
to good analysis.
 Reporting translates a raw data into information
 Reporting usually raises a question – What is happening?
 Analysis transforms the data into insights - Why is it happening? What you can do about
it?
Thus, Analysis and Reporting is synonym to each other with respect their need and utilizing
in the needy context.

Why Use Data Manipulation Tools?

Data manipulation is critical for process efficiency and optimisation. Beyond processes, data
manipulation has a direct impact on what kind of business decisions get made because it
29
provides a way to see data more clearly within a big picture.

Data manipulation tools provide value to organisations in multiple ways. Data manipulation
tools deliver:

1. Data consistency

With data in a consistent format and structured, it becomes easier to analyse, interpret, and
read data. With data automation solutions businesses pull data from multiple and often
disparate sources, so the ability to format it in a unified way will add value for reporting.

2. Data removal

Since your business sources data from different locations, there’s a chance that you have
redundant data. Data manipulation tools will spot and remove redundancies so that your data
analysis is not negatively impacted.

3. Data projection

Data is at the heat of business intelligence, providing insights you need to make informed
decisions. When you can make use of historical data for future projections and forecasts, you
can use the past to your advantage.

4. Data interpretation

Complex data in a variety of formats poses a challenge for interpretation. With data
manipulation tools, the complexity is removed since data can be formatted accordingly. Once
it’s structured, it can be transformed into a visual experience so that it can actually be of
value to the person reviewing the data.

30