MASTERJI DEGREE AND PG COLLEGE

[Link](M)-DS-2-1(DATA SCIENCE-DATA ENGINEERING WITH PYTHON)

Unit 1 chapter 1 INTRODUCTION TO DATA SCIENCE

Data science is all about finding and exploring data in the real world and using thatknowledge to
solve business problems. Some examples of data science are:

 Customer Prediction - System can be trained based on customer behaviorpatterns to

predict the likelihood of a customer buying a product
 Service Planning - Restaurants can predict how many customers will visit onthe weekend
and plan their food inventory to handle the demand

When it comes to data science, we need some sort of programming language or tool,like Python.
Although there are other tools for data science, like R and SAS, we will focus on Python and how it is
beneficial for data science
Python is used as a programming language for data science because it contains costly tools from a
mathematical or statistical perspective. It is one of the significant reasonswhy data scientists around
the world use Python. If you track the trends over the pastfew years, you will notice that Python has
become the programming language of choice, particularly for data science.

There are several other reasons why Python is one of the most used programminglanguages for data
science, including:

 Speed - Python is relatively faster than other programming languages

 Availability - There are a significant number of packages available that otherusers have
developed, which can be reused
 Design goal - The syntax roles in Python are intuitive and easy to understand,thereby
helping in building applications with a readable codebase

APPLICATIONS OF DATA SCIENCE:

Applications that build upon the concepts of Data Science, exploring various domainssuch as the
following:

 Fraud and Risk Detection

 Healthcare
 Internet Search
 Targeted Advertising
 Website Recommendations
 Advanced Image Recognition
 Speech Recognition
 Airline Route Planning
 Gaming
 Augmented Reality
Fraud and Risk Detection

The earliest applications of data science were in Finance. Companies were fed up ofbad debts and
losses every year. However, they had a lot of data which use to get collected during the initial
paperwork while sanctioning loans. They decided to bring in data scientists in order to rescue them
from losses.

Over the years, banking companies learned to divide and conquer data via customer profiling, past
expenditures, and other essential variables to analyze the probabilities of risk and default. Moreover,
it also helped them to push their banking products basedon customer’s purchasing power.

Healthcare

The healthcare sector, especially, receives great benefits from data scienceapplications.

1. Medical Image Analysis

Procedures such as detecting tumors, artery stenosis, organ delineation employ various different
methods and frameworks like MapReduce to find optimal parameters for tasks like lung texture
classification. It applies machine learning methods, support vector machines (SVM), content-based
medical image indexing, and wavelet analysis for solidtexture classification.

2. Genetics & Genomics

Data Science applications also enable an advanced level of treatment personalization through research
in genetics and genomics. The goal is to understand the impact of the DNA on our health and find
individual biological connections between genetics, diseases, and drug response. Data science
techniques allow integration of different kinds of data with genomic data in the disease research,
which provides a deeper understanding of genetic issues in reactions to particular drugs and diseases.
As soon aswe acquire reliable personal genome data, we will achieve a deeper understanding of the
human DNA. The advanced genetic risk prediction will be a major step towards more individual care.

3. Drug Development
The drug discovery process is highly complicated and involves many disciplines. The greatest ideas
are often bounded by billions of testing, huge financial and time expenditure. On average, it takes
twelve years to make an official submission.

Data science applications and machine learning algorithms simplify and shorten this process,
adding a perspective to each step from the initial screening of drug compounds to the prediction of the
success rate based on the biological factors. Such algorithms can forecast how the compound will
act in the body using advanced mathematical modeling and simulations instead of the “lab
experiments”. The idea behind the computational drug discovery is to create computer model
simulations as a biologically relevant network simplifying the prediction of future outcomes with high
accuracy.

4. Virtual assistance for patients and customer support

Optimization of the clinical process builds upon the concept that for many cases it is not actually
necessary for patients to visit doctors in person. A mobile application can give a more effective solution
by bringing the doctor to the patient instead.
 The AI-powered mobile apps can provide basic healthcare support, usually [Link] simply
describe your symptoms, or ask questions, and then receive key information about your medical
condition derived from a wide network linking symptoms to causes. Apps can remind you to take
your medicine on time, and if necessary, assign an appointment with a doctor.

This approach promotes a healthy lifestyle by encouraging patients to make healthy decisions,
saves their time waiting in line for an appointment, and allows doctors to focus on more critical
cases.

Internet Search:

Data Science is the study of data, how it is stored and how can it be efficiently used accessed.
Data Science is used for internet search to provide relevant results and fast. Suppose you type
"Data Science" in the search bar and hit enter, you expect the results to be regarding Data
Science and not something else. And out of Zettabytes of data (even more than that), when you
search for something, you get results within a few seconds. All this is done with the help of Data
Science.

Targeted Advertising:

It provides advertiser transparence and control over the audience selected for targeting.
Property Targeting – is a simple and popular targeting mechanism. The advertiser specifies set of
pages where the ad should be shown. For example the company who sells tracks could show
advertisement on website about vehicles.
Targeted advertising is a marketing technique that uses consumer data to direct advertisements to
specific audiences. Brands use analytics related to personal interests, demographics, locations,
online habits, and even past purchases to craft and distribute the most relevant ads possible.
Website Recommendations:
Data science helps companies make better decisions, and recommender systems help data
scientists succeed in it. Recommender systems are tools designed for interacting with large and
complex information spaces and prioritizing items in these spaces that are likely to be of interest to
the user.

Advanced Image Recognition:

In the field of artificial intelligence, we often hear of advanced image recognition (AIR), a
particular technology that, through exploiting convolutional neural networks, simulates the
human brain with regard to sight.
The most common example of image recognition can be seen in the facial recognition system of
your mobile. Facial recognition in mobiles is not only used to identify your face for unlocking
your device; today, it is also being used for marketing

Speech Recognition:
Speech recognition, also known as automatic speech recognition (ASR), computer speech
recognition, or speech-to-text, is a capability which enables a program to process human
speech into a written format.
 Speech recognition uses a broad array of research in computer science, linguistics and computer
engineering. Many modern devices and text-focused programs have speech recognition functions
in them to allow for easier or hands-free use of a device.
Speech recognition and voice recognition are two different technologies and should not be
confused:
 Speech recognition is used to identify words in spoken language.
 Voice recognition is a biometric technology for identifying an individual's voice.

Airline Route Planning:

Airline Route Planning is nothing but planning on the proposed route for flights. It involves the
calculation of amount of fuel required for an aircraft to travel from the original airport to the
destination airport. The route planning also involves −
 Plan of flight with adequate amount of fuel and most reasonable route.
 Compliance of flight with Air Traffic Control requirements.
The following are the various plans −
 Summary plan for Air Traffic Control (FAA/ICAO format).
 Summary plan for direct download into on-board flight management system.
 Detailed plan for the use by pilots.
For route planning, flight planning officers need to consider the weather conditions, weight of the
flight at different stages, length of the journey, height at which it can be taken, and the rules set by
ICAO for safe planning
Gaming:
Gaming companies have become an integral part of the global entertainment industry, with users
spending more time than ever before in gaming. These companies excel at reviving social morality,
art, and sharing, making them masters of integrated entertainment.
The use of data in gaming is to create strategies based on the player’s behavior and to constantly
collect data in order to make predictions and decisions. The new generation of gaming is reliant on
data collection by analyzing the player’s behavior, as well as the use of Artificial Intelligence and
Machine Learning for better development.

Augmented Reality:

Augmented reality (AR) is the integration of digital information with the user's environment
in real time. Unlike virtual reality (VR), which creates a totally artificial environment, AR users
experience a real-world environment with generated perceptual information overlaid on top of it.
Examples of Augmented Reality
 IKEA Mobile App.
 Nintendo's Pokémon Go App.
 Google Pixel's Star Wars Stickers. ...
 Disney Coloring Book.

DATA ACQUISITION SEQUENCE:

The steps of a typical data analysis study are generally consistent with a general scientific
discovery sequence.
Your data science discovery starts with the question to be answered and the type ofanalysis to be
applied. The simplest analysis type is descriptive, where the data set isdescribed by reporting its
aggregate measures, often in a visual form.
exploratory data analysis, you try to find new relationships between existing variables. If you have a
small data sample and would like to describe a bigger population, statistics- based inferential
analysis is right
for you. A predictive analyst learns from the past to predict the future. Causal analysisidentifies variables
that affect each other. Finally, mechanistic data analysis explores exactly how one variable affects
another variable.
getting the raw data from the web or from a database is not that hard, and there areplenty of Python
tools that assist with downloading and deciphering it
In this imperfect world, there is no perfect data. “Dirty” data has missing values, outliers, and
other “non-standard” items. Some examples of “dirty” data are birth dates in the future, negative
ages and weights, and email addresses not intended foruse (noreply@). Once you obtain the raw
data, the next step is to use data-cleaningtools and your knowledge of statistics to regularize the
data set.

Data Acquisition Pipeline

Data acquisition is all about obtaining the artifacts that contain the input data from avariety of
sources, extracting the data from the artifacts, and converting it into representations suitable for
further processing,

The three main sources of data are the Internet (namely, the World Wide Web), databases, and
local files (possibly previously downloaded by hand or using additionalsoftware). Some of the local
files may have been produced by other Python programsand contain serialized or “pickled” data
consider ways and means of working with the most popular formats:
• Unstructured plain text in a natural language (such as English or Chinese)
• Structured data, including:
– Tabular data in comma separated values (CSV) files
– Tabular data from databases
– Tagged data in HyperText Markup Language (HTML) or, in general, in eXtensibleMarkup
Language (XML)
– Tagged data in JavaScript Object Notation (JSON)
data processing pipeline (obtaining, cleaning, and transforming raw data; descriptiveand exploratory
data analysis; and data modeling and prediction)
Report Structure
The project report is what we (data scientists) submit to the data sponsor (thecustomer). The report
typically includes the following:
• Abstract (a brief and accessible description of the project)
• Introduction
• Methods that were used for data acquisition and processing
• Results that were obtained (do not include intermediate and insignificantresults in
this section; rather, put them into an appendix)
• Conclusion
• Appendix