Scribd
Upload
9 views7 pages

Overview of Data Science and Machine Learning

The document provides a comprehensive overview of data science, including definitions of data, science, and data science itself, along with the importance of probability, statistics, and linear algebra. It discusses various machine learning techniques such as supervised, unsupervised, semi-supervised, and reinforcement learning, along with their applications and algorithms. Additionally, it highlights real-world use cases of data science across different industries, emphasizing its role in customer retention, sentiment analysis, online advertising, and personalized medicine.

Uploaded by

daretoanswer2217
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
9 views7 pages

Overview of Data Science and Machine Learning

The document provides a comprehensive overview of data science, including definitions of data, science, and data science itself, along with the importance of probability, statistics, and linear algebra. It discusses various machine learning techniques such as supervised, unsupervised, semi-supervised, and reinforcement learning, along with their applications and algorithms. Additionally, it highlights real-world use cases of data science across different industries, emphasizing its role in customer retention, sentiment analysis, online advertising, and personalized medicine.

Uploaded by

daretoanswer2217
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

 According to Oxford Dictionary, Data is defined as "facts and statistics collected together for

reference or analysis".
 According to the Science Council, Science is the pursuit and application of knowledge and
understanding of the natural and social world following a systematic methodology based on
evidence.
 Data Science is an interdisciplinary field about processes and systems to extract knowledge
or insights from data in various forms, either structured or unstructured, which is a
continuation of some of the data analysis fields such as statistics, data mining, and
analytics. (Ref. Wikipedia)
 Data Science is the empirical synthesis of actionable knowledge from raw data through the
complete data lifecycle process. (Ref. NIST - The National Institute of Standards and
Technology)

 Probability is a mathematical subject which enables us in determining or predicting how


likely it is that an event will happen. The probability of occurrence is assigned a value from 0
to 1
 Statistics is another mathematical subject which deals primarily with data. It helps us draw
inferences from data by having procedures in place for collecting, classifying and presenting
the data in an organized manner. The analysis and interpretation of the refined data helps in
providing further insights.
 Linear Algebra is a mathematical subject the deals with the theory of systems of linear
equations, matrices, vector spaces and linear transformations.
 In 1959, Arthur Samuel defined machine learning as "A field of study that gives computers
the ability to learn without being explicitly programmed".
 Types of machine learning:
 Supervised learning : used in applications when labelled historical data predicts likely future
events.

1. Learning phase : A machine is taught to identify various fruits by building a model with
the help of images.
2. Testing phase : A new set of images is given to this model as test data so that it can
classify different fruits.
3. Evaluation phase : We should always evaluate a model to know whether it will do a
good job of predicting the target, given new data points. One way to do so is to compute
the ratio of correctly classified test data points to the total number of test data points
available, thus determining the accuracy of the [Link] our example, out of eight test
images, machine was able to classify 6 images correctly and 2 images incorrectly.
Hence, accuracy of this supervised machine learning model is 6/8 i.e. 75%

There can be two types of supervised machine learning techniques as shown below:

Classification: Used to predict discrete results.

Regression: Used to predict continuous numeric results

Famous algorithms are : Linear regression, Logistic regression, naïve bayes, k-nearest
neighbors, decision trees, random forests, support vector machines
 Unsupervised learning : used when historical data is not labelled and is used to discover
unknown patterns in the data.

There is a basket filled with some fresh fruits. The machine’s task is to group similar colored
fruits together. But here, unlike supervised learning, the machine is not exposed to any prior
knowledge. Machine identifies four clusters of fruits based on their color,

In the first step, we fix some variables or parameters based on which the machine will
arrange the given data (in our example, we have taken "color" as the parameter).

In the second step, the machine groups similar data points together. In our example, fruits
with the same color are grouped together.

Famous algorithms are : k – means clustering, hierarchical clustering, mixture models

 Semi supervised : used in same applications as supervised learning. This uses both
labelled and unlabelled data for training

In real-time, it may so happen that the unlabeled data points exceed the number of labeled
data points in a data set. In order to fit a model to such data we use the semi-supervised
machine learning technique, wherein we perform the following steps:

 Step 1: Train the model with labeled data points only.


 Step 2: Use the above model to predict the labels of the unlabeled data points
 Step 3: Combine the existing labeled data points with the newly labeled data points
and use it to retrain the model
 Step 4: Repeat the 2nd and 3rd steps until it converges

Applications of semi-supervised learning are text processing, video-indexing, bioinformatics,


web page classification and news classification among others.

Famous algorithms are graph based methods, generative models, low-density separation,
heuristic approaches

 Reinforcement learning : works by discovering which action yields maximun reward through
trail and error. This is often used in robotics, gaming and navigation

Reinforcement machine learning algorithm is a reward based and immediate feedback


technique. Here, the machine's goal is to maximize the numerical reward at each and every
step. In the process of learning, the machine is not provided any supervision as opposed to
the previous ML algorithms we discussed till now. Instead, the machine is expected to figure
out the optimum actions which will reap the maximum reward at each step, all on their own,
without any interference.
The actions that the machine takes at each step might not only affect the immediate reward
but may also affect all the subsequent rewards. The ultimate aim is to reach the max
possible reward in the least amount of steps [Link], trial and error search
methodology and immediate feedback in the form of a numerical reward are the two main
characteristics of reinforcement learning.

An example of reinforcement learning would be when a machine, learning to play chess,


decides whether a move is right by planning the possible moves, anticipating the
corresponding counter moves and finally choosing one based on reward based appeal for a
particular position or set of moves. Another example could be when a trash collecting bot's
charge is about to reach critical levels and it needs to make a decision, to clean one more
room before reaching out for the charging station or to immediately rush to the nearest
charging station. The decision taken by the bot depends on the ease with which it can reach
the charging station, based on its prior knowledge.

Famous algorithms are Markov decision processes, monte carlo methods, temporal
difference learning.

 Computer Science provides us with the necessary programming languages, database


management systems, statistical analysis and machine learning tools.

Historical data or past data from various sources are cleaned and subjected to Feature
Engineering. fFeature Engineering is the process of using domain knowledge to select or
create significant features from the historical data relevant to the problem statement.
This engineered data is divided into two sets: Train data and Test data. Data science
models are built using train data, and then the performance of the model is evaluated
on the test data. This validated model is used for taking various decisions on
new/unseen data points.
Data science project life cycle:
Real world use cases of data science:
1. Churn prediction:

Churn implies loss of customers to competition. For any company, it costs more
to acquire new customers than to retain the old ones. As churn prediction aids in
customer retention, it is extremely important especially for businesses with a
repeat customer base. The application of this model cuts across domains such as
Banking, E-Retail, Telecom, Energy and Utilities.

2. Sentiment analysis

Also referred to as opinion mining, it is the process of computationally identifying


what customers like and dislike about a product or a brand. A domain which
relentlessly makes use of sentiment analysis is the Retail industry. Companies
like Amazon, Flipkart, Reliance, Paytm use customer feedback from social
networking sites like Facebook, Twitter, etc. or their own company websites to
find out what their customers are talking about and how they feel i.e. positive,
negative, or neutral. They leverage this information to reposition their products
and provide better/new services.

3. Online advertisement

The incremental growth in the complexity of ads industry is due to the ease of
access to the internet via a wide variety of devices around the world. This gives
the advertisers an opportunity to study user preferences and online trends. The
insights offered to them through these analysis, translates to actionable items on
issues and opportunities such as reducing ad-blindness or optimizing cost-per-
action (CPA) and click-through-rates (CTR).

4. Recommendations:
Many e-retail companies like Amazon, Netflix, Spotify, Best Buy, You Tube among
many others use recommender systems to improve a customer's shopping
experience. This offers the companies a chance to gather information on
customer's preferences, purchases and other browsing patterns which
lend insights that can amplify their return on investment.

5. Truth and veracity:


In today's digital world, the quantity of fake news is on the rise. Not only does
vast majorities of population fall prey to misinformation but it effects businesses
negatively. Data Science is being used to ensure data veracity or in other words,
verify the truthfulness of data based on both accuracy and context. Companies
such Facebook, Twitter, Starbucks, Costco and many others are combating fake
news currently with the help of various data science techniques.

6. News aggregation:
A news aggregator gathers and clusters stories of the same topic from several
leading news websites and also traces the genuine source of a news item and
what is the course of the story. It has a special interactive timeline that allows
the reader to flip swiftly between headlines, refining their search by country or by
specific news sites. Notable examples include Google News, Reddit, Flipboard,
Pulse etc.
7. Scalability:
Scalability refers to an enterprise's ability to handle increased demands. In the
corporate environment, a scalable company is one that can maintain or improve
profit margins while sales volume increases. Many a times the process is slowed
down by human intervention for decisions. For example, the credit operations in
banks invest substantial time in assessing the credit worthiness of a client. Client
management teams take long time to suggest the right product to the
customer/suggest alternatives. The client help desk takes long time to provide
the desired info to the client. If these processes can be automated, the business
can scale up. Data science helps build systems like recommender systems,
Chabot’s etc. to achieve scalability.

8. Content discovery/search:

Content discovery involves using predictive algorithms to help make content


recommendations to users based on how they search. Search engines such
as Google, Bing and Yahoo and various other platforms are now using intelligent
learning mechanisms to understand user preferences to be able to suggest
content that’s most suitable for them.

Few more platforms that use content discovery algorithms are Facebook and
YouTube. The content that appears in an individual's Facebook news feed and
the videos that appear in the "Recommended for You" section of YouTube user's
account, are both altered according to each user's past behavior and personal
preferences.

9. Intelligent leraming:
Intelligent learning has become a part of our day to day lives in various forms.
For example, Google Maps uses undesignated location data from various smart
devices to predict the flow of traffic in real time. It also utilizes user based reports
on incidents that might affect the traffic, like road construction and accidents, to
help suggest fastest routes for travel, to users.

Another example would be ride sharing apps like Uber and Ola. They optimize
the ride experience by not only minimizing the ride time but also by matching
users with other passengers for least amount of detours in shared rides

Other examples of intelligent learning include self-driving cars, smart-email


categorization, credit-card fraud detection, etc.

10. Personalised medicine:


In many cases, the success of a particular treatment for a patients' condition
cannot be predicted beforehand. Thus, many medical practitioners follow a non-
optimal trial-and-error approach.

In personalized medicine, a doctor needs to study an individual's genes,


environment and lifestyle. This would help tailor treatments for specific medical
conditions as opposed to a trial and error approach. This would also enable
pharmaceutical researchers to create combination drugs targeting a specific
genomic profile which in turn increases safety and efficiency.
Companies that are active in the field of personalized medicine are Roche,
Novartis, Johnson & Johnson among others.

Overview:

NUMPY: (Numerical python)

is a Python's scientific module which is created to perform typical numerical


calculations on multidimensional arrays along with some other sophisticated
functions. It provides ease working with linear algebra and carrying out complex
calculations like Matrix inversion, Fourier transforms, etc.

NumPy array works with a similar datatype and hence show a significant boost in
the processing time.

Common questions

Powered by AI

Reinforcement learning differs in that it doesn't use predefined labels (supervised learning) or only discover patterns (unsupervised learning). Instead, it focuses on learning optimal actions through trial and error based on rewards, effectively dealing with decision-making problems where the sequence of actions impacts the outcome. Unique challenges include exploration-exploitation trade-offs and the need to balance short-term and long-term rewards, which are not addressed by supervised or unsupervised learning.

Supervised learning involves using labeled data to predict outcomes, which helps in understanding specific patterns related to predefined classes or quantities. Unsupervised learning, on the other hand, finds hidden patterns or groupings in data without pre-labeled outcomes. They complement each other by offering a balanced approach; supervised learning offers predictions and clarity in patterns while unsupervised learning reveals insights and structures not initially known. These insights are often foundational for supervised models.

The Data Science lifecycle involves stages such as data collection, data engineering (including cleaning and feature engineering), model building, evaluation, and deployment. Traditional data analysis methodologies, like statistics and probability, are integral at various stages—statistics for inference during data analysis and probability for predicting outcomes. Machine learning models, a key part of data science, evolve from these methodologies by automating the learning of patterns from data.

A practical application of semi-supervised learning is in text classification, such as organizing news articles. This approach allows models to use a small amount of labeled data to infer labels for a large set of unlabeled articles, thereby optimizing the labeling process. Steps involve training on labeled data, predicting labels for unlabeled data, and retraining iteratively. This balances the scarcity of labeled data against the abundance of unlabeled data, improving overall model accuracy.

Intelligent learning transforms technologies by utilizing real-time data analysis and prediction models to improve user experiences. For navigation, like Google Maps, it uses historical and live traffic data to predict optimal routes. In fraud detection, it employs machine learning algorithms to identify suspicious behavior patterns, protecting against unauthorized activities. Both applications showcase adaptability and proactive responses to dynamic environments through intelligent data-driven decisions.

Personalized medicine uses data science to tailor medical treatments by analyzing patient-specific data, including genomic information and lifestyle factors. This approach addresses challenges like the non-optimal trial-and-error in traditional treatment methods by predicting treatment efficacy and reducing adverse drug reactions. Data science improves safety and efficiency, ultimately contributing to better patient outcomes.

Probability is essential for quantifying uncertainty and making predictions about an event's likelihood, forming the basis for risk assessment and decision-making in machine learning models. Statistics provide methods for data collection, analysis, interpretation, and inference, enabling data scientists to derive insights and validate hypotheses. Together, they underpin the mathematical framework for learning from data and optimizing models in data science applications.

Feature engineering involves the creation or selection of significant variables that enhance a model's predictive power. By using domain knowledge, data scientists can transform raw data into informative features that capture essential patterns, thereby improving model accuracy and efficiency. Effective feature engineering often can make the difference between a successful model and one that underperforms.

Linear algebra is crucial in machine learning as it deals with vectors, matrices, and linear transformations, which are foundational for operations such as data transformations and high-dimensional space representations. It is essential in optimizing algorithms like support vector machines and in computations performed by neural networks, enabling efficient data processing and model training.

Data science techniques combat fake news by analyzing content accuracy and context sophistication through machine learning algorithms designed for pattern recognition. Natural language processing and text mining allow platforms like Twitter and Facebook to identify misleading information quickly. These techniques help businesses maintain brand reputation and trust, vital for customer loyalty and operational effectiveness.

You might also like