AUGMENTED TOPIC-1
Course Faculty : [Link]
Topic : Big Data
Introduction:
The definition of big data is data that contains greater variety, arriving in
increasing volumes and with more velocity. This is also known as the three Vs.
Put simply, big data is larger, more complex data sets, especially from
new data sources. These data sets are so voluminous that traditional data
processing software just can’t manage them. But these massive volumes of data
can be used to address business problems you wouldn’t have been able to tackle
before.
Volume
The amount of data matters. With big data, you’ll have to process high
volumes of low-density, unstructured data. This can be data of unknown value,
such as Twitter data feeds, clickstreams on a web page or a mobile app, or
sensor-enabled equipment. For some organizations, this might be tens of
terabytes of data. For others, it may be hundreds of petabytes.
Velocity
Velocity is the fast rate at which data is received and (perhaps) acted on.
Normally, the highest velocity of data streams directly into memory versus being
written to disk. Some internet-enabled smart products operate in real time or
near real time and will require real-time evaluation and action.
Variety
Variety refers to the many types of data that are available. Traditional
data types were structured and fit neatly in a relational database. With the rise
of big data, data comes in new unstructured data types. Unstructured and semi
structured data types, such as text, audio, and video, require additional
preprocessing to derive meaning and support metadata.
The value and truth of big data
Two more Vs have emerged over the past few years: value and veracity.
Data has intrinsic value. But it’s of no use until that value is discovered. Equally
important: How truthful is your data—and how much can you rely on it.
Today, big data has become capital. Think of some of the world’s biggest
tech companies. A large part of the value they offer comes from their data, which
they’re constantly analyzing to produce more efficiency and develop new
products.
Recent technological breakthroughs have exponentially reduced the cost
of data storage and compute, making it easier and less expensive to store more
data than ever before. With an increased volume of big data now cheaper and
more accessible, you can make more accurate and precise business decisions.
Finding value in big data isn’t only about analyzing it (which is a whole
other benefit). It’s an entire discovery process that requires insightful analysts,
business users, and executives who ask the right questions, recognize patterns,
make informed assumptions, and predict behaviour.
Big data challenges
First, big data isbig. Although new technologies have been developed for
data storage, data volumes are doubling in size about every two years.
Organizations still struggle to keep pace with their data and find ways to
effectively store it.
But it’s not enough to just store the data. Data must be used to be
valuable and that depends on correlation. Clean data, or data that’s relevant to
the client and organized in a way that enables meaningful analysis, requires a lot
of work. Data scientists spend 50 to 80 percent of their time curating and
preparing data before it can actually be used.
Finally, big data technology is changing at a rapid pace. A few years ago,
Apache Hadoop was the popular technology used to handle big data. Then
Apache Spark was introduced in 2014. Today, a combination of the two
frameworks appears to be the best approach. Keeping up with big data
technology is an ongoing challenge.
How big data works
Big data gives you new insights that open up new opportunities and
business models. Getting started involves three key actions:
[Link]
Big data brings together data from many disparate sources and applications.
Traditional data integration mechanisms, such as extract, transform, and load
(ETL) generally aren’t up to the task. It requires new strategies and technologies
to analyze big data sets at terabyte, or even petabyte, [Link] integration,
you need to bring in the data, process it, and make sure it’s formatted and
available in a form that your business analysts can get started with.
[Link]
Big data requires storage. Your storage solution can be in the cloud, on premises,
or both. You can store your data in any form you want and bring your desired
processing requirements and necessary process engines to those data sets on an
on-demand basis. Many people choose their storage solution according to where
their data is currently residing. The cloud is gradually gaining popularity because
it supports your current compute requirements and enables you to spin up
resources as needed.
[Link]
Your investment in big data pays off when you analyze and act on your data. Get
new clarity with a visual analysis of your varied data sets. Explore the data
further to make new discoveries. Share your findings with others. Build data
models with machine learning and artificial intelligence. Put your data to work.
AUGMENTED TOPIC -2
Course Faculty : [Link]
Topic : Automated Machine Learning
Automated machine learning (AutoML) is the process
of automating the tasks of applying machine learning to real-world problems.
AutoML potentially includes every stage from beginning with a raw dataset
to building a machine learning model ready for deployment. AutoML was
proposed as an artificial intelligence-based solution to the growing challenge of
applying machine learning. The high degree of automation in AutoML aims to
allow non-experts to make use of machine learning models and techniques
without requiring them to become experts in machine learning. Automating the
process of applying machine learning end-to-end additionally offers the
advantages of producing simpler solutions, faster creation of those solutions, and
models that often outperform hand-designed models [
The raw data may not be in a form that all algorithms can be applied to.
To make the data amenable for machine learning, an expert may have to apply
appropriate data pre-processing, feature engineering, feature extraction,
and feature selection methods. After these steps, practitioners must then
perform algorithm selection and hyperparameter optimization to maximize the
predictive performance of their model. If deep learning is used, the architecture
of the neural network must also be chosen by the machine learning expert.
Each of these steps may be challenging, resulting in significant hurdles to using
machine learning. AutoML aims to simplify these steps for non-experts, and to
make it easier for them to use machine learning techniques correctly and
effectively.
AutoML plays an important role within the broader approach of
automating data science, which also includes challenging tasks such as data
engineering, data exploration and model interpretation [3] and prediction.[
Targets of automation
Automated machine learning can target various stages of the machine learning
process.[2] Steps to automate are:
Data preparation and ingestion (from raw data and miscellaneous formats)
o Column type detection; e.g., boolean, discrete numerical, continuous
numerical, or text
o Column intent detection; e.g., target/label, stratification field, numerical
feature, categorical text feature, or free text feature
o Task detection; e.g., binary classification, regression, clustering,
or ranking
Feature engineering
o Feature selection
o Feature extraction
o Meta-learning and transfer learning
o Detection and handling of skewed data and/or missing values
Model selection - choosing which machine learning algorithm to use, often
including multiple competing software implementations
Ensembling - a form of consensus where using multiple models often gives
better results than any single model
Hyper parameter optimization of the learning algorithm and featuraization
Pipeline selection under time, memory, and complexity constraints
Selection of evaluation metrics and validation procedures
Problem checking
o Leakage detection
o Misconfiguration detection
Analysis of obtained results
Creating user interfaces and visualizations
Supervised learning:
In this approach input data also referred to as training data has a known
result e.g. email is spam or not. A model undergoes training where it makes
predictions that are corrected when they are incorrect. This process is
repeated up to the point when the level of accuracy is acceptable.
Classification and regression are problems that can be solved this way.
Logistic Regression and Neural Networks are examples of algorithmic
examples of Supervised learning.
Unsupervised learning :
In this approach there are no known results. A model is developed by
drawing upon structures present in the data. Clustering and dimensionality
reduction are some problems solved this way.
Semi-supervised learning:
In this approach there are known and unknown results. The model has to
learn structures present in the data and also make predictions.
Reinforcement learning :
In this approach loss function of the learning system is unclear. This is an
area of machine learning inspired by behaviorist psychology, concerned with
how software agents ought to take actions in an environment so as to
maximize some notion of cumulative reward.
Feature Engineering is the Key
If you have many independent features that each correlates well with the
class, learning is easy. On the other hand, if the class is a very complex
function of the features, you may not be able to learn it. Often, the raw data
is not in a form that is amenable to learning, but you can construct features
from it that are. This is typically where most of the effort in a ma- chine
learning project goes. It is often also one of the most interesting parts, where
intuition, creativity and “black art” are as important as the technical stuff.
First-timers are often surprised by how little time in a ma- chine learning
project is spent actually doing machine learn- ing. But it makes sense if you
consider how time-consuming it is to gather data, integrate it, clean it and
pre-process it, and how much trial and error can go into feature design. Also,
machine learning is not a one-shot process of building a data set and running
a learner, but rather an iterative process of running the learner, analyzing the
results, modifying the data and/or the learner, and repeating. Learning is
often the quickest part of this, but that’s because we’ve already mastered it
pretty well! Feature engineering is more difficult because it’s domain-specific
while learners can be largely general-purpose. However, there is no sharp
frontier between the two, and this is another reason the most useful learners
are those that facilitate incorporating knowledge.
Of course, one of the holy grails of machine learning is to automate more
and more of the feature engineering process. One way this is often done
today is by automatically generating large numbers of candidate features and
selecting the best by (say) their information gain with respect to the class.
But bear in mind that features that look irrelevant in isolation may be
relevant in combination. For example, if the class is an XOR of k input
features, each of them by itself carries no information about the class. (If you
want to annoy machine learners, bring up XOR.) On the other hand, running a
learner with a very large number of features to find out which ones are useful
in combination may be too time-consuming, or cause overfitting. So there is
ultimately no replacement for the smarts you put into feature engineering.
Criticism by Machine Learning Experts
Machine learning lacks in some major and minor components. One of the
common problems with machine learning is the debugging process. The
automated process of debugging in machine learning can be extremely time-
consuming, which can make some users uncomfortable. The lack of statistical
prediction invention in machine learning can cause the learning to lack in
details. Also, the difficulty lies in the fact that the set of all possible behaviors
are given all possible inputs is too large to be covered by the set of observed
[Link] makes the learner generalize from the given data, so as to be
able to produce a useful output in new cases.
Machine learning is the body of research related to automated large-scale
data analysis. Historically, the field was centeredaround biologically inspired
models, and the long-term goals of much of the community are oriented to
producing models and algorithms that can process information as well as
biological systems.
The field also encompasses many of the traditional areas of statistics with,
however, a strong focus on mathematical models and also prediction.
Machine learning is now central to many areas of interest in computer science
and related large-scale information processing domains.