Introduction
Lecture one
Data
• Data is the raw form of information, a collection of facts, figures,
symbols or observations that represent details about events, objects
or phenomena. By itself, data may appear meaningless, but when
organized, processed and interpreted, it transforms into valuable
insights that support decision-making, problem-solving and
innovation.
• Digital data is any information represented as binary code (1s and 0s)
that computers can process, store, and transmit, encompassing text,
images, audio, video, and more, existing in structured (databases),
unstructured (emails, social media), and semi-structured (JSON logs)
formats.
Structured Data
• Structured data conforms to a data model or schema and is often
stored in tabular form. It is used to capture relationships between
different entities and is therefore most often stored in a relational
database.
• Databases that natively support structured data, it rarely requires
special consideration in regards to processing or storage. Examples of
this type of data include banking transactions, invoices, and customer
records.
Unstructured Data
• Data that does not conform to a data model or data schema is known as
unstructured data. It is estimated that unstructured data makes up 80% of the
data within any given enterprise. Unstructured data has a faster growth rate than
structured data.
• This form of data is either textual or binary and often conveyed via files that are
self-contained and non-relational. A text file may contain the contents of various
tweets or blog postings. Binary files are often media files that contain image,
audio or video data. Technically, both text and binary files have a structure
defined by the file format itself, but this aspect is disregarded, and the notion of
being unstructured is in relation to the format of the data contained in the file
itself.
Semi-structured
• Data Semi-structured data has a defined level of structure and
consistency, but is not relational in nature. Instead, semi-structured
data is hierarchical or graph-based. This kind of data is commonly
stored in files that contain text. For instance, XML and JSON files are
common forms of semi-structured data. Due to the textual nature of
this data and its conformance to some level of structure, it is more
easily processed than unstructured data.
Examples of common sources of semi-structured data
include spreadsheets, RSS feeds and sensor data. Semi-
structured data often has special pre-processing and storage
requirements, especially if the underlying format is not text-
based. An example of pre-processing of semi-structured data
would be the validation of an XML file to ensure that it
conformed to its schema definition.
Metadata
• Metadata provides information about a dataset’s characteristics and
structure. This type of data is mostly machine-generated and can be
appended to data. The tracking of metadata is crucial to Big Data
processing, storage and analysis because it provides information about the
pedigree of the data and its provenance during processing. Examples of
metadata include:
• XML tags providing the author and creation date of a document
• attributes providing the file size and resolution of a digital photograph
Big Data solutions rely on metadata, particularly when processing semi-
structured and unstructured data.
Different Source of Data
• Data solutions can be human-generated
or machine-generated, although it is
ultimately the responsibility of machines
to generate the analytic results. Human-
generated data is the result of human
interaction with systems, such as online
services and digital devices
• Machine-generated data is generated by
software programs and hardware
devices in response to real-world events.
Data and computing
• Data is related with different activities in computer devices:
• Collecting data
• representing
• Storing data
• Processing data
• Analysis data
• Sharing data
• Security and Privacy of data
Big Data set characteristics
• Five Big Data characteristics that can be used to help differentiate
data categorized as “Big” from other forms of data:
• • volume • velocity • variety • veracity • value
Volume
• The anticipated volume of data that is
processed by Big Data solutions is
substantial and ever-growing.
• Data sources that are responsible for
generating high data volumes can include:
• online transactions, such as point-of-sale
and banking
• scientific and research experiments, such as
the Large Hadron Collider and Atacama Large
Millimeter/Submillimeter Array telescope
• sensors, such as GPS sensors, RFIDs, smart
meters and telematics Example: Organizations and users world-wide create
over 2.5 EBs of data a day. As a point of comparison,
• social media, such as Facebook and Twitter the Library of Congress currently holds more than 300
TBs of data
Velocity
• In Big Data environments, data can
arrive at fast speeds, and enormous
datasets can accumulate within very
short periods of time. From an
enterprise’s point of view, the velocity
of data translates into the amount of
time it takes for the data to be
processed once it enters the Examples of high-velocity Big Data
enterprise’s perimeter. Coping with the datasets produced every minute
fast inflow of data requires the include tweets, video, emails and GBs
enterprise to design highly elastic and generated from a jet engine
available data processing solutions and
corresponding data storage capabilities
Variety
• Data variety refers to the multiple formats and types of data that
need to be supported by Big Data solutions. Data variety brings
challenges for enterprises in terms of data integration,
transformation, processing, and storage.
Examples of high-variety Big Data datasets include structured, textual, image, video,
audio, XML, JSON, sensor data and metadata.
Big Data Characteristics
are Challenges in
Computing as related
with their activities in
Computer
Big Data
• Big Data is a field dedicated to the analysis, processing, and storage of
large collections of data that frequently originate from disparate
sources.
• Big Data solutions and practices are typically required when
traditional data analysis, processing and storage technologies and
techniques are insufficient
•Reference