DATA SCIENCE TYCS SEM VI UNIT 3
Data Curation
• Data Curation is the organization, publication and presentation of data such that value of data
is maintained over time and available for reuse.
• It determines what information is worth saving and for how long.
• The main purpose is to ensure data is reliably retrievable for future research purposes.
• The data curation life cycle consists of the following actions :
SEQUENTIAL ACTIONS
1) Conceptualise
Conceive and plan the creation of data, including capture method and storage options.
2) Create or Receive
Create data including administrative, descriptive, structural and technical metadata. Preservation
metadata may also be added at the time of creation.
Receive data, in accordance with documented collecting policies, from data creators, other
archives, repositories or data centres, and if required assign appropriate metadata.
3) Appraise and Select
Evaluate data and select for long-term curation and preservation. Adhere to documented guidance,
policies or legal requirements.
4) Ingest
Transfer data to an archive, repository, data centre or other custodian. Adhere to documented
guidance, policies or legal requirements.
5) Preservation Action
Undertake actions to ensure long-term preservation and retention of the authoritative nature of
data. Preservation actions should ensure that data remains authentic, reliable and usable while
maintaining its integrity. Actions include data cleaning, validation, assigning preservation
metadata, assigning representation information and ensuring acceptable data structures or file
formats.
6) Store
Store the data in a secure manner adhering to relevant standards.
7) Access, Use and Reuse
Ensure that data is accessible to both designated users and reusers, on a day-to-day basis. This may
be in the form of publicly available published information. Robust access controls and
authentication procedures may be applicable.
8) Transform
Create new data from the original, for example by migration into a different format, or by creating
a subset, by selection or query, to create newly derived results, perhaps for publication
1 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
OCCASIONAL ACTIONS
1) Dispose
Dispose of data, which has not been selected for long-term curation and preservation in accordance
with documented policies, guidance or legal requirements.
Typically data may be transferred to another archive, repository, data centre or other custodian. In
some instances data is destroyed. The data's nature may, for legal reasons, necessitate secure
destruction.
2) Reappraise
Return data which fails validation procedures for further appraisal and re-selection.
3) Migrate
Migrate data to a different format. This may be done to accord with the storage environment or to
ensure the data's immunity from hardware or software obsolescence.
Query languages and Operations to specify and transform data
• A query can either be a select or action query – select queries pick parts of your data, while
action queries manipulate retrieved data. A query can also work with the combination of both
actions to perform more varied tasks, for example, to review, insert, modify, or delete data, as
well as calculate and combine data from multiple tables.
• Choosing the database and its language is crucial when working with queries. In addition to
SQL, there is another type of database called NoSQL (Not Only Structured Query Language).
The main difference between the two is the data structure.
• SQL databases are relational and use predefined schemas that require you to specify your data
structure. On the other hand, NoSQL databases are non-relational and have dynamic schemas
for unstructured data.
Below are some of the most common query commands along with their functions:
SELECT – fetch data from the database. It’s one of the most popular commands, as every request
begins with a select query.
CREATE TABLE – build different tables and specify the name of each column within.
ORDER BY – sort data results either numerically or alphabetically.
SUM – summarize data from a specific column.
UPDATE – modify existing rows in a table.
INSERT – add new data or rows to an existing table.
WHERE – filter data and get its value based on a set condition.
Below are the main benefits of using a query:
2 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
• Review data from multiple tables simultaneously.
• Filter records containing only certain fields and of certain criteria.
• Automate data management tasks and perform calculations.
Examples of queries:
SELECT Name, Occupation FROM Participant
DELETE FROM Participant WHERE Occupation = ‘Unemployed’
UPDATE Participant SET Occupation = ‘Headmaster’ WHERE ID = ‘3’
Big Data
• Big Data is data whose scale, distribution, diversity, and/or timeliness require the use of new
technical architectures and analytics to enable insights that unlock new sources of business
value.
Three attributes stand out as defining Big Data characteristics:
• Huge volume of data: Rather than thousands or millions of rows, Big Data can be billions of
rows and millions of columns.
• Complexity of data types and structures: Big Data reflects the variety of new data sources,
formats, and structures, including digital traces being left on the web and other digital
repositories for subsequent analysis.
• Speed of new data creation and growth: Big Data can describe high velocity data, with rapid
data ingestion and near real time analysis.
Although the volume of Big Data tends to attract the most attention, generally the variety and
velocity of the data provide a more apt definition of Big Data. Due to its size or structure, Big Data
cannot be efficiently analyzed using only traditional databases or methods. Big Data problems
require new tools and technologies to store, manage, and realize the business benefit.
Transforming Data using R
• The [Link]() function is used to import the CSV file. This dataset is stored to the R variable
using the assignment operator ←
Once the file has been imported, it is useful to examine the contents to ensure that the data was
loaded properly as well as to become familiar with the data.
• The head() function, by default, displays the first six records.
• The summary() function provides some descriptive statistics, such as the mean and median, for
each data column. Additionally, the minimum and maximum values as well as the 1st and 3rd
quartiles are provided.
• Plotting a dataset’s contents can provide information about the relationships between the
various columns. The plot() function generates a scatterplot.
3 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
• The $ is used to reference a specific column in the dataset.
• A generic function is a group of functions sharing the same name but behaving differently
depending on the number and the type of arguments they receive. The summary() function is an
example of a generic function.
• Other import functions include [Link]() and [Link](), which are intended to import other
common file types such as TXT.
• The analogous R functions such as [Link](), [Link]() enable exporting of R datasets to an
external file.
• The View() function can be used to view the data in a tabular format.
• The dplyr package is used to manipulate data.
Filtering a dataset
Filtering the dataset enables you to focus on a subset of the rows instead of the entire dataset.
The dplyr package includes a filter() function that supports this capability.
The filter() function is used to create a subset of records based on some value.
filter(dataframe_name,condition1,condition2,..)
Examples:
filter(starwars, species == "Human")
filter(starwars, mass > 1000)
filter(starwars, hair_color == "none" & eye_color == "black")
filter(starwars, hair_color == "none" | eye_color == "black")
Narrowing the list of columns with select()
To use the select() function, simply pass in the name of the data frame along with the columns
to include.
Select(dataframe,column1,column2,..)
Arranging Rows
The arrange() function in the dplyr package can be used to order the rows in a data frame. Use
the desc() helper function to order the rows in descending order.
arrange(dataframe, desc(columnname))
Adding Rows with mutate()
The mutate() function is used to add new columns to a data frame that are the result of a
function you run on other columns in the data frame. Any new columns created with the
mutate() function will be added to the end of the data frame.
mutate(dataframe,newvar=expression involving existing var)
Summarizing and Grouping
The summarise() function produces a single row of data containing summary statistics from a
data frame. This function is normally paired with the group_by() function to produce group
summary statistics. The group_by() function handles the split portion of the paradigm by
creating groups of data using one or more columns.
4 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
grp = group_by(df, columnname)
sm = summarise(grp, mean(columnname))
Useful functions of summarise
• Center: mean(), median()
• Spread: sd(), IQR()
• Range: min(), max()
• Count: n(), n_distinct()
Piping
Each dplyr function returns a new data frame, and this data frame is typically used as the input
to the next dplyr function in the series. These data frames are intermediate datasets not needed
beyond the current step.
Piping is a more efficient way of handling these temporary, intermediate datasets. In sum,
piping is an efficient way of sending the output of one function to another function without
creating an intermediate dataset and is most useful when you have a series of functions to run.
The syntax for piping is to use the %>% characters at the end of each statement that you want to
pipe.
The syntax for piping is to use the %>% characters at the end of each statement that you want to
pipe.
df %>% select(STATE, Yr, TOTALACRES, CAUSE) %>% filter(TOTALACRES >= 1000)
NoSQL
NoSQL (Not only Structured Query Language) data stores that are being developed to address
specific Big Data use cases. It describes those data stores that are applied to unstructured data.
NoSQL databases are non-tabular databases and tend to be more flexible than the traditional,
SQL-based, relational database tables. They provide flexible schemas and scale easily with
large amounts of data.
NoSQL databases have the following features:
• Non- Relational - NoSQL databases never follow the relational model and never provide tables
with fixed-column records.
• Flexible schemas - NoSQL databases are either schema-free or contain schemas that are more
loose. There is no requirement for any kind of data structure specification.
• Horizontal scaling - NoSQL databases are horizontally scalable, which means that they can
handle increased traffic simply by adding more servers to the database. It refers to bringing on
additional nodes to share the load.
• Simple API - Provides simple user interfaces for storing and querying data.
• Distributed - A distributed execution of many NoSQL databases is possible.
Types of NoSQL
Four major categories of NoSQL tools are
1) Key/value stores contain data (the value) that can be simply accessed by a given identifier (the
key. The values can be complex. In a key/value store, there is no stored structure of how to use the
5 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
data; the client that reads and writes to a key/value store needs to maintain and utilize the logic of
how to meaningfully extract the useful elements from the key and the value. Here are some uses
for key/value stores:
• Using a customer’s login ID as the key, the value contains the customer’s preferences.
• Using a web session ID as the key, the value contains everything that was captured during the
session.
Eg. Redis, Voldemort
2) Document stores are useful when the value of the key/value pair is a file and the file itself is
self-describing (for example, JSON or XML). The underlying structure of the documents can be
used to query and customize the display of the documents’ content. Because the document is self-
describing, the document store can provide additional functionality over a key/value store. For
example, a document store may provide the ability to create indexes to speed the searching of the
documents. Otherwise, every document in the data store would have to be examined. Document
stores may be useful for the following:
• Content management of web pages
• Web analytics of stored log data
Eg. CouchDB, MongoDB
3) Column family stores are useful for sparse datasets, records with thousands of columns but
only a few columns have entries. The key/value concept still applies, but in this case a key is
associated with a collection of columns. In this collection, related columns are grouped into
column families. For example, columns for age, gender, income, and education may be grouped
into a demographic family. Column family data stores are useful in the following instances:
• To store and render blog entries, tags, and viewers’ feedback
• To store and update various web page metrics and counters
Eg. Cassandra, HBase
4) Graph databases are intended for use cases such as networks, where there are items (people or
web page links) and relationships between these items. While it is possible to store graphs such as
trees in a relational database, it often becomes cumbersome to navigate, scale, and add new
relationships. Graph databases help to overcome these possible obstacles and can be optimized to
quickly traverse a graph (move from one item in the network to another item in the network).
Following are examples of graph database implementations:
• Social networks such as Facebook and LinkedIn
• Geospatial applications such as delivery and traffic systems to optimize the time to reach one
or more destinations
Eg. FlockDB, Neo4j
6 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
MongoDB
• MongoDB is a NoSQL document database.
• It stores data in a type of JSON format.
• A record in MongoDB is a document, which is a data structure composed of key value pairs
similar to the structure of JSON objects.
• The field values may include numbers, strings, booleans, arrays, or even nested documents.
• The data in MongoDB is stored in form of documents. These documents are stored in
Collection and Collection is stored in Database.
Database
Database is a physical container for collections.
Collection
• Collection is a group of MongoDB documents.
• It is the equivalent of an RDBMS table. A collection exists within a single database.
• Collections do not enforce a schema.
• Documents within a collection can have different fields. This is possible because MongoDB is
a Schema-free database. Typically, all documents in a collection are of similar or related
purpose.
Document
• A document is a set of key-value pairs.
• Documents have dynamic schema. Dynamic schema means that documents in the same
collection do not need to have the same set of fields or structure, and common fields in a
collection's documents may hold different types of data.
• Practically, you don't need to define a column and it's datatype unlike in RDBMS, while
working with MongoDB.
Commands in MongoDB
• To create a collection with options before inserting the documents, use createCollection()
method.
Syntax:
[Link](name, options)
name is the collection name and options is an optional field that we can use to specify certain
parameters such as size, max number of documents etc. in the collection.
The cool thing about MongoDB is that you need not to create collection before you insert
document in it. With a single command you can insert a document in the collection and the
MongoDB creates that collection on the fly.
Syntax: db.collection_name.insert({key:value, key:value…})
• To select data from a collection in MongoDB, we can use the find() method.
This method accepts a query object. If left empty, all documents will be returned.
Syntax: db.collection_name.find()
To query, or filter, data we can include a query in our find()
db.collection_name.find(selection_criteria)
eg [Link]({StudentName : "Steve"})
7 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
find methods accept a second parameter called projection.
This parameter is an object that describes which fields to include in the results.
[Link]({}, {_id: 0, StudentId : 1})
• To update a document in MongoDB, we provide a criteria in command and the document that
matches that criteria is updated.
db.collection_name.update(criteria,update_data)
[Link]({name:"Jon Snow"},{$set:{name:"Kit Harington"}})
• MongoDB's remove() method is used to remove a document from the collection.
db.collection_name.remove(delete_criteria)
[Link]({StudentId: 3333})
If you don't specify deletion criteria, then MongoDB will delete whole documents from the
collection. This is equivalent of SQL's truncate command.
• To drop a collection, first connect to the database in which you want to delete collection and
then type the following command to delete the collection:
db.collection_name.drop()
Unstructured Data acquisition and structuring
Importing the corpus
• A corpus is basically a collection of text documents that you want to include in the analytics.
There are functions to read and parse MS Word, PDFs, plain text, or XML files among a few
other file formats.
Cleaning the corpus
• In R, the options are available to import a corpus with the tm package.
• We should usually start with removing the most frequently used, so called stopwords from the
corpus.
• Stopwords are the most common, short function terms, which usually carry less important
meanings than the other expressions in the corpus, especially the keywords. The package
already includes such lists of words in different languages.
Eg. > stopwords("english")
[1] "i" "me" "my" "myself" "we"….
• Sometimes, we can first call the tolower function from the base package to transform all
characters from upper to lower case.
• Then, remove all the punctuation marks from the text with the help of the removePunctutation
function. And we also remove the multiple whitespace characters from the document, so that
we find only one space between the filtered words.
Further cleanup
Further, if we do not really want to keep numbers in the package descriptions at all, and there are
some frequent technical words that can be ignored as well. Showing the plural version of nouns is
also redundant. removeNumbers function is used for removing the numbers.
8 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
Stemming words
• To get rid of the plural forms of the nouns and past tense and a few other similar variations of
the same terms, we can use some stemming algorithms, especially Porter's stemming algorithm
Eg > wordStem(c('cats', 'mastering', 'modelling', 'models', 'model'))
[1] "cat" "master" "model" "model" "model"
Lemmatisation
• While stemming terms, we remove characters from the end of words in the hope of finding the
stem, which is a heuristic process sometimes resulting in not-existing words.
• Another way to reduce the number of inflectional forms of different terms, instead of
deconstructing and then trying to rebuild the words, is morphological analysis with the help of
a dictionary. This process is called lemmatisation, which looks for lemma (the canonical form
of a word) instead of stems.
Analyzing the associations among terms
• Next step is to identify the association between the cleaned terms found in the corpus.
• This simply suggests the correlation coefficient computed on the joint occurrence of term-pairs
in the same document, which can be queried easily with the findAssocs function.
For eg, to see which words are associated with ‘data’
Segmentation of documents
• To identify the different groups of cleaned terms, based on the frequency and association of the
terms in the documents of the corpus, one might directly use tdm matrix to run, for example,
the classic hierarchical cluster algorithm.
• In this way we can structure plain English texts into numbers for further analysis using
machine learning algorithms.
Text mining
Text mining is the process of analyzing natural language text; in most cases from online content,
such as emails and social media streams (Twitter or Facebook). A big part of big data world is this
text data generated and stored in large volumes. Another important aspect of text data is that it can
be generated by anybody and have implications on business.
For example, a bad product review can damage the market image of the product or a social media
post about a social cause can create a campaign. In all these cases, text data plays a pivotal role of
influencing behavior. In the 21st century, it becomes important for organizations to invest in text
data and understand what insights it has on consumer behavior or product performance. Some
statistics suggest 80% of the information we store today is in text format, signifying the
commercial value of text mining. The field of Natural Language Processing (NLP), though a vast
9 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
field, could be thought of as a subfield of ML. In an alternative view, the text mining approaches
help in turning text into data for analysis, via the application of NLP and analytical methods.
TF-IDF
TF-IDF stands for “Term Frequency — Inverse Data Frequency”.
Term frequency works by looking at the frequency of a particular term you are concerned with,
relative to the document. Term Frequency (tf) gives us the frequency of the word in each document
in the corpus. It is the ratio of number of times the word appears in a document compared to the
total number of words in that document.
It increases as the number of occurrences of that word within the document increases. Each
document has its own tf.
Inverse document frequency looks at how common (or uncommon) a word is amongst the corpus.
Inverse Data Frequency (idf): used to calculate the weight of rare words across all documents in
the corpus. The words that occur rarely in the corpus have a high IDF score.
idf for a term ‘w’ is defined as
Where N denotes the total number of documents and dft is the number of documents where the
term ‘t’ appears.
Combining these two we come up with the TF-IDF score (w) for a word in a document in the
corpus. It is the product of tf and idf:
tfi,j is the number of occurrences of i in j,
dfi is number of documents containing i
and N is the total number of documents
Let’s take an example to get a clearer understanding.
Sentence 1: The car is driven on the road.
Sentence 2: The truck is driven on the highway.
10 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
In this example, each sentence is a separate document.
We will now calculate the TF-IDF for the above two documents, which represent our corpus.
From the above table, we can see that TF-IDF of common words was zero, which shows they are
not significant. On the other hand, the TF-IDF of “car”, “truck”, “road”, and “highway” are non-
zero. These words have more significance.
Text Summarization
1. Decompose the document D into individual sentences and use these sentences to form the
candidate sentence set S.
2. Construct the terms by sentences matrix A for the document D.
3. Perform the SVD (Singular value decomposition) on A to create the weighted term-frequency
vector Ai for each sentence i ϵ S, and the weighted term-frequency vector D for the whole
document.
4. For each sentence i ϵ S, Compute the relevance score between Ai and D, which is the inner
product between Ai and D.
5. Select sentence k that has the highest relevance score, and add it to the summary.
6. Delete k from S, and eliminate all the terms contained in k from the document. Re-compute the
weighted term-frequency vector D for the document.
7. If the number of sentences in the summary reaches the predefined value, terminate the
operation; otherwise, go to Step 4.
SVD is a method of matrix decomposition. Singular value decomposition is essentially trying to
reduce a rank R matrix to a rank K matrix. It means that we can take a list of R unique vectors, and
approximate them as a linear combination of K unique vectors.
11 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
Large scale data systems
Google File System (GFS)
• Google File System (GFS) is a scalable distributed file system (DFS) created by Google
Inc. and developed to accommodate Google’s expanding data processing requirements.
• GFS is made up of several storage systems built from low-cost commodity hardware
components. It is optimized to accommodate Google's different data use and storage needs,
such as its search engine, which generates huge amounts of data that must be stored.
• The GFS node cluster is a single master with multiple chunk servers that are continuously
accessed by different client systems.
• Chunk servers store data as Linux files on local disks. Stored data is divided into large
chunks (64 MB), which are replicated in the network a minimum of three times.
• Files are stored in hierarchical directories identified by path names. Metadata - such as
namespace, access control data, and mapping information - is controlled by the master,
which interacts with and monitors the status updates of each chunk server through timed
heartbeat messages.
GFS features include:
• Fault tolerance
• Critical data replication
• Automatic and efficient data recovery
• High aggregate throughput
• Reduced client and master interaction because of large chunk server size
• Namespace management and locking
• High availability
MapReduce
• The distributed processing using MapReduce is at the core of how a task on a big dataset is
divided according to the distributed storage.
• At a broad level, it consists of two procedures.
• Map, which performs operations like filtering and sorting; it processes the key-value pair
and generates a intermediate key-value pair.
• Reduce merges all the intermediate values with the same key.
• If a problem could be expressed this way, then it’s possible to use a MapReduce to break
the problem into smaller parts. Over the years, this model has been successfully used in
many real-world problems.
12 Bindy Wilson
DATA SCIENCE TYCS SEM VI UNIT 3
Hadoop Ecosystem
There are plenty of resources on Hadoop. Taking a broad view, the Hadoop framework consists of
the following three modules.
Hadoop Distributed File System: This is the storage part of Hadoop; the core where the data
chunks really reside.
Hadoop YARN: Yet Another Resource Negotiator, this is also known as the data operating system.
Hadoop MapReduce: MapReduce decides the execution logic of what needs to be done with the
data. The logic should be designed in such a way that it can execute in parallel with smaller chunks
of data residing in a distributed cluster of machines.
On top of this, there are many additional software packages specially designed to work on the
Hadoop framework, namely Apache Pig, Hive, HBase, Spark, and more.
Spark
• Spark provides lightning-fast cluster computing (similar to distributed computing with
multiple nodes working together).
• Spark has an advanced Directed Acyclic Graph (DAG) based execution engine which
makes it 100 times faster than Hadoop MapReduce in RAM or memory and 10 times faster
on disk.
• Contrary to Hadoop, which supports only Java, in Spark, you can write applications using
Java, Scala, Python, and R.
• Spark also offers SQL, streaming, machine learning, and graph libraries that could be
combined in any fashion to create an application pipeline.
• Apart from accessing data from HDFS, in Spark, you can connect to HBase, Cassandra,
S3, and many more.
13 Bindy Wilson