Big Data Viewer

Level 01

Introduction

The person who completes this course will be able to identify the necessary principles.
for the utilization of data, with large volumes, variety, and growth.

As a Big Data viewer, you can develop applications that obtain, clean
and process all kinds of data from various sources, the reason for doing this is to generate and
to present graphs that allow for an overview of the behavior of a
organization, this vision is highly valued, as it can suggest the direction of an entire
company aiming to improve processes, minimize costs, and find
growth opportunities.

Registration and data collection from various sources

Filters, enrichment and classification of data
Analysis, modeling, and prediction of the data
Delivery and visualization of the data

Lesson 01

What is Big Data and why is it important?

It refers to the exploitation of an immense amount of data, it requires the

ability to obtain, store, manipulate, and analyze millions of data, which would be
impossible to perform with conventional analysis tools (Data capture,
Relational databases, Pivot tables.

Characteristics of Big Data and fields of application

Main characteristics

Volume
Variety (Structured and unstructured data)
Speed

Other features
 Veracity
Value

Purposes of Big Data

Improve operations
Complex decision making
Cost reduction
Reduction of times
Deployment of personalized offers
Business intelligence

Lesson 02

The Big Data model

Ecosystem

Information Technology Services

They are divided into two parts:

Data management and storage. It consists of three areas: the sources of

data (structured and unstructured), the core of Big Data, the
Management of operational data.

Analysis and applications. Big Data analytics and the Users.

Workflow

Data acquisition
Big Data
Big Analytics
Users

Components of Big Data

Life cycle

Registration and data collection from various sources

Filters, enrichment and classification of data
 Analysis, modeling, and prediction of the data
Delivery and visualization of the data

Big Data Infrastructure

Clusters. Various computers connected to each other, each one is known as

NODE. Advantages:

Trabajo en paralelo
High performance
Support for high workloads
Scalability

Software used in the Clusters is based on open source platforms.

Distributed file system

Core that maps and reduces the data
Set of libraries

Level 02

Lesson 01

What is a JSON and how is it used?

It is a simple data exchange format, which started by communicating

technologies like JavaScript with Python, PHP, '.net' and others. It has become popular for
its lightness and ease of use.

What is an API and what is it for?

An API (Application Programming Interface) is a series of instructions that allows you

they indicate to the system how to satisfy the user's request and return with a response,
thanks to this tool, information can travel from one side to another and different
devices or applications can connect with each other, allowing for purchases
reservations or publications.

Types of API

Select. Returns the record of an object

Returns a list of records related to a specific object.
Update. Makes changes to the database such as creating or deleting records.
Preparation for data extraction in social networks

The API of a social network is the main method to receive and send data from the
platform.

Adquisición de datos de redes sociales

Before starting data acquisition, install and import the required libraries
in Python (Social network library, Requests Library), then declare the
following variables:

Import Facebook
Import requests
Token = "EAAC."
Graph = Facebook GraphAPI (token)
cantidadComentarios = 100
PageId = 13254565767
CuentaLikes = 0
ListaComents = [ ]
Bandera = False
Coments = graph.get_connections (PageId, ‘feed’)

Extraction of audio and video files

Lesson 02

Types of databases

NoSQL (Not only SQL) or non-relational databases:

Key-Value Oriented
Document-oriented
Column-oriented
Graph-oriented

Management of NoSQL databases

Connection to NoSQL database with programming

Level 03

Lesson 01

Big Data Analysis

Identify:

Patterns
Correlations
Tendencias
Customer preferences

Business intelligence generation

Objectives

Reduction of operating costs

Improve decision-making
Offer new products and services

Benefits

Service improvement
Generation of efficiency in operations that provide an advantage over
competition.

Big Data analysis techniques

Specialization in data science

The disciplines for analyzing Big Data are:

Predictive analysis
Data mining
Text analysis
Statistical analysis
Machine Learning
Data visualization
Other tools based on NoSQL data analysis
Sectors that use Big Data

Travel agencies and hotel chains

Medicine and healthcare
Government
Department stores and online stores

Machine Learning or Automatic Learning

It refers to the study of data with the ability for the machine to learn without
need to be explicitly programmed. It is achieved through the construction of
algorithms that create a model from a sample of data, based on it, the
machines create predictions or express decisions on their own. They work to
complex models, difficult to carry out in a conventional way, as these are finished
building alone.

Classification

Machine Learning–Big Analytics or Big Data–Unsupervised

Statistical classification
Clustering
Regression
Anomaly detection
Association rules

Sentiment analysis

Classify the polarity of a comment

Programming logic

Import requests

Apikey = ‘ab881ef9-5941-45d7-95ª7-595fc89d129d’
Lenguaje = ‘spa’
leagueRequest =
‘[Link]
=(2)’

mensaje = “me gusto mucho el apoyo pero todo esta muy mal”
ligaPeticion = [Link]([Link])
jsonRespuesta = [Link](ligaPeticion).json()
Structure of the analysis result

List of positive elements

List of negative elements
Total

Lesson 02

Web Scraping

Web Scraping Techniques

Techniques for obtaining data from a webpage, 'scraping' means to scrape:

Copy and paste

Regex
Data mining algorithms
HTML parsing
Applications or programs

Data acquisition from a webpage

For example, to obtain the exchange rate data for the dollar.

Import the necessary libraries

Read the webpage and convert it into string type
Locate the data that interests you
Extract only the data you require
Give them treatment according to their needs
Test the development

Take the following into account when developing this type of programs:

Use the find method to locate the substring within the text, this method returns
the position within the complete text, you must count the positions to define the data
What do you want from the beginning to the end? Create functions to obtain the data of interest, in the
part of the treatment of the acquired data, is where you will program the logic to
continue, consider the type of update, keep a program in a cycle so that
run continuously until you want to stop it.
Lesson 03

Creation of graphics on the web

To program and deploy a scatter plot, follow these steps:

In a file add the basic HTML structure

Import the jQuery libraries and the graph libraries
Add a Script tag to start coding in JavaScript
Add a char type object to indicate which graph you are going to use, in this
case, will be scatter.
Agrega los siguientes objetos para los títulos de la gráfica: title, subtitle, xAxis,
yAxis, legend, finally add the series list.
Save the HTML file and open it with the browser to view the chart.
of complete dispersion.

Bar chart with detail

In an HTML file, add the structure of the HTML, the libraries, and the titles.
of the graphs
Add the plotOptions object to include the percentage of each bar.
Add the series object including the total data for each bar to indicate
the display of the detail of each bar.
Then add the drilldown property with its respective name.
Agrega otro objeto llamado drilldown, seguido de una lista series, para
specify the details of each bar.
Agrega un objeto con información a detalle de cada barra y añade el ID con el
name you indicated as drilldown in the total objects.
Reload the page and check the detail functionality on each bar.

Creation of a web service

A web service is a program that runs on the server side to exchange

information between applications. These can provide information in two formats XML
and JSON. A web service requires the following:

PHP server version 5.6

No SQL database
No SQL data library for PHP

Library installation on PHP Server

 Go to the folder where your server files are located and paste the
Library ".dll" in the extensions folder, rename the library.
Open the configuration file and register the library, restart the server.
Check the installed extensions.

Creation of a web service

Para programar un servicio que regrese una colección en una base de datos no
relational, follow these steps:

Open the collection with the No SQL database

Select the database you are going to consult
Select the collection you will need from the database
Retrieve all elements from the collection with the find instruction
Enlista todos los elementos y conviértelos en JSON
Print each of the elements of the collection
Close the database connection
Finally, include the lines you just wrote in an exception for
control the error, in case the connection cannot be made.

Real-time chart creation

To develop an application that works in real time, you need to consult a

database using a web service in PHP that returns a JSON, for
to program it, follow these steps:

Create a PHP file named data

Connect the No SQL database and select the collection you need to graph.
Add a switch that captures the consult parameter from the get request and adds two.
cases
When the parameter to query is 1, add a query to the database where only
I returned the last record added to the collection.
When the parameter to query is not 1, add a query to the database in
where I return all the elements of the collection, this is known as default.
En los dos casos agrega los resultados en una lista y regrésalos como JSON.
Test the web service in both cases and verify that the JSON is correct.
built.

To draw a graph with HTML and JavaScript in real time, starting from a chart
online, follow these steps:

Add the series variable in the load part, so that the graph updates every time.
that a new record is entered into the database.
Add two temporary variables in the document ready part, one for x and another for.
Yes, there you store the last record added to the chart.
Create the get and set functions that will allow you to input and modify those two.
variables.
Add an ajax request, this will allow you to perform queries and update the chart.
without having to refresh the page, this request will require two pieces of data, the url of
web service with the request to bring all the data and the type of request in this case
it will be get.

Go through the JSON of the content of each request and save each of its elements.
in a row.
Store the temporary variables in a register of X and Y.
Finally, assign the array to the data parameter so that the data is drawn in the
chart.
Now you can see how the data is plotted on the graph, but it's still not in
real time.

To make the graph update in real time, follow these steps:

Modify the interval function, which already includes the graph, by adding a request to the service.
web, but this time with the parameter of 1 to bring only the last record.
Compare the X and Y of the last record in the database with the record you drew.
in the graph, if the two are different, it means that a new record was added to the
database and it is when the new points will be added to the graph.
Store the new records in the temporary variables.
Finally, assign an interval of 1000 milliseconds to run this request again.

In this way, you can create applications that allow you to observe and analyze in
real-time behavior of users on social networks, up to the latest
movements in the stock markets.

Structure of a dashboard

Control panel or dashboard. It is a real-time visual summary, with information

of the business and can issue alerts for critical situations, it even offers the possibility
for consultation from any electronic device.

Process to design a dashboard

Define los KPIs de la organización, enlístalos por orden de importancia, selecciona los
three or four most important, if there are more than four you must divide them into different ones
groups to avoid overloading it, select the graphs that show the behavior of
the chosen KPIs. Arrange the graphs on a single panel, place filters, functions of
ordering and descriptions.
Evaluation

Assuming that the request returns a JSON, what would it print to the console?
next segment of code:
$.ajax({
url: "[Link]?Consultar=0",
type: 'get',
success: function(DatosRecuperados) {
DatosRecuperados = [Link](DatosRecuperados);
$.each(DatosRecuperados, function(i,o){
[Link](parseInt(o.x));
[Link]( parseInt(o.y));
});
Respuesta del usuario:
The coordinates of the JSON that were sent as a response
Resultado:

Correct!
Results of the question

If with the following line of code a query is made to the NoSQL database of
last entered record, how should you modify the instruction to bring all the
elements in ascending order?
Respuesta del usuario:
$cursor = $collection->find()->sort(array('$natural' => 1))->asd(1);
Result:

You need to reinforce the topic: Web Services and Creation of real-time graphics
Results of the question

Iván is programming an application to display a line graph, but the

the graph does NOT display the data. If you are using the following object for the data,
What should Iván do with the data to solve the problem?

data: [[161.2], [167.5], [159.5], [157.0], [155.8]]

Respuesta del usuario:
Place the data in coordinate form so that it can be read.
Resultado:

Correct!
Results of the question
What output does a web service in PHP have with the following lines of code
suponiendo que la variable "$cursor" contiene esta lista de objetos: [{“x”: “1”, “y”:
“2”},{“x”: “2”, “y”: “5”},{“x”: “3”, “y”: “8”}]

foreach ($cursor as $doc) {

array_push($lst, $doc);
}
echo json_encode($lst);
Respuesta del usuario:
[{1,2},{2,5},{3,8}]
Resultado:

You need to reinforce the topic: Web Services and Creating real-time graphics
Results of the question

It is a visual summary of an analysis, containing information about a business:

User's response:
Dashboard
Resultado:

Correct!
Results of the question

Assuming that the 'ajax' request from the following segment of code returns a JSON,
what would it print to the console:

$.ajax({
url: "[Link]?Consultar=0",
type: 'get',
success: function(DatosRecuperados) {
$.each(RecoveredData, function(i, o) {
[Link]( parseInt(o.x));
[Link](parseInt(o.y));
});
User's response:
The coordinates of the JSON that were sent as a response
Resultado:

You need to reinforce the topic: Creating real-time graphs

Results of the question
Martha is programming a graph, but it does NOT display the title due to a
error in this segment of the program. How can the error be corrected?

title: {
titulo: 'Cuenta usuarios'
},
User's response:
Changing the title name to text
Resultado:

Correct!
Results of the question

Maria is installing a NoSQL database in php and has already placed the file in the
extensions folder. What do you need to do to be able to use the library?
User's response:
Check in the installed extensions
Resultado:

You need to reinforce the topic: Web Service

Results of the question

If with the following line of code a query is made to the NoSQL database of
last entered record, how should you modify the instruction to bring the first one
record? $cursor = $collection->find()->sort(array('$natural' => -1))->limit(1);
User's response:
$cursor = $collection->find()->sort(array('$natural' => 1))->limit(1);
Result:

Correct!
Results of the question

José has the following output from a web service and is using this instruction to
read the data. What will José get as a result?

<Sender>
<Nombre>Nombre del remitente</Nombre>
Sender's email
</Sender>
<Recipient>
<Nombre>Nombre del destinatario</Nombre>
Recipient's email
 </Recipient>

</Remitente>
User's response:
A graph without data because the information being graphed does not correspond to the
type of graph
Result:

You need to reinforce the topic: Creating real-time charts

José has the following output from a web service and is using this instruction to
read the data. What will José obtain as a result?

<Sender>
<Nombre>Nombre del remitente</Nombre>
Sender's email
</Sender>
<Recipient>
<Nombre>Nombre del destinatario</Nombre>
Recipient's email
</Destinatario>

</Sender>
Respuesta del usuario:
An error in the reading because it is not a valid input for the instruction being used.
Resultado:

Correct!
Results of the question
How many dashboards do you need if you selected 10 KPIs?
User's response:
3
Resultado:

Correct!
Results of the question
Yvette is programming a graph, but it does NOT display the x-axis title.
graphic and found the error in this part of the program. What is the cause of this error?

ejex: {
title: {
enabled: true,
Height (cm)
}
},
Respuesta del usuario:
El nombre del objeto es incorrecto
Resultado:

Correct!
Results of the question
If with the following line of code a query is made to the NoSQL database of
last record entered, how should you modify the instruction to bring the first one
registro? $cursor = $coleccion->find()->sort(array('$natural' => -1))->limit(1);
User's response:
$cursor = $collection->find()->sort(array('$natural' => 1))->limit(1);
Result:

Correct!
Results of the question
What output does a web service in PHP with the following lines of code have?
assuming that the variable "$cursor" contains this list of objects: [{"x": "1", "y":
“2”},{“x”: “2”, “y”: “5”},{“x”: “3”, “y”: “8”}]

foreach ($cursor as $doc) {

array_push($lst, $doc);
}
echo json_encode($lst);
Respuesta del usuario:
[{“x”: “1”, “y”: “2”},{“x”: “2”, “y”: “5”},{“x”: “3”, “y”: “8”}]
Result:

Correct!
Results of the question
What should you do in case the KPIs you selected to make a
Does the dashboard exceed 4?
Respuesta del usuario:
Distribute them in different dashboards
Result:

Correct!
Results of the question
Assuming that the 'ajax' request from the following code segment returns a JSON,
what would it print to the console:

$.ajax({
url: "[Link]?Consultar=0",
type: 'get',
success: function(DatosRecuperados) {
$.each(RecoveredData, function(i,o){
 [Link](parseInt(o.x));
[Link](parseInt(o.y));
});
User's response:
A chart with the data included in the JSON of the request
Resultado:

You need to strengthen the topic: Creating real-time graphics

Results of the question
Carlos is developing a bar chart but he has a problem with it.
code fragment and the graph is NOT displayed. What is the cause of the error?

chart: {
type: 'graph-bar'
},
User response:
The object you are programming does not exist.
Resultado:

You need to reinforce the topic: Creating graphics on the web

Results of the question
To install a No-SQL database library, Juan renamed the library.
and placed it in the extensions folder and then restarted the server, but NOT
She installed the library. What is causing the failure?
User's response:
The library is not enabled in the PHP configuration.
Result:

Correct!
Results of the question
Karen is having problems programming a php web service and needs to fetch from a database.
of NoSQL data called "Departments" the kitchen collection. What is the error in
the code?

$mongo = new MongoClient();

$db = $mongo->selectDB('kitchen');
$collection = new MongoCollection($db, 'Departments');
Respuesta del usuario:
The name of the collection and the database are swapped.
Result:

Correct!
Level 04

Lesson 01

Data science

Its goal is to have a better understanding of Big Data through study techniques.
different from conventional ones. Data science is a mix of: Statistics and
mathematics, computer science, business administration.

Functions

In order for a data scientist to gain knowledge of Big Data, they must undertake the
next tasks with the data:

Acquire them
Analyze them
Filter them
Extract them
Represent them
Refine them
Interact with them

Framework for Big Data

To the structure that serves as a basis for developing and organizing software with various
Integrated tools are called a framework.

The most used in Big Data is Apache Hadoop, which allows processing of
large datasets distributed across clusters with the help of simple models
programming, is designed to scale vertically being able to have thousands of
computers, where each one can offer local storage and processing.

Apache Hadoop includes four modules:

Common
Distributed file system
YARN
MapReduce

Other tools: Database, Infrastructure for DataWarehouse, Engines.

The differences between Apache Hadoop and others are:

Handles data smoothly

It has a simplified programming model.
It is easy to manage

Formation of a work team

The requirements to start taking advantage of Big Data are as follows:

The entire company must be aware of the impact of the proper use of data on the
business and in their daily work
It is not necessary to know technical aspects
The company director must be the first to understand the benefit it brings.
this technology and communicate it to others
Hire a Big Data specialist

The person in charge of Big Data:

You must define the needs and areas of opportunity of the company, the technology.
suitable to meet those needs, they can be applications with Machine
Learning, another type of real-time analytics or just another business intelligence.
robust. It should also be defined whether the processing will be in the cloud or internal.

Hiring the right personnel

Project Director
Cloud Computing

If you want to work internally, then more staff must be hired.

Staff dedicated to servers, networks, and software development.

Another necessary person is the Data Scientist who is an expert in Big Data analytics.
Data or at least one business analyst who relies on cloud tools.

Creative freedom in the work team.

Lesson 02

Evaluation

Which protocol is recommended to connect your Web service with the database?
of relational data?

User's response:
HTTP
Result:

Correct!
Results of the question

What is the type of database schema that is most recommended for you?
integrative application?

User's response:
Star
Resultado:

Correct!
Results of the question

In the relational database you created, the following are essential fields.
for your model, except:

User's response:
Comment
Resultado:

Correct!
Results of the question

¿Cuál es el ID del siguiente perfil de la red social \"capacitateparaelempleo\"?

User's response:
388594514598480
Resultado:

Correct!
Results of the question
In your application, which of the following methods is correct to display the
frequency of likes according to the hour?

User's response:
Histogram
Resultado:

Correct!
Results of the question

In the application you created, what is the minimum number of nested cycles needed to
get the comments of each post on the social network?

Respuesta del usuario:

2
Result:

Correct!
Results of the question

To build a dashboard with two elements, in addition to the behavior of

feelings by hour, what other KPI can you include?

User's response:
Likes per week
Resultado:

Correct!
Results of the question

To enable the web page of your application's server, you must perform the following
tasks, except:

User's response:
Upload the files via FTP
Resultado:

Correct!