BIG DATA
TOO BIG TO IGNORE
SÜMEYYE KAYNAK
OUTLINE
ElasticSearch
ELASTICSEARCH
It is the Nosql technology used in text search operations in big data.
Elasticsearch infrastructure includes Apache Lucene and Solr libraries.
Apache Lucene and Solr libraries are open source.
In the Elasticsearch module, data is indexed while it is being saved.
ELASTICSEARCH
In the word search, not all data is searched. Results can be found quickly through the
created index list.
COMPANIES USING ELASTICSEARCH
Term Documents
1 The old night keeper the keep in the town And <6>
2 In the big old house in the big old gown. Big <2> <3>
THE
3 INVERTED
The house in the INDEX
town had the big old keep. Dark <6>
4 Where the old night keeper never did sleep. Did <4>
5 The night keeper keeps the keep in the night. Gown <2>
6 And keeps in the dark and sleeps in the light. Had <3>
House <2> <3>
6 document to index In <1> <2> <3> <5> <6>
Keep <1> <3> <5>
Keeper <1> <4> <5>
Term Documents Keeps <1> <5> <6>
Sleeps <6> Light <6>
The <1> <2> <3> <4> <5> <6> Never <4>
Town <1> <3> Night <1> <4> <5>
Where <4> Old <1> <2> <3> <4>
Sleep <4>
TERMINOLOGY
Relation Databases Elasticsearch
Database Index
Table Type
Row Document
Column Fields
Schema Mapping
DOCUMENT AS JSON
Since Elasticsearch is a document-type
Nosql database, documents are stored in
json format.
GENERAL FEATURES
Elasticsearch is scalable up to petabytes of structured and unstructured data.
Elasticsearch uses denormalization to improve the search performance.
Elasticsearch is one of the popular enterprise search engines, and is currently being
used by many big organizations like Wikipedia, The Guardian, StackOverflow,
GitHub etc.
Elasticsearch is an open source and available under the Apache license version 2.0
KEY CONCEPTS
Node: It refers to a single running instance of Elasticsearch.
Cluster: It is a collection of one or more nodes. Cluster provides collective indexing
and search capabilities across all the nodes for entire data.
Index: It is a collection of different type of documents and their properties. Index
also uses the concept of shards to improve the performance.
KEY CONCEPTS
Document: It is a collection of fields in a specific manner defined in JSON format.
Every document belongs to a type and resides inside an index. Every document is
associated with a unique identifier called the UID.
Shard: Indexes are horizontally subdivided into shards. This means each shard
contains all the properties of document but contains less number of JSON objects than
index. The horizontal separation makes shard an independent node, which can be
store in any node.
KEY CONCEPTS
Replicas: Elasticsearch allows a user to create replicas of their indexes and shards.
Replication not only helps in increasing the availability of data in case of failure, but
also improves the performance of searching by carrying out a parallel search
operation in these replicas.
ADVANTAGES
Elasticsearch is developed on Java, which makes it compatible on almost every
platform.
Elasticsearch is real time, in other words after one second the added document is
searchable in this engine.
Elasticsearch is distributed, which makes it easy to scale and integrate in any big
organization.
Creating full backups are easy by using the concept of gateway, which is present in
Elasticsearch.
ADVANTAGES
Handling multi-tenancy is very easy in Elasticsearch when compared to Apache Solr.
Elasticsearch uses JSON objects as responses, which makes it possible to invoke the
Elasticsearch server with a large number of different programming languages.
DISADVANTAGES
Elasticsearch does not have multi-language support in terms of handling request and
response data (only possible in JSON) unlike in Apache Solr, where it is possible in
CSV, XML and JSON formats.
ELASTIC SEARCH-INSTALLATION
For Windows OS;
Download Elasticsearch from [Link]
The downloaded file is copied to the C:
ELASTIC SEARCH SERVER
To run the Elastic Search server;
CRUD OPERATION
In client/server architecture, a request-response communication is performed between
client-server.
Http methods determine the purpose of the request made to the server side.
HTTP METHODS
Common request types are:
GET Method Description
POST GET Request to read a Web page
HEAD Request to read a Web page’s header
HEAD PUT Request to store a Web page
OPTIONS POST Append to a named resource (e.g. a Web page)
DELETE Remove the web page
TRACE
TRACE Echo the incoming request
PUT CONNECT Reserved for future use
DELETE OPTIONS Query certain options
CONNECT
GET AND POST METHODS
Get method: GET is used to request data from a specified resource.
Post method: POST is used to send data to a server to create/update a resource.
GET METHOD
Get requests can be cached.
In GET method, values are visible in the URL.
Get requests remain in the browser history.
Get requests can be bookmarked.
Get requests should never be used when dealing with sensitive data
GET requests have length restrictions
GET requests are only used to request data (not modify)
/test/demo_form.php?name1=value1&name2=value2
GET METHOD
POST METHOD
POST requests are never cached
In POST method, values are not visible in the URL.
POST requests do not remain in the browser history
POST requests cannot be bookmarked
POST requests have no restrictions on data length
/test/demo_form.php
POST METHOD
DIFFERENCE BETWEEN A GET AND POST
ELASTIC SEARCH
MAPPING CONCEPTS ACROSS SQL AND ELASTIC SEARCH
SQL ELASTIC SEARCH
column field
row document
table index
schema implicit
PUT METHOD
This information is sent to the table named
product of the database named exam with the id
number of 123.
UPDATE WITH POSTMAN
POST METHOD
QUERY WITH POSTMAN
QUERY WITH POSTMAN
QUERY WITH POSTMAN
DELETE COMMAND WITH POSTMAN
UPDATE COMMAND WITH POSTMAN
DELETE COMMAND WITH POSTMAN
SEARCH COMMAND WITH POSTMAN
GET ALL INFORMATION IN THE DATABASE
ELASTIC SEARCH [Link] INTEGRATION
You can install NEST from the package manager console.
PM> Install-Package NEST
Alternatively, simply search for NEST in the package manager UI.
CONNECTION
Connecting to a single node
Connecting to multiple nodes using a connection pool
INDEXING
All the calls have async variants:
var response = await [Link](tweet, idx => [Link]("mytweetindex"));
// awaits a Task<IndexResponse>
GETTING A DOCUMENT
SEARCH
UPDATE
DELETE