ELK

Before ELK stack consisted only eleas4csearch, logstash and kibana and later beats got
added with the other three components due to community expansion and increase in use
cases, then renamed to Elas4c Stack.
The ELK stack is the elas4c stack with more flexibility.
Elas%c Stack - Elas4c Stack refers to a set of opensource products that have been developed
by elas4c to help it's users collect data from different types of sources and then analyze the
collected data and represent it in an easy to understand and aesthe4c visualiza4on. This is
done so that meaningful observa4ons can be made.
• Highly scalable
• Does all the processing in real 4me

Components of ELK:`
1. Elas%c Search - Used for storing & searching collected Data
2. Logstash - Used for collec4ng & filtering the input data
3. Kibana - Provides a graphical user interface
4. Beats - Mul4ple light weight data collectors

1. Elas%c search: JSON based search engine over HTTP

Elas4csearch is a NOSQL Database that was developed based on apache Lucene search
engine. It can be used to index and store mul4ple different types of structured and
unstructured data and documents because it is basically schema less and works in real 4me
and also a RESTful interface (helps to interact with it easily from mul4ple interfaces).
Apache Lucene:
It is a highly performed, fully featured search engine or text engine built in java and
considered as heart of elas4csearch.

2. Logstash:
Logstash is a collec4on agent and is used to collect both heterogenous/non-
heterogenous data from various sources. It has the capability to screen, breakdown and make
string altera4ons in the data it collects. AXer it has collected and filtered the data it then send
it to elas4c search for storage.

3. Kibana:
Kibana is graphical user interface that is used to display the data that was collected
and stored in Elas4csearch. It displays them with appealing visuals so that the data could be
easily understood and analyzed, it does so using mul4ple different types of visuals like bar
charts, pie chart, world maps, heat maps, co-ordinate maps etc.
Features of Kibana:
• Discover the data by exploring it.
• Analyze the data by applying different metrics.
• Visualize the data by crea4ng different types of charts.
• Apply machine learning on the data to get data anomaly.
 • Monitor your applica4on using APM.
• Manage users and roles.
• A console to run Elas4csearch expressions.
• Play with 4me-series data using Timeline.
• Monitor elas4c Stack using monitoring.

4. Beats:
Beats is similar to Logstash in the maZer of fact that they both collect the data that
will be later stored and analyzed, but beats differs in the method of collec4on.
Here beats are mul4ple small soXware installed on different servers from where they collect
the data and send it to Elas4csearch.
Types of beats:
1. Filebeat – log files (ingest log data and sends to ES to store it)
2. Metricbeat – metrics (gather metrics (cpu, memory, server) from server)
3. Packetbeat – network data (gathers network data (use to prevent hackings))
4. Auditbeat – audit data (audi4ng informa4on)
5. Winlogbeat – windows event logs (events happening in windows server)
6. Heartbeat – up4me monitoring (whether the server is available or not)
7. fuc4onbeat – serverless shipper

ELK flow: Any errors in the whole process will be sent in the form of no4fica4ons to whoever
responsible.

First the Beats are aZached to remote servers from where the beats
collect informa4on from various sources.

AXer collec4ng all the data needed they either ship the data to
Logstash for filtra4on or directly send the data to Elas4csearch.

The data is then stored in Elas4csearch, from there it will not be

directly sent to Kibana. Kibana needs to check where elas4csearch is
and then go and get the data itself.
Features if ELK:
1. System Performance Monitoring
2. Log Management
3. Applica4on Performance Management
4. Applica4on data Analysis
5. Security Monitoring and Altering
6. Data Visualisa4on

ELK hands-on:
1. Collect sta4c Apache logs using Logstash and analyze them using Kibana
2. Collect sta4c 'CSV' using Logstash and analyze them using Kibana
3. Collect real-4me web logs & configure beats to inject them into Elas4csearch, and analyze
them using Kibana

Х-РАСК:
X-Pack is an Elas4c Stack extension that provides security, aler4ng, monitoring, repor4ng,
machine learning, and many other capabili4es. By default, when you install Elas4csearch, X-
Pack is installed.

Elas%csearch cluster:
An Elas4csearch cluster is a group of one or more Elas4csearch nodes that are
connected together. Although each node has its own purpose and responsibility, each node
can forward client requests (coordina4on) to the appropriate nodes.
The following are the nodes used in an Elas4csearch cluster:
• Master-eligible node: The master node's tasks are primarily used for lightweight
cluster-wide opera4ons, including crea4ng or dele4ng an index, tracking the cluster
nodes, and determining the loca4on of the allocated shards. By default, the master-
eligible role is enabled. A master-eligible node can be elected to become the master
node (the node with the asterisk) by the master-elec4on process. You can disable this
type of role for a node by sejng node, master to false in the elas4csearch ym! file.
• Data node: A data node contains data that contains indexed documents. It handles
related opera4ons such as CRUD, search, and aggrega4on. By default, the data node
role is enabled, and you can disable such a role for a node by sejng the [Link] to
false in the [Link] file.
• Ingest node: Using an ingest nodes is a way to process a document in pipeline mode
before indexing the document. By default, the ingest node role is enabled-you can
disable such a role for a node by sejng [Link] to false in the [Link]
file.
• Coordina%ng-only node: If all three roles (master eligible, data, and ingest) are
disabled, the node will only act as a coordina4on node that performs rou4ng requests,
handling the search reduc4on phase, and distribu4ng works via bulk indexing.

NOTE: There should be atleast 2 or 3 master nodes, as many as possible data nodes, 1
coordina4ng node and 2 or 3 ingest nodes.