Q) APPLICATIONS OF BIG DATA Q) What is NOSQL? Q) HADOOP ARCHITECTURE ?

[Link] -Big data has started making a massive difference in the NOSQL is a type of database system designed for handling large volumes of The Hadoop Architecture Mainly consists of 4 components. 1. MapRedu
healthcare sector -Big data is used to analyze large volumes of health- unstructured and semi-structured data. Unlike traditional relational HDFS(Hadoop Distributed File System) 3. YARN(Yet Another Resource
related data, including patient records, and genetic information, to predict databases, NoSQL databases use flexible data models that adapt to changes Negotiator) 4. Common Utilities or Hadoop Common
disease outbreaks. Example: Predictive Analytics for Disease Outbreaks and can scale horizontally for increased data capacity. Originally meaning
[Link] and Banking Sector -Financial institutions use big data analytics "non-relational," NoSQL has evolved to signify "not onlySQL," Features of
to detect fraudulent activities in real-time. -Big data analytics help banks Nosql 1. Non-relational data storage system: They are not like to the
and customer behaviour on the basis of investment patterns, shopping relational data model. NoSQL systems store and retrieve data from many
trends, motivation to invest -Example: Fraud Detection [Link] -Retailers formats like key-value pairs or document-oriented or column-oriented 2.
use big data to analyze customer shopping behaviours, including purchase Schema free: NoSQL databases are gaining popularity because of their
history and online browsing patterns. For example, Amazon's support for flexibility to the schema 3. Free of joins: NoSQL databases allow
recommendation system suggests products based on a user's browsing and users to extract data using simple interfaces without joins 4. Works on
purchase history [Link] and Logistics -Transportation companies many processors: NoSQL systems allow users to store the database on
use big data to optimize delivery routes, reduce fuel consumption, and multiple processors and still maintain high-speed performance. 5. Supports
improve overall efficiency. Example: Route optimization. [Link] - linear scalability:. There is an increase in the performance after adding
Manufacturers use big data and IoT sensors to monitor the condition of multiple processors. NoSQL allows easy scaling and supports linear [Link]
machinery and equipment in real-time. It can identify when equipment is scalability 6. Adherence to CAP theorem: NoSQL databases do not offer 1)In MapReduce Input is provided to the Map() function then it’s output
likely to fail.6. Marketing and Advertising -Marketers use big data to support for ACID properties (Atomicity, Consistency, Isolation and as an input to the Reduce function and after that, we receive our final o
segment their customer base into distinct groups based on demographics, Durability). On the contrary, they support by Brewer‘s CAP theorem Let’s understand What this Map() and Reduce() does. 2)The Map() fu
behaviours, and preferences. [Link] and Utilities-Big data is used in the (Consistency, Availability and Partition tolerance) here breaks this DataBlocks into Tuples that are nothing but a key-valu
energy sector to manage and optimize the distribution of electricity in These key-value pairs are now sent as input to the Reduce (). 3)The Re
smart grid, Eg-smart grid management [Link]-Farmers utilize big function then combines this broken Tuples or key-value pair based on
data and IoT sensors to monitor soil conditions, weather patterns, and crop value and form set of Tuples, and perform some operation . Finally, the O
health. This data is used to make informed decisions about planting, is Obtained. Job tracker a)Job Tracker's role is to accept MapReduce job
irrigation, and pest control. [Link] and Media -Example: Content clients and process the data using Name Node. Task tracker a) It perfor
Recommendations -Streaming platforms like Netflix and Spotify use big function of a slave node for Job Tracker. b) It receives the task and cod
data algorithms to recommend movies, TV shows, and music to users Job Tracker and applies it to the file. This procedure is also known as a M
based on their viewing or listening history.

Q) Rack Awareness
What is a rack? The Rack is the collection of around 40-50 DataNodes
connected using the same network switch. If the network goes down, the Q) Hbase architecture? -HBase architecture has 3 main components:
whole rack will be unavailable. A large Hadoop cluster is deployed in HMaster, Region Server, Zookeeper. All the 3 components are below :
multiple racks. What is Rack Awareness in Hadoop HDFS? Rack Awareness HMaster – The implementation of Master Server in HBase is HMaster. It is a
in Hadoop HDFS is a feature that helps with fault tolerance by placing process in which regions are assigned to region server as well as DDL
replicas of data blocks in different racks. This ensures data availability in (create, delete table) operations. It monitor all Region Server instances
case of network issues or partition failures within the cluster. 1. Block present in the cluster. In a distributed environment, Master runs several
Placement for Fault Tolerance: - HDFS uses rack awareness to decide background threads. HMaster has many features like controlling load
balancing, failover etc. Region Server –HBase Tables are divided horizontally [Link]
where to place replicas of data blocks, enhancing fault tolerance. 2. Cluster
by row key range into Regions. Regions are the basic building elements of The Hadoop Distributed File System (HDFS) is a Hadoop distributed file s
Worker Rack ID: - The Hadoop master daemon obtains the rack ID for
HBase cluster that consists of the distribution of tables and are comprised of It is built with master/slave architecture. This architecture consists of a
cluster workers by calling an external script or Java class specified in the Name Node acting as the master and multiple Data Nodes acting as
Column families. Regions of Region Server are responsible for several things,
configuration file. 3. Topology Information Format: - Topology information HDFS in Hadoop provides Fault-tolerance and High availability to the s
like handling, managing, executing as well as reads and writes HBase
is in the format of '/myrack/myhost,' where '/' is the topology delimiter, operations on that set of regions. The default size of a region is 256 MB. layer Name Node -NameNode works as a Master in a Hadoop cluste
'myrack' is the rack identifier, and 'myhost' is the individual host. 4. Java Zookeeper – It is like a coordinator in HBase. It providing distributed guides the Datanode(Slaves). Namenode is mainly used for storin
Classes for Topology Mapping: - Performance benefits: Using Java classes synchronization, server failure notification etc. Clients communicate with Metadata. -It manages the file system namespace by performing ope
instead of external scripts avoids forking external processes when region servers via zookeeper. Advantages of HBase –1) Can store large data such as file opening, renaming, and closing. -Because it is a single node,
registering new worker nodes. Why Rack Awareness? The reasons for the sets 2) Database can be shared 3)Cost-effective 4)High availability be the source of a single point failure. DataNode -DataNodes works as
Rack Awareness in Hadoop are: ● To reduce the network traffic while file Disadvantages of HBase –1)No support SQL structure, 2)No transaction DataNodes are mainly utilized for storing the data in a Hadoop cluster
read/write, which improves the cluster performance. ● To achieve fault support, 3)Sorted only on key, 4)Memory issues on the cluster Data Node contains a number of data blocks. -When instructed by the
tolerance, even when the rack goes down (discussed later in this article). ● Node, it creates, deletes, and replicates blocks.
Achieve high availability of data so that data is available even in
unfavorable conditions. ● To reduce the latency, that is, to make the file
read/write operations done with lower delay.

Q) CAP Theorem ?
Eric Brewer first introduced the CAP theorem in year 2000 This theorem is
widely adopted today by large web companies like Amazon as well as the
NoSQL community. The acronym CAP stands for Consistency, Availability
and Partition Tolerance. [Link] Consistency means that all nodes
in the network see the same data at the same time. Consistency refers to
every client having the same view of the data. [Link]: Availability is
Q) HDFS (Hadoop Distributed File System) Architecture
a guarantee that every request receives a response about whether it was
Key Components: 1)NameNode (Master Node): a) Acts as the central brain 3)YARN -YARN, stands for Yet Another Resource Negotiator -YARN
successful or failed. However, it does not guarantee that a read request
of HDFS. b) Manages the file system namespace, storing metadata like: File Framework on which MapReduce works. YARN performs 2 operations th
returns the most recent write 3. Partition Tolerance Partition Tolerance is
names, Directories, Block locations, Replication factors, c) Regulates access Job scheduling and Resource [Link] architecture Apach
a guarantee that the system continues to operate despite message loss or
to files by clients. d)Coordinates DataNode operations. 2)DataNodes (Slave Framework consists of a master daemon known as “Resource Manager”
failure of part of the system. In other words, even if there is a network
Nodes): a)Store actual data blocks of files. b)Responsible for block daemon called node manager and Application Master [Link] - It submit
failure in the data center and some of the computers are unreachable, still
creation, deletion, and replication upon instructions from NameNode. reduce jobs [Link] manager It is Yarn's master daemon. The
the system continues to perform. CA (Consistency and Availability): The
c)Regularly send heartbeats and block reports to NameNode. Key Features: allocation of resources (CPU and memory) across all apps is managed by
system prioritizes availability over consistency and can respond with
1)Master/Slave Architecture: NameNode as master, DataNodes as slaves. resolves conflicts over system resources between competing program
possibly stale data. Example databases: Cassandra, CouchDB,
2)Block Structure: Files are split into large blocks (typically 128 MB) for main components of Resource Manager are: i)Application manager: It h
Riak,Voldemort. AP(Availability and Partition Tolerance)- The system
distributed storage. 3)Replication: Blocks are replicated across multiple the running of Application Masters in the cluster ii)Scheduler: It pe
prioritizes availability over consistency and can respond with possibly stale
DataNodes, typically 3x, for fault tolerance. 4)Write-Once, Read-Many scheduling based on the allocated application and available resources. 3
data. Example databases: Amazon DynamoDB, Google Cloud Spanner.
(WORM) Model: Files can be written once and read multiple times, ensuring Manager (NM): It is Yarn's slave daemon. - Its primary job is to keep-u
CP(Consistency and Partition Tolerance) The system prioritizes consistency
data consistency. Additional Components: 1)Secondary NameNode the Resource Manager. -It is in charge of monitoring container re
over availability and responds with the latest updated [Link]
(SNN): Assists NameNode by periodically merging fsimage and edit logs for utilization and reporting it to the Resource Manager. -It monitors the no
databases: Apache HBase MongoDB, Redis.
faster recovery in case of NameNode failure. 2)Checkpoint Node: Helps in which it is executing.
faster NameNode startup by storing checkpoints of namespace Benefits:
1)Scalability: Easily accommodates massive datasets by adding more
DataNodes. 2)Fault Tolerance: Replication ensures data availability even if
some DataNodes fail. 3)High Throughput: Designed for efficient data access
for large-scale processing. 4)Cost-Effective: Leverages commodity
hardware, making it affordable

[Link] COMMON OR COMMON UTILITIES 1)Hadoop common or Co

utilities are nothing but our java library and java files that we
need for all the other components present in a Hadoop cluster 2)These u
are used by HDFS, YARN, and MapReduce for running the cluster. H
Common verify that Hardware failure in a Hadoop cluster.