Unit 1: Introduction to Big Data

1. Big Data: A massive collection of data that continues to increase dramatically over
time. It is a data set so huge and complex that no typical data management
technologies can effectively store or process it.
2. Volume: The term refers to a large amount of data. The magnitude of data plays a
critical role in determining its worth.
3. Velocity: This term refers to the rapid collection of data. Data comes in at a high rate
from machines, networks, social media, and other sources.
4. Variety: This describes the different forms data comes in, from structured data in
databases to unstructured documents, emails, videos, and audios.
5. Veracity: This refers to the quality of data. It is difficult to link, match, cleanse, and
convert data across systems since it originates from so many diverse places.
6. Data Analytics: The use of advanced analytic techniques to analyze very large,
heterogeneous datasets.
7. Hadoop: An open-source framework that allows distributed processing of large
datasets across clusters of computers using simple programming models.
8. NoSQL: A category of non-relational databases that are designed to handle massive
volumes of data across many commodity servers while maintaining high availability
and avoiding single points of failure.
9. Predictive Analytics: The application of data-driven, proactive maintenance
approaches to examine equipment status and anticipate when repair should be
conducted.
10. Data Management: The process of organizing and maintaining data to ensure its
accuracy, reliability, and accessibility.

Unit 2: Foundations of Big Data

1. File System (FS): A program that controls how and where data is saved, retrieved,
and managed on a storage disk.
2. Distributed File System (DFS): A type of file system that allows many hosts to share
files over a computer network.
3. MapReduce: A programming paradigm that permits tremendous scalability across
hundreds or thousands of computers in a Hadoop cluster.
4. Map Stage: The job of the map or mapper is to process the input data and create
several small chunks of data.
5. Reduce Stage: The job of the reducer is to process the data that comes from the
mapper and produce a new set of output.
6. Fault Tolerance: The ability of a Distributed File System to continue in case of any
partial failures like a link failure, a node failure, or a storage drive crash.
7. Replication: The process of copying the same pieces of information into multiple
clusters on different racks to achieve fault tolerance and high concurrency.
8. Scalability: The capacity of a system to increase or decrease in performance and cost
in response to changes in application and system processing demands.
9. Directed Acyclic Graph (DAG): A directed graph that has no directed cycles.
10. Message Passing: A communication model that allows multiple processes to read and
write data to a message queue without being connected to each other.
Unit 3: Data Models

1. Data Mart: A data store designed for a particular department of an organization, or a

subset of a Data Warehouse that is usually oriented to a specific purpose.
2. Data Warehouse: A large centralized repository of data that contains information
from many sources within an organization.
3. Data Lake: A storage repository that stores a large amount of raw data in its original
format until it is required for analytics applications.
4. Data Streaming: A process in which big data is quickly processed in order to extract
real-time insights from it. The data on which processing is done is data in motion.
5. Dependent Data Mart: A type of data mart that is built by drawing data from an
already existing central data warehouse.
6. Independent Data Mart: A type of data mart that is built by drawing from
operational or external sources of data, or both.
7. Hybrid Data Mart: A data mart that combines input from sources apart from a Data
Warehouse.
8. HDFS: The storage component of Hadoop, which stores data in a distributed manner
across a cluster of machines.
9. Predictive Maintenance: The application of data-driven, proactive maintenance
approaches to examine equipment status and anticipate when repair should be
conducted.
10. Data Model: A method by which data can be organized and stored. A data model
explicitly determines the structure of data.

Unit 4: NoSQL Management

1. Relational Database: A collection of data elements that are linked together by pre-
defined connections in a tabular format with columns and rows.
2. NoSQL Database: Non-tabular databases that store data differently from relational
tables and are designed to handle large volumes of data and heavy user loads with
ease.
3. ACID: An acronym for Atomicity, Consistency, Isolation, and Durability, which are
properties associated with transaction processing using relational databases.
4. BASE: An acronym for Basically Available, Soft State, and Eventually Consistent,
which are properties that NoSQL databases support.
5. Columnar Databases: A database structure that includes only the relevant columns
of the dataset, along with key-identifying information.
6. Key/Value Pair Databases: A simple database that employs an associative array as
its basic data model, with each key corresponding to one and only one item in a
collection.
7. Document Databases: Databases that store an entire document of any size as a single
value for a key element.
8. Graph Databases: Databases that are very well suited for problem spaces with
connected data, such as social networks, spatial data, and recommendation engines.
9. Sharding: The practice of optimizing database management systems by separating
the rows or columns of a larger database table into multiple smaller tables, called
"shards".
10. Replication: A process that takes the same data and copies it over multiple nodes. It
can be done in two forms: master-slave and peer-to-peer.
Unit 5: Introduction to Hadoop

1. Hadoop: An Apache open-source framework written in Java that allows distributed

processing of large datasets across clusters of computers.
2. HDFS (Hadoop Distributed File System): A distributed file system based on the
Google File System (GFS) that provides a distributed file system designed to run on
commodity hardware.
3. MapReduce: A distributed computing model that processes big data fast. The data is
processed in parallel with others.
4. YARN: A framework for job scheduling and cluster resource management in
Hadoop. It manages and schedules resources and decides what should happen in each
data node.
5. NameNode: The central master node in HDFS that manages the file system
namespace and regulates access to files.
6. DataNode: The commodity servers in HDFS where the data is actually stored.
7. Hadoop Ecosystem: A platform or a suite that provides various services to solve the
big data problems, including Apache projects and commercial tools.
8. Fault Tolerance: A feature of Hadoop where data and application processing are
protected against hardware failure.
9. Scalability: The ability to easily grow your system to handle more data simply by
adding nodes.
10. HBase: A NoSQL database that is part of the Hadoop ecosystem, providing
capabilities similar to Google's Big Table for handling big data sets effectively.

Unit 6: Hadoop Administration

1. Hadoop: A software framework for storing and processing data on commodity

hardware clusters.
2. HDFS (Hadoop Distributed File System): A distributed file system designed using
low-cost hardware and is highly fault-tolerant.
3. NameNode: The master server that manages the file system namespace and regulates
access to files by clients. It tracks where across the cluster the file data is kept.
4. DataNode: A slave server that manages storage attached to the nodes they run on.
They are responsible for storing the actual data in HDFS.
5. Blocks: User data is stored in HDFS files, which are divided into one or more
segments called blocks. The default block size is 128MB.
6. Fault Tolerance: A key goal of HDFS, as failure of components is frequent with
commodity hardware, so there must be quick and automatic fault detection and
recovery mechanisms.
7. hadoopnamenode -format: A command used to format the HDFS file system, which
initializes the directory specified by the [Link] variable.
8. [Link]: A command used to start the Hadoop Distributed File System
daemons, including the NameNode and DataNodes.
9. jps: A Java Virtual Machine Process Status Tool used to check whether expected
Hadoop processes are up and running.
10. ssh-keygen: A tool for creating new authentication key pairs for SSH, used for
automating logins and single sign-on in a Hadoop cluster.
Unit 7: Hadoop Architecture

1. Hadoop: An open-source software framework that stores data in a distributed manner

and processes that data in parallel.
2. HDFS (Hadoop Distributed File System): The storage component of Hadoop,
which stores data in a distributed manner across a cluster of machines.
3. NameNode: The master node that manages the filesystem namespace and is aware of
the locations of all the blocks of a file and their size.
4. DataNode: The worker nodes that are inexpensive commodity hardware that store,
retrieve, and replicate blocks when asked by the NameNode.
5. MapReduce: The processing layer in Hadoop that processes data stored in HDFS in
parallel across various nodes in the cluster.
6. YARN: Yet Another Resource Negotiator, which is the resource management unit of
Hadoop and is responsible for managing cluster resources and job scheduling.
7. ResourceManager: The master daemon of YARN that manages the resources across
the cluster.
8. NodeManager: The slave daemon of YARN that runs on all the slave nodes in the
cluster and is responsible for launching and managing containers.
9. Fault Tolerance: A feature of HDFS where data blocks are replicated and stored on
multiple nodes across the cluster to provide high availability of data.
10. Failover: A process in which the system transfers control to a secondary system in
the event of a failure.

Unit 8: Hadoop Master Slave Architecture

1. MapReduce: A programming model for processing large amounts of data distributed

across a number of clusters using steps such as input splits, Map, Shuffle, and Reduce.
2. Hadoop-Streaming: A Hadoop distribution feature that lets developers and
programmers to construct Map-Reduce programmes in a variety of programming
languages such as Ruby, Perl, Python, and C++.
3. Mapper: The component that maps input key/value pairs to a set of intermediate
key/value pairs.
4. Reducer: The component that takes the data returned by the map function and
processes it by decreasing the role to generate a new set of effects.
5. JobTracker: A program that schedules jobs and assigns them to Task Tracker. It
takes job requests from clients.
6. Task Tracker: Keeps track of the task and updates JobTracker on its progress.
7. Job: A program that runs a Mapper and Reducer on a dataset.
8. Task: A Mapper or Reducer task is the execution of a Mapper or Reducer on a slice
of data.
9. ssh-keygen: A tool for creating new authentication key pairs for SSH.
10. HDFS: Hadoop Distributed File System, which allows application data to be accessed
from a distributed file.

Unit 9: Hadoop Node Commands

1. HDFS: Hadoop File System, a distributed file system design built on commodity
hardware.
2. NameNode: The node in the Hadoop Distributed File System that keeps track of all
the data stored in the DataNode.
 3. DataNode: The node in HDFS where the actual data is stored.
4. hadoopnamenode -format: The command used to format the configured HDFS file
system, which deletes all information from the namenode.
5. [Link]: A command used to start the HDFS daemons (Namenode and
Datanodes) from the master machine.
6. hdfs dfs -ls <path>: A command that lists all the files in a specified directory in
HDFS.
7. hdfs dfs -put <localfile> <hdfsdir>: A command to transfer and store a data
file from a local system to HDFS.
8. hdfs dfs -get <hdfsfile> <localdir>: A command to retrieve a file from the
Hadoop file system to the local file system.
9. scp (secure copy): A command-line utility that allows you to securely copy files and
directories between two locations.
10. chmod: A command used to set the correct permissions on the file system.

Unit 10: MapReduce Applications

1. Unit Testing: A testing technique in which individual modules are checked by the
developer to see if there are any flaws.
2. MRUnit: A Java library based on JUnit that lets us unit test Hadoop MapReduce
programmes. It can test Mappers and Reducers separately, as well as combined
MapReduce computations.
3. MapDriver: A core class of MRUnit that is in charge of calling the Mapper's map()
method.
4. ReduceDriver: A core class of MRUnit that calls the Reducer's reduce() method.
5. Shuffling: The process by which the intermediate output from mappers is transferred
to the reducer.
6. Sorting: The process in which the keys generated by the mapper are automatically
sorted by the MapReduce Framework before the reducer starts.
7. Hadoop Scheduler: A component introduced in Hadoop 2 (YARN) that is
responsible for allocating resources to the various running applications.
8. FIFO Scheduler: The default scheduling policy in Hadoop that places applications in
a queue and executes them in the order of their submission (first in, first out).
9. Capacity Scheduler: A scheduler that allows multiple-tenants to securely share a
large Hadoop cluster while maximizing throughput and utilization.
10. JobTracker: A Hadoop service that distributes MapReduce tasks to specified nodes
in the cluster.

Unit 11: Hadoop Ecosystem

1. Apache Hive: An open-source data warehousing solution built on the Hadoop

platform that may be used to analyze and query huge datasets contained in Hadoop
files.
2. HiveQL (HQL): A query language for Hive that is similar to SQL and translates
SQL-like queries into MapReduce jobs.
3. Apache Pig: An open-source tool developed by Yahoo researchers that executes
MapReduce jobs on extensive datasets and provides an easy interface for developers
to process data efficiently.
 4. Pig Latin: A high-level data processing language used in Apache Pig that offers a
rich set of data types and operators to perform several operations on data.
5. Metastore: The central repository where the metadata about Hive tables is stored. It
is a relational database that holds information like database names, table names, and
column details.
6. HBase: A distributed column-oriented NoSQL database built on top of the Hadoop
file system, designed to provide quick random access to huge amounts of structured
data.
7. Zookeeper: An open-source project that provides services like maintaining
configuration information, naming, and providing distributed synchronization for
Hadoop clusters.
8. OLAP (Online Analytical Processing): A category of software tools that provide the
ability to analyze data from different perspectives. Hive is designed for OLAP.
9. Data Ingestion: The process of acquiring and importing data for immediate use or
storage in a database.
10. ETL (Extract, Transform, Load): The process of extracting data from a source,
transforming it to fit business needs, and loading it into a target system. Pig and Hive
have many features common with ETL tools.

Unit 12: Predictive Analytics

1. Linear Regression: A statistical method that allows us to summarize and study

relationships between two continuous (quantitative) variables. It tries to model the
relationship between variables by fitting a linear equation to observed data.
2. Multiple Linear Regression: A method used to estimate the relationship between
two or more independent variables and one dependent variable.
3. Correlation: A metric for determining how closely two variables are linked. The
correlation coefficient is a statistic used to quantify it, with a value from 0 to 1.
4. Data Visualization: The graphical depiction of information and data. It makes it easy
to identify and comprehend trends, outliers, and patterns in data.
5. Univariate Analysis: An analysis technique for data visualization where a single
characteristic is used to examine its features.
6. Bivariate Analysis: A data visualization technique where two characteristics are
compared to discover their correlations.
7. Predictive Analytics: A form of advanced analytics that uses historical data,
statistical modeling, and machine learning to create predictions about future events.
8. Customer Targeting: The process of segmenting a customer base into groups of
people with similar characteristics that matter to marketing.
9. Churn Prevention: The process of anticipating which consumers will leave a firm,
when they will leave, and why they will go.
10. Sales Forecasting: The process of examining past performance, seasonality, and
other factors to produce a reasonable estimate of demand for a product or service.

Unit 13: Data Analytics with R

1. Machine Learning (ML): An application of artificial intelligence that gives systems

the ability to automatically learn and improve from experience without being
explicitly programmed.
2. Supervised Learning: An ML method that uses labeled examples to apply what it
has learned in the past to fresh data and predict future events.
 3. Unsupervised Learning: An ML method used when the data being trained is neither
categorized nor labeled. It infers a function from unlabeled data to describe a hidden
structure.
4. Reinforcement Learning: A branch of machine learning that studies how intelligent
agents should act in an environment to maximize a cumulative reward.
5. Neural Network: An ML algorithm that reflects the behavior of the human brain,
allowing computer programs to recognize patterns and solve problems.
6. Naïve Bayes: A family of simple "probabilistic classifiers" based on applying Bayes'
theorem with strong independence assumptions between features.
7. Clustering: A way of arranging items into clusters so that those with the most
similarities stay in one group.
8. Association Rule: An unsupervised learning approach used to discover associations
between variables in a large database.
9. Markov Decision Process (MDP): A mathematical approach for mapping a solution
in reinforcement learning that uses a set of actions, states, rewards, policies, and
values.
10. Q-learning: A value-based approach of reinforcement learning that helps an agent
decide which action to perform.

Unit 14: Big Data Management using Splunk

1. Splunk: A software used to search and analyze machine data. It serves the needs of
IT infrastructure by analyzing logs generated in various processes.
2. Data Ingestion: The process of acquiring and importing data for immediate use or
storage in a database. In Splunk, it happens through the Add Data feature.
3. Data Indexing: The process of transforming incoming data to enable fast searching
and analysis. The processed results are stored in the index as events.
4. Search & Reporting App: A powerful feature in Splunk that allows you to search
the entire data collection that has been ingested.
5. Dashboards: Splunk dashboards are panel-based displays that can show search
results as charts, reports, and pivot tables.
6. Field Searching: A feature where Splunk evaluates machine data and separates it into
numerous fields, which can then be used to refine search results.
7. Splunk Enterprise: A product category of Splunk used by organizations with
significant IT infrastructure to collect and analyze data from websites, applications,
and devices.
8. Splunk Cloud: A cloud-hosted platform with the same functionality as the enterprise
edition.
9. Splunk Light: A version that allows users to search, report, and get alerts on all log
data in real-time from a single location, with fewer features than other versions.
10. Datameer: A tool that acts as a job compiler or code generator, translating functions,
filters, or joins into native Tez code for execution on a Hadoop cluster