UNIT – II: Data Ingestion into Big Data Systems and ETL:

Big Data Ingestion Tools, Apache Sqoop, Benefits of Apache Sqoop, Sqoop

Connectors, Importing and Exporting to and from Hadoop using Sqoop,

Limitations of Sqoop, Apache Flume Model, Data Sources for FLUME,

Components of FLUME Architecture.

 Data ingestion
Data ingestion the first step in Big Data Pipeline

It is the process of collecting and moving data from various sources into a central
repository (Database).

Example:

A company may ingest data from various sources, including email marketing platforms,
CRM systems, financial systems, and social media platforms.

There are two main types of data ingestion:

Real-time ingestion: involves streaming data into a data warehouse in real-time.

Batch ingestion: involves collecting large amounts of raw data from various sources into
one place and then processing it later.
 Data Ingestion Data Pipeline

The process of collecting and importing dataA series of processes that move and transform data from
from multiple sources into a storage systemsource to destination while ensuring data quality and
(HDFS, databases, cloud, etc.). enrichment.

Focuses on bringing data into a systemFocuses on processing, transforming, and delivering data
(HDFS, cloud, database, etc.). to the final destination.

End-to-end data movement, including ingestion,

Initial stage of a data workflow.
transformation, and loading.

Extracting data from structured,

Includes data ingestion, transformation (cleaning,
semi-structured, or unstructured sources and
aggregation), validation, and storage.
loading it into a destination.

- ETL Pipelines: Apache NiFi, Talend, Informatica

- Batch Ingestion: Sqoop, Talend, Informatica
PowerCenter

- Streaming Pipelines: Kafka Streams, Apache Flink,

- Real-Time Ingestion: Kafka, Flume, Kinesis
Spark Structured Streaming
Key Components of Data Ingestion

The key components of data ingestion ensure that data is collected, processed, and
stored effectively.

Data Sources: Where the data originates, such as databases, APIs, IoT devices, social
media platforms, and cloud storage.

Data Collection: Methods for gathering data, which may involve scraping, querying, or
streaming data from various sources.

Data Transformation: The process of converting raw data into a standardized format
suitable for analysis, including cleaning, filtering, and enriching data.

Data Loading: Storing processed data into a data warehouse, data lake, or other storage
solutions for further analysis.
 Data ingestion tools
Batch Data Ingestion Tools

1. Apache Nifi – Automates data flow across systems.

2. Talend – Open-source and enterprise ETL tool.

3. Informatica PowerCenter – Enterprise-grade ETL for structured and unstructured data.

4. Microsoft SSIS (SQL Server Integration Services) – ETL tool for Microsoft
environments.

5. AWS Glue – Managed ETL for AWS services.

Real-Time Data Ingestion Tools

1. Apache Kafka – Distributed streaming platform for high-throughput ingestion.

2. Apache Flume – Designed for log data collection.

3. Amazon Kinesis – Real-time data streaming in AWS.

4. Google Cloud Dataflow – Stream and batch processing with Apache Beam.

5. Confluent – Enterprise-grade Kafka solution.

Hybrid (Batch + Streaming)

1. Apache Spark Structured Streaming – Unified batch and streaming data ingestion.

2. Flink – Stream processing engine with batch capabilities.

3. Debezium – CDC (Change Data Capture) for databases.

4. Airbyte – Open-source data pipeline tool supporting batch and streaming.

Apache NiFi:

Apache NiFi is a dataflow system that ingests, processes, and distributes data.

It's used to automate data movement and transformation between systems.

It is based on Niagara Files technology developed by NSA.

It supports a wide variety of data formats like logs, geo location data, social feeds, etc.

It also supports many protocols like SFTP, HDFS, and KAFKA, etc.

It Provides an easy-to-use web-based UI to design data pipelines.

It supports real-time monitoring, data provenance, and backpressure handling.

NiFi can process structured, semi-structured, and unstructured data.

Talend

Talend is an ETL tool for Data Integration.

It contains the different products like data quality, application integration, data management, data
integration, data preparation, and big data

It is specialized in Big Data because it has all the plugins to integrate with big data efficiently.

Talend's data integration had an ability which combines the data from the various sources on to a
single view that is highly advanced and of a great utility.

It improves collaboration between different teams in the organization trying to access

organization data.

Provides a drag-and-drop interface for designing pipelines. And offers features like data quality
checks, transformation, and enrichment.
Informatica PowerCenter

Informatica PowerCenter is an ETL tool that is used to extract, transform, and load the
data from the various sources.

We can build enterprise data warehouses with the help of the Informatica PowerCenter.

It is extracting data from its source, transforming this data according to requirements, and
loading this data into a target data warehouse.

Supports structured and unstructured data from multiple sources.

Informatica PowerCenter can access any data source from one platform.

PowerCenter delivers the data on-demand

Microsoft SSIS (SQL Server Integration Services)

SQL Server Integration Services is a platform for building enterprise-level data integration
and data transformations solutions.

It can extract and transform data from a wide variety of sources such as XML data files,
flat files, and relational data sources, and then load the data into one or more
destinations.

Supports data migration, cleansing, and integration.

Can handle both on-premise and cloud data sources.

Integration Services includes:

A rich set of built-in tasks and transformations.

Graphical tools for building packages.

An SSIS Catalog database to store, run, and manage packages.

AWS Glue:

AWS Glue is a service that helps you discover, prepare, move, and integrate data from
multiple sources

It manage ETL service on AWS that automates data preparation.

AWS Glue facilitates all the data integration procedures so you can quickly put your
merged data to good use.

Ideal for batch processing large datasets in AWS environments.

Apache Kafka

Apache Kafka is a distributed data store optimized for ingesting and processing streaming
data in real-time.

Streaming data is data that is continuously generated by thousands of data sources.

Kafka is primarily used to build real-time streaming data pipelines and applications that
adapt to the data streams.

Kafka has three primary capabilities:

It enables applications to publish or subscribe to data or event streams.

It stores records accurately (i.e., in the order in which they occurred) in a fault-tolerant
and durable way.

It processes records in real time (as they occur).

Provides durability, fault tolerance, and scalability. Also used in applications like fraud
detection, IoT, and real-time analytics.
Apache Flume

Apache Flume is a data ingestion mechanism for collecting aggregating and transporting
large amounts of streaming data (log files, events, etc...) from various sources to a
centralized data store.

It is principally designed to copy streaming data (log data) from various web servers to
HDFS.

It has its own query processing engine which makes it to transform each new batch of data
before it is moved to the intended sink.
Amazon Kinesis

Amazon Kinesis is a set of services that helps process and analyze streaming data in real
time.

Kinesis Data Streams enables real-time data intake, and analysis for large data streams,
creating data-processing applications and Directed Acyclic Graphs.

Core Services of Kinesis

Kinesis Streams: consist of shards.

Kinesis Firehose: is a service used for delivering streaming data to destinations

Kinesis Analytics: streaming data is processed and analyzed using standard SQL
Google Cloud Dataflow:

Dataflow is a Google Cloud service that provides unified stream and batch data
processing at scale.

Use Dataflow to create data pipelines that read from one or more sources, transform the
data, and write the data to a destination.

Dataflow uses the same programming model for both batch and stream analytics.

Scales automatically to handle large data streams.

Commonly used for IoT analytics, real-time fraud detection, and log processing.
Confluent:

Confluent Platform is a full-scale streaming platform that enables you to easily access,
store, and manage data as continuous, real-time streams.

It is an enterprise-grade version of Apache Kafka.

Also Provides additional features like schema registry, monitoring, and security.

Used for event-driven microservices, fraud detection, and log analytics.

Supports connectors for databases, cloud services, and enterprise applications.

Apache Spark Structured Streaming:

Apache Spark is an open-source distributed computing system designed for big data
processing and analytics.

Structured Streaming is a scalable and fault-tolerant stream processing on the Spark SQL
engine.

It can process large-scale data in micro-batches with low latency.

Apache Flink

It is a real-time stream processing engine with batch capabilities.

Offers high-throughput, low-latency event processing.

Supports complex event processing (CEP) and machine learning integration.

Debezium

Change Data Capture (CDC) tool that tracks database changes in real time.

Captures inserts, updates, and deletes from databases like MySQL, PostgreSQL, and
MongoDB.

Works with Kafka to stream database changes.

Used for real-time analytics, replication, and event-driven architectures.

Airbyte

Open-source data ingestion tool supporting both batch and real-time processing.

Provides pre-built connectors for over 300 data sources.

Can run in self-hosted or cloud environments.

Supports ELT (Extract, Load, Transform) workflows with easy setup.

 Apache Sqoop
Apache Sqoop is a tool designed to transfer data between Hadoop and relational
database servers.

It is a command-line interface tool

It is used to import data from relational databases such as MySQL, Oracle to Hadoop
HDFS, and export from Hadoop file system to relational databases.
Sqoop Import

The import tool imports individual tables from RDBMS to HDFS.

Each row in a table is treated as a record in HDFS.

All records are stored as text data in text files or as binary data in Avro and Sequence
files.

Sqoop Export

The export tool exports a set of files from HDFS back to an RDBMS.

The files given as input to Sqoop contain records, which are called as rows in table.

Those are read and parsed into a set of records and delimited with user-specified
delimiter.
Features of Apache Sqoop

Sqoop uses the YARN framework to import and export data. Parallelism is enhanced by
fault tolerance in this way.

We may import the outcomes of a SQL query into HDFS using Sqoop.

For several RDBMSs, including MySQL and Microsoft SQL servers, Sqoop offers
connectors.

Sqoop supports the Kerberos computer network authentication protocol, allowing nodes
to authenticate users while securely communicating across an unsafe network.

Sqoop can load the full table or specific sections with a single command.
Advantages of using Sqoop:

It entails data transfer from numerous structured sources, like Oracle, Postgres, etc.

Due to the parallel data transport, it is quick and efficient.

Many procedures can be automated, which increases efficiency.

Integration with Kerberos security authentication is feasible.

Direct data loading is possible from HBase and Hive.

It is a powerful tool with a sizable support network.

As a result of its ongoing development and contributions, it is frequently updated.

Architecture
Using connectors, Sqoop facilitates data migration between Hadoop and external storage
systems.

These connectors enable Sqoop to work with various widely-used relational databases,
such as MySQL, PostgreSQL, Oracle, SQL Server, and DB2.

Sqoop executes user commands via a command-line interface.

The transferred dataset is divided into various divisions, and a map-only job is created
with distinct mappers.

Sqoop uses the database information to deduce the data types, handling each data
record in a type-safe manner.

Sqoop for big data is compatible with various third-party connectors for data storage,
enterprise data warehouses and NoSQL stores
 Sqoop Connectors
Sqoop Driver:

Apache Sqoop uses the JDBC driver to connect to databases and perform required
operations.

each database vendor creates drivers, offered with restrictive licenses

To make the connection between Sqoop to different databases we need to download it

and install it separately.

JDBC is a standard Java API for accessing relational databases and some data
warehouses.

The JDBC drivers from different databases can be installed on the client system under the
$SQOOP_HOME/lib path.

Sqoop uses the JARs present on this path $SQOOP_HOME/lib and loads the classes to
Database Version Support (--direct)? Connect String
HSQLDB 1.8.0+ No jdbc:hsqldb:*//

MySQL 5.0+ Yes jdbc:mysql://

Oracle 10.2.0+ No jdbc:oracle:*//

PostgreSQL 8.3+ Yes (import only) jdbc:postgresql://

CUBRID 9.2+ NO jdbc:cubrid:*

Sqoop Connectors:

Communication with relational database systems, Structured Query Language (SQL) is

designed.

every database has some of its own dialect of SQL. The basics were usually the same, but there
are some changes in some conditions.

using Sqoop Connectors, Sqoop can overcome the differences in SQL dialects supported by
various databases along with optimized data transfer.

A connector in Apache Sqoop is a pluggable piece which is used for fetching metadata about the
transferred data (such as columns, data types, …)

The connector will work on various databases and we don’t need to download the extra
connectors for starting data transfer.

Sqoop, connectors interact with databases using four main components: Partitioner, Extractor,
Loader, and Destroyer.

These components handle different stages of data ingestion and export.

Sqoop Connectors – Partitioner

In this phase, the partitioner Determines how data is divided into chunks (splits) for
parallel execution.

it also -generate conditions that can be used by the extractor.

If there is no specification from the user end then a primary key will be used to partition
the data.

Example:

If we have 1 million records and 4 mappers, the partitioner divides the data into 4 equal
chunks.

Syntax:

sqoop import --connect jdbc:mysql://localhost/db --username user --password pass --table

employees --split-by emp_id --num-mappers 4
Sqoop Connectors – Extractor:

In this phase, the JDBC data source reads data from the database using database-specific
queries.

Converts relational database records into Hadoop Writable format (Text, Avro, Parquet, etc.).

Syntax:

sqoop import --connect jdbc:mysql://localhost/db --username user --password pass --table

employees --SELECT * FROM <table name>

Sqoop Connectors – Loader:

In this phase, the JDBC data source is queried using SQL to load data into HDFS, Hive, or HBase.

SQL queries will vary based on your configuration.

Syntax:

sqoop import --connect jdbc:mysql://localhost/db --username user --password pass --table

employees --hive-import --as-parquetfile
Sqoop Connectors – Destroyer:

In this phase, below operations are performed.

Handles cleanup tasks after job execution.

in case of failure, Removes temporary tables, intermediate files, or rollback operations

Perform the copy operation of the staging table to the concerned table.

Once the copy operation is completed then empty the staging table.
How to use Sqoop Drivers and Connectors:
Sqoop uses connectors and drivers to facilitate data transfer between Hadoop and
relational databases.

The Sqoop Connector acts as a bridge, enabling communication between Sqoop and
the target database by understanding its structure.

Each database, such as MySQL, Oracle, or PostgreSQL, has its own specialized
connector for efficient data transfer.

The Database Driver, typically a JDBC driver, is responsible for executing the actual SQL
queries needed to fetch or insert data into the database.

When a Sqoop import or export command is executed, the connector interacts with the
driver, which then communicates directly with the database to process the data transfer.
 Sqoop – Import/Export
Sqoop tool in the Hadoop ecosystem, simplifies the process of importing and exporting data
between Hadoop and relational databases

when data is transferred from a relational database to HDFS, is importing data.

we transfer data from HDFS to relational databases, we say we are exporting data.

Syntax:

sqoop TOOL_NAME [TOOL_OPTIONS] [GENERIC_OPTIONS] [TOOL_ARGUMENTS]

TOOL_NAME: Specific Sqoop tool being used, such as import or export.

TOOL_OPTIONS: Options specific to the chosen tool, specifying details about the import/export
process.

GENERIC_OPTIONS: These options are common across all sqoop import and export tools and
control general behaviors.

TOOL_ARGUMENTS: Additional arguments or parameters required by the tool.

Sqoop Import:

The Sqoop import command is used to transfer data from relational databases to the
Hadoop Distributed File System (HDFS).

Syntax

sqoop import \

--connect connection_string \

--username user \

--password pass \

--table tablename \

--columns col1,col2 \

--target-dir /user/hadoop/hdfs-dir
 --connect: Specifies the JDBC connection URL for the source database.

Example: --connect jdbc:mysql://localhost/mydb

--username: Specifies the username for connecting to the source database.

--password: The password for the source database user.

--target-dir: The HDFS directory where imported data will be stored.

--table: Name of the source database table from which data will be imported.

--columns: A comma-separated list of columns to be imported from the source table.

Examples:

sqoop import \

--connect jdbc:mysql://localhost:3306/mydb \

--username myuser \

--password mypass \

--table orders \

--columns order_id,order_date,product_name \

--where "order_date >= '2023-01-01'" \

--target-dir /user/hadoop/order_data
Supported Data Formats:

sqoop tools offers support for various data formats when importing data from relational
databases into Hadoop's Distributed File System (HDFS).

The supported formats are:

Avro

Parquet

SequenceFile

Text
Importing Incremental Data:

It is useful in scenarios where only the newly added or modified records need to be
transferred from a source database to the Hadoop ecosystem.

Working:

Sqoop compares the specified column between the source and target datasets.

Sqoop then imports only the records that have a higher value in the specified column than
the maximum value present in the target dataset.

Now, only the new or modified records are transferred, significantly reducing the amount
of data transferred and improving overall efficiency.
Syntax:

sqoop import \

--connect jdbc:mysql://hostname:port/database \

--username user \

--password pass \

--table tablename \

--target-dir /user/hadoop/hdfs-dir \

--check-column column_name \

--incremental mode \

--last-value last_value
Example:

sqoop import \

--connect jdbc:mysql://localhost:3306/mydb \

--username myuser \

--password mypass \

--table transactions \

--target-dir /user/hadoop/transaction_data \

--check-column transaction_date \

--incremental lastmodified \

--last-value '2023-01-01'
Importing Data with Hive Integration

• When importing data with Hive integration, Sqoop directly populates Hive tables with the
imported data.

• Sqoop import and export optimizes Hive's metadata and data management capabilities.

• The imported data is immediately available for analysis and querying using Hive's
SQL-like language, HiveQL.
Syntax:

sqoop import \

--connect jdbc:mysql://hostname:port/database \

--username user \

--password pass \

--table tablename \

--hive-import \

--create-hive-table \

--hive-table hive_tablename \

--hive-partition-key key \

--hive-partition-value value \

--hive-overwrite
Example:

sqoop import \

--connect jdbc:mysql://localhost:3306/mydb \

--username myuser \

--password mypass \

--table products \

--hive-import \

--hive-table hive_products \

--hive-overwrite
Sqoop Export:
Sqoop export command allows transfer of data from Hadoop to external relational
databases.

Sqoop maps the columns in HDFS to the columns in the target database table and
efficiently inserts or updates the data based on the specified criteria.
Syntax:

sqoop export \

--connect jdbc:mysql://hostname:port/database \

--username user \

--password pass \

--table tablename \

--update-mode mode \

--update-key key \

--batch \

--export-dir /user/hadoop/hdfs-dir
Example:

sqoop export \

--connect jdbc:mysql://localhost:3306/mydb \

--username myuser \

--password mypass \

--table results \

--export-dir /user/hadoop/data \

--update-key id \

--update-mode updateonly
Exporting with Hive Integration:

Exporting data from Hive to external relational databases is used to transfer processed
or analyzed data from Hive tables to specific tables in external databases.

Working:

• Exporting data with Hive integration involves transferring data from Hive tables to external
databases using Sqoop.

• Sqoop interacts with the Hive metastore to understand the structure of the Hive table,
including column names, data types, and partitioning information.

• It then maps the Hive table's columns to the columns in the target database table and
efficiently inserts or updates the data based on the specified criteria.
Syntax:

sqoop export \

--connect jdbc:<database_connection_url> \

--username <db_username> \

--password <db_password> \

--table <target_table_name> \

--hcatalog-table <hive_table_name>

--table: Specifies the target database table where data will be exported.

--hcatalog-table: Name of the Hive table from which data will be exported.
Example:

sqoop export \

--connect jdbc:mysql://localhost:3306/mydb \

--username myuser \

--password mypass \

--table employees \

--hcatalog-table hive_employees
Limitations of Sqoop

Sqoop is designed to work primarily with relational databases (RDBMS) and does not
support for non-relational data sources

Once data is imported/exported, it cannot be rolled back in case of failure or errors during
the process.

Sqoop allows parallel execution by using multiple mappers, Hence, it is slow

Sqoop is not well-suited for real-time data transfer. It is designed for batch processing

Sqoop does not provide advanced ETL (Extract, Transform, Load) capabilities.

When importing/exporting data between Hadoop and RDBMS, certain data types may not
map directly.

Sqoop supports basic authentication using usernames and passwords, it lacks more
advanced security mechanisms
 Apache Flume Model
Apache Flume is a tool in the Hadoop ecosystem for transferring data from one location to
another efficiently and reliably.

It is principally designed to copy streaming data (log data) from various web servers to
HDFS.

Apache Flume is a distributed system for collecting, aggregating, and transferring log data
from multiple sources to a centralized data store.

It can be used to transport large amounts of social-media generated data, network traffic
data, email messages, and many more to a centralized data store.
Key Components:

Data Generators

Data generators generate real-time streaming data.

The data generated by data generators are collected by individual Flume agents that are running
on them.

The common data generators are Facebook, Twitter, etc.

Flume Event:

Flume event is the basic unit of the data that is to be transported inside Flume.

It contains a payload of byte array that is to be transported from the source to the destination
accompanied by optional headers.

It consists of two parts:

Header: Metadata or additional information about the event (e.g., timestamps, source
information).

Body: The actual data or message (e.g., log entry, JSON record, etc.).
Flume Agent:

The agent is a JVM process in Flume.

It receives events from the clients or other agents and transfers it to the destination or
other agents.

Flume may have more than one agent.

Flume Agent contains three main components namely, source, channel, and sink.
Source:

A source is the component of an agent and responsible for ingesting or collecting data
into Flume.

It listens incoming data (events) from external systems (log files, network sources) and
transfer into Flume’s channel.

Different types of sources include:

❑ Avro Source: Accepts events in Avro format over the network.

❑ Syslog Source: Reads events from syslog servers.

❑ Spooling Directory Source: Watches a directory for new files and ingests events from
them.

❑ Exec Source: Executes a command (like tail or shell scripts) and reads its output.
Channel

Flume Channel acts as a bridge between Flume sources and Flume sinks. It is a passive
data store that acts as a buffer

When a source receives events, it writes those events to the channel. The sink then reads
from the channel and processes the data further.

The channels are fully transactional and they can work with any number of sources and
sinks.

Types of channels:

Memory Channel: Stores events in memory, fast but volatile (events are lost on failure).

File Channel: Stores events in the file system, more reliable but slower than memory.

Kafka Channel: Uses Apache Kafka as a channel to store and forward events.
Sink

The Flume sink retrieves the events from the Flume channel and pushes them on the
centralized store like HDFS, HDFS, or passes them to the next agent.

The destination of the sink might be another agent or the central stores.

Common types of sinks:

HDFS Sink: Writes events to Hadoop Distributed File System (HDFS).

HBase Sink: Writes events to HBase tables.

Kafka Sink: Publishes events to a Kafka topic.

ElasticSearch Sink: Sends data to an Elasticsearch cluster.

Data collector

The data collector collects the data from individual agents and aggregates them.

It pushes the collected data to a centralized store.

Centralized store: Centralized stores are Hadoop HDFS, HBase, etc.

Interceptors

Interceptors are for altering or inspecting Flume events transferred between Flume source
and channel.

They are used to apply transformations such as filtering certain types of logs, adding
headers, or converting event formats.
 Data Sources for FLUME
Data Source Description Use Case Example

Monitors directories and captures

Ingesting server/application logsWeb server logs from
Log Files log files (e.g., web server logs,
into Hadoop for analysis. Apache/Nginx.
application logs).
Collects logs from syslog servers
Centralized logging of system
Syslog for system and network System logs from Linux servers.
events for monitoring.
management.
Ingests data from network streams
Network Streaming data from networkedCollecting network data for traffic
using protocols like Avro, Netcat,
Streams devices for real-time processing. analysis.
or Thrift.
Twitter Ingesting social media feeds for
Collects real-time data from social Real-time tweets about a specific
Streaming sentiment analysis or trend
media platforms like Twitter. hashtag.
API detection.
Receives data via HTTP POST
HTTP POST Capturing user activity from webCollecting data from a RESTful
requests from web applications or
Requests applications or APIs. API.
other systems.
Executes shell commands (e.g.,
Capturing real-time log output or
Exec Source tail -f on log files) and captures the Running tail -f on a log file.
command results.
output.
Data Source Description Use Case Example

Reads data from Java

Ingesting messages from aReading from ActiveMQ or
JMS Source Messaging Service (JMS)-based
JMS-based message queue. RabbitMQ.
messaging systems.
Ingests data from Apache Kafka
Processing data from real-timeReading messages from a Kafka
Kafka Source topics in real-time for further
streaming platforms. topic.
processing.
Spooling
Monitors a directory for new filesBatching and processing filesCollecting new log files as they
Directory
(e.g., log files) and ingests them. from a directory in Hadoop. arrive.
Source
Accepts Avro-formatted dataExchanging Avro data
Transmitting data between Flume
Avro Source over the network from otherbetween Flume agents in a
nodes.
Flume agents or sources. distributed setup.
Ingests data from Thrift RPCRPC-based data ingestionCapturing data from Thrift-based
Thrift Source
calls over the network. from distributed systems. applications.
Allows the creation of customIngesting data from proprietary
Custom Custom source for IoT sensors or
sources to capture data fromsystems or custom
Applications APIs.
specific applications or systems. applications.
 Sqoop Flume

Designed for batch data transfer between RDBMS and Hadoop Designed for real-time data ingestion from streaming sources to
ecosystems (HDFS, Hive, etc.). Hadoop or other systems.

Imports/exports large datasets between relational databases

Collects, aggregates, and moves real-time log/event data to Hadoop.
and Hadoop.

Sources structured data from relational databases (MySQL, Ingests data from streaming sources like logs, social media feeds,
Oracle, PostgreSQL, etc.). syslog, etc.

Transfers data in batch mode, typically large datasets at

Transfers data in real-time, streaming continuously as it's generated.
scheduled intervals.
Can handle structured, semi-structured, and unstructured data (e.g.,
Works well with structured data (tabular format).
logs).
Destinations are primarily Hadoop components (HDFS, Hive, Can deliver data to multiple systems like HDFS, HBase, Kafka, and
HBase, etc.). Elasticsearch.
MapReduce-based for parallel data transfer. Agent-based architecture for real-time event-driven data flow.

Best for moving large datasets between databases and Hadoop. Best for real-time log collection and streaming data ingestion.

Works with a wide range of sources and destinations, not limited to

Works with SQL databases and Hadoop.
Hadoop.